Yannix has been pioneering virtual camera matchmove and object tracking technologies for the past 25 years. We have developed the ability to reconstruct many different kinds of image distortion issues that occur in footage captured in movies, streaming video and advertising where VFX is added. These include the typical spherical lens distortion that a real lens will have, but the “perfect” digital camera lenses do not.

But there are also image artifacts caused by rolling shutter, anamorphic lens distortion, focus changes on anamorphic lenses, or even footage where the action is happening in a mirror or on the other side of a glass window.

In every case, Yannix has developed a VFX pipeline that can model and reconstruct what is happening when the live action footage is captured so that the CG elements match as tightly as is possible.

While VFX is done on digital images made up of rows of pixels, the images are captured from the real world where rays of light bounce off of objects, pass through a lens and are focused onto an image capture plane (previously film, but nowadays a digital image sensor such as a CCD or CMOS).

In order to understand image artifacts such as “lens distortion” or “rolling shutter”, it helps to think about what is happening at the moment of the image capture.

“Rolling shutter” is a method of image capture where the image is scanned (normally top to bottom) such that different rows of pixels in the resulting digital image are captured at different times.

This is in contrast to “global shutter” where all pixels of the image are captured at the same instant in time. Rolling shutter may be either mechanical or digital, but the effect on the images is the same. When creating VFX for footage captured with a rolling shutter, any camera motion or objects moving in the plate will result in skewing or stretching which can have a significant impact on the VFX pipeline.

Image 1 represents a simple demonstration of the effect of rolling shutter when panning the camera to the left while capturing a building, causing the building to move right on the plate. In a simplified scenario where the top third of the image is captured first, the middle third is captured a bit later, and the bottom third is captured a little later than that, the building will have moved progressively rightward in the lower sections of the image as seen in Image 1B. Image 1C is closer to what happens in a real rolling shutter situation where every row of the image is scanned at a slightly later point in time.

Image 2 is real video footage captured of a vertical line on a spinning cylinder. You can see that while the line is vertical when the cylinder is stopped, the line skews when it is moving to the right.

While less easy to see, rolling shutter also has an effect on objects moving vertically, where objects will stretch slightly when moving downwards (when the camera scans pixels from top to bottom) and compresses them when moving upwards. Unless this effect is compensated for, this, too, can cause noticeable slippage between the CG and the live action plate.

When viewing footage of a normal scene captured with a professional digital video camera with normal amounts of motion, the effect of rolling shutter may be slight enough that an average viewer will not notice it. But even small amounts of rolling shutter artifacts can be particularly problematic in a VFX pipeline since the virtual elements must “stick” to the live action precisely or risk viewers feeling the effect is fake.

Therefore, the CG camera used to render the CG elements must line everything up to the skewing and stretching of real objects in the live action plates to avoid the appearance of slippage between CG and live action elements.

For more about how Yannix addresses rolling shutter in a VFX pipeline, please see as this link: How To Handle “Rolling Shutter” (Part Two)