Tutorial 3.5: Tracking and Stabilizing

Eat at Joe's: Basics of Tracking

Get On the Bus: 4-Point Tracking

This tutorial discusses the three tracking nodes in Shake: Tracker, Stabilize, and MatchMove.

The first section ("Eat At Joe's") corrects a clip with a drifting camera. It discusses the basic mechanics of tracking, and how to convert stabilization data to matchmove data.

The second example ("Get On the Bus") discusses tracking guidelines and executes a classic 4-point track to match a sign onto a bus side. As original as eggs for breakfast.

For more detailed information on tracking parameters and how a tracker functions, jump to About Tracking. For formatting examples of the three tracking nodes, jump to the Tracker, Stabilize, or MatchMove reference pages.

Basic Tracking and Stabilizing

Tracking allows you to do two things: First, you can eliminate unwanted movement in a plate, for example, the movement caused by film gate weave. This is known as stabilizing. Second, you can match a stable element to a moving element, such as swapping the heads of two actors. This is called matchmoving. Both involve tracking patterns in an image sequence from frame to frame. This can be done manually, but tends to get really tired, really fast. In Shake, three nodes track sequences. The three nodes are:

In the first section, stabilize a moving clip, and then switch the stabilization data to track an image to the moving clip in a matchmove.

This does not affect your rendered frames, but helps you adjust your track points.

You have by now no doubt discovered the plate drifts to the left. This is a prime candidate for stabilization, which locks the image down.

A tracker appears in the Viewer.

When the cursor is positioned over the tracker box, the tracker is highlighted. You can move a highlighted tracker box in the Viewer.

The tracker is composed of the track point, the reference pattern, and the search region. A pattern is isolated with the inner box called the reference pattern. Once tracking begins, the tracker looks for that pattern within the larger box called the search region. When a match is found, a track point (keyframe) is created.

Tracking Large Plates. To decrease image load times for large plates, activate limitProcessing in the Stabilize parameters.

When the plate drifts, the Stabilize node moves it back to the original point. This eventually causes a black border appear around the image. The simple (lazy) way to fix this is with a Scale node to adjust the plate size. This concatenates with the Stabilize node. This is good.

As this is the Lazy Man's Method, and therefore appealing to compositors across the world, it is not the ideal method. Instead, a clean plate should be constructed and placed behind the image. See How to Make a Clean Plate.

Converting Stabilization Data to MatchMove Data

Use the same building clip to build a simple matchmove. In this case, remove the stabilization data from the building clip, invert the stabilization data, and apply the data to a second image.

Since you have already tracked the background, you can attach the Stabilize1 node to the EatAtJoes node, and then use a Layer - Over node to composite the two together.

The EatAtJoes image drifts away from the background image. The Stabilize node is trying to stabilize the image—the opposite of what you want. To match the EatAtJoes image to the movement of building clip, invert the transform with the shrewdly-named invertTransform button (by setting it to match).

The EatAtJoes sign moves with the background plate.

As shown, you can use Stabilize for stabilization and matchmoving. Others lurking around the office prefer to use the MatchMove node, described in the following section. There is one gotcha with using Stabilize for matchmoving: You must turn off the transform and connect the Stabilize node to the background (tracked clip) in order to re-track.

Destabilizing. The inverseTransform toggle can also be used to reapply the same jitter that was removed. You may have a slight jitter due to gate weave. Stabilize the shot, apply all of your elements to the stabilized plate, and then copy and paste the original Stabilize node to the end of the tree. Then, set inverseTransform to match, and you arrive with the same natural-feeling jittery movement.

Tracking Behaviors

The next example uses more complicated tracking information to match a background clip that changes perspective. The MatchMove node is used to match the movement of the background. That's where the devilishly clever name MatchMove comes from. In this example, read in a clip of a moving bus and an advertisement image, and apply the image to the side of the moving bus. This tutorial also discusses some solutions for common tracking problems you are likely to have in future projects.

The office of the Shake team appears in the clip. The building, not the bus.

The following illustration shows is what you are trying to achieve in this lesson. Yeah, this is the shot you put first on your reel, right after "I am a dork. Don't hire me."

Before After

Although this project calls for a 4-point track (one for each corner), begin with only one tracker to keep things simple. Several intentional mistakes occur in this tutorial to help you better recognize problems with tracking your own footage. As they always do.

Doh! The tracker advances a few frames and stops. The last number in the text next to the tracker box, reads "c=.2" (approximately). This indicates the correlation, or accuracy of the track. A score of 1 means a perfect correlation with the reference pattern between frame 2 and frame 1 (frame 1 as the reference pattern). A score of 0 is amazingly bad. On frame 2, you have a correlation of .2. This is bad.

The search region is too small to encompass the movement in the clip—the bus moves too far to the right. In this illustration, the desired area, artificially marked in green, extends beyond the zone of the red search region.

If the search region is too large, you waste time processing. If the search region is too small, the tracker does not find successful matches. There is also a black line around the sign along the bottom of the pattern. There is a chance due to the perspective shift in later frames that the tracker will confuse the two lines. Adjust the search area with these in mind.



Adjusting the Tracking Boxes. Grabbing an edge adjusts just that edge. Grabbing a corner scales the entire box. Grabbing the center moves everything. What is adjusted is highlighted in yellow.

Gee, everything was going fine until around frame 34, right? At that frame (results vary), the tracker spitefully decided that the words on the sign were the best match to the original reference pattern at frame 1.

The tracker continually compares the new images back to frame 1 (or whatever frame you start the track). As the bus moves away, the sign gets smaller. At a certain point, the words match the original thickness of the sign border from frame 1 better than the sign border itself.

This matching behavior is controlled by the referenceBehavior setting in the tracker parameters under the tolerance behavior subtree. It is set to use start frame by default—it compares the new samples to the original start frame (not necessarily frame 1 if you start at a later frame).

Two settings in the referenceBehavior can help with scaling changes in the image sequence. The settings are update every frame, that uses the sample of the previous frame for the reference pattern, and update if below reference tolerance, that checks the correlation of every frame. If the correlation falls below the referenceTolerance value (right above referenceBehavior), it uses the previous frame as the reference pattern. It continues to use that pattern until the correlation once again dips below the referenceTolerance. The second one (update if below...) is more accurate because you get inherent drift with update every frame as all of the tiny errors accumulate.

Although you can track from frame 33 (the frame before the tracker planted), it is better to find a frame with a high correlation. The correlation is displayed in the yellow text next to the tracker box. Ideally, this is of course frame 1, but you may find a high correlation at approximately frame 14.

Well, that's one track. Now the other three trackers.

The three extra trackers, pre-positioned on the image, are activated. Each tracker is located in the relative corner where it should stay. For example, the upper-left corner of the foreground image is connected to track4 of the MatchMove. Shake uses Cartesian coordinates, so the trackers are ordered according to the following image.

Yes, you are right, this is different from some other prominent systems. Deal with it.

The first track point, track1, is already set and its keyframes are visible.

Every time you start a track, all visible trackers analyze the image and create new keyframes. Therefore, turn off the visibility of track1. The four trackers are listed in the lower portion of the MatchMove parameters. Click the track name text field (that is, where is says "track1") to change the name. Change the display color of a tracker by clicking on the color picker button. The V button toggles the visibility of a tracker.

An active tracker is green in the tracker list, and yellow in the Viewer.


Manual Tracking? Don't ever ever ever believe a product demo—tracking, like keying, is rarely straightforward and usually involves a lot of manual labor. Ensure AutoKey is on and away you go. You frequently must adjust points by hand. Welcome to the exciting world of production! See what this magical world looks like as you drive home at 4:00 a.m.! You have been warned...

Positioning the Foreground Element

With the four tracks plugged into the MatchMove, you can now apply the foreground element.

In the MatchMove parameters, the outputType list contains several compositing options. Background is selected by default to allow you to immediately start tracking. To test the track, switch the outputType to a compositing operation.

This produces a somewhat less-than-convincing composite.

As with Stabilize, you must activate the transformation.

 

If an image appears brighter in an Over operation (in effect adding the two images), you should immediately suspect the absence of a mask in the foreground. Since the sign image is a JPEG and does not support alpha channels, you must add an alpha channel to the image.

 

This is an improvement. The four corners of the foreground image are snapped to the four corners of the trackers. If the image is twisted around, you have done something such as put track3 in track4's corner. You can unravel that in the next step.

The sign probably may not fit exactly into the background's sign limits. A little bit peeks out from behind. You therefore need to adjust the position of the foreground sign.

To adjust the foreground image position:



What Happened to my MatchMove? When you toggle the Viewer BG/FG button , MatchMove automatically switches its outputType from Foreground to Over. Occasionally, this little bit gets overwhelmed and confused and you end up losing the composite setting because Shake doesn't reset outputType. Manually switch it back to Over.


Sign Color Correction

Anybody who has visited Los Angeles knows that a clean bus is a mythical beast. As the final step, color correct the sign to better match it to the bus.

Before After

For the color correction, first adjust the whites and blacks with the Compress node. Because the sign has pure whites and blacks, and the bus has good representations of what should be white and black, Compress works well.

As the bus changes its lighting levels as it turns, animate the High Color parameter.

The sign is a neutral grey color. Some warm color makes it stick out a bit and draws the viewer's attention (though not too much—it still has to be integrated into the scene). You can tint it yellow with Compress1, but you then must have the same modification for all of your keyframes, which is cumbersome. A better idea is to use the Mult node to tint it. As Compress and Mult concatenate, there is no loss of color quality. This is good. Very good.

The three nodes (Compress1, Mult1, and Saturation1) demonstrate an advantage of working with nodes and concatenation. Because Compress1 is animated, it is awkward to apply adjustments to the overall color for the entire clip length within the Compress node. By breaking these steps into separate nodes, you don't have to adjust each keyframe.

Next, because the Compress raised the black levels in infinite space, the Color - SetBGColor is used. This sets area outside of the DOD to a specific color (black by default). You can also use a MMult since the mask is pure white, but you don't want any confusion about the premultiplied status. If the sign has a soft mask, you must use a Mdiv/MMult pair.

The sign is color balanced in the scene, but the shadow, created with the AddShadow node, needs to be adjusted. Without this node, the sign has no feeling of depth.

No Shadow With AddShadow

 

More Stuff The script doc/pix/bus_track2.shk adds a little color correction and masking to take care of the reflections that go across the side of the bus from frames 36 to 40. Although interesting, it does not seem necessary for World Peace to have it included in the tutorial.

Stabilize as an Alternative to MatchMove

You may have picked up that tuning the corners of the sign using the FG/BG Viewer controls was less than ideal. Some might use the word "maddening and suicide-inducing," as there is no interactive feedback. An added drawback is onscreen controls from upstream are not properly adjusted by the MatchMove node. This is one advantage that the Stabilize node has over the MatchMove node. As demonstrated in the Eat At Joe's tutorial, you can use Stabilize instead of MatchMove to create matchmoves. This gives you a more intuitive control mechanism, but requires more plumbing in the Node View. Note for this exercise, you are copying the tracks from MatchMove1, but you could have generated them with Stabilize.

The following is the general workflow for matchmoving with Stabilize:

This technique has several advantages:

Switch the tracking from the MatchMove node to a Stabilize node:

Next, copy the tracks from MatchMove1 to Stabilize1.

 

A pop-up window appears that lists all tracks.

Things don't quite line up in the Viewer. Oops.

Frame 1 Frame 30

MatchMove has controls for adjusting the foreground position. Stabilize, by default, assumes that frame 1 has the relative position that you want. The selected frame can be modified with the referenceFrame parameter, but it still doesn't modify the relative position. You must therefore rely on external nodes to adjust this, in this case the CornerPin node.

The controls are now completely intuitive, as well as interactive. Just like the DMV.