About Tracking, Stabilizing and MatchMoving

This section discusses tracking in depth, including interface features, workflow issues, and tips on successful tracking and manipulation the data. For specific formats of each node, jump to the appropriate reference page. For a tutorial on how to use the trackers, jump to the Tracking Tutorial.

There are three tracking nodes available in Shake – Tracker, Stabilize, and MatchMove.


Tracking Workflow

This is a general overview of the steps you take to generate a track. Each of these is detailed in the following sections.

General Tracking Workflow:

  1. Play your background clip several times to determine a good tracking point.

  2. Attach your tracking (Transform - Tracker, Stabilize, or MatchMove) node to the background clip.

  3. Make sure the onscreen controls are visible in the Viewer .

  4. Go to the frame that you want to start the track.

  5. Position the tracker on the point you want to track and adjust the reference pattern and the search region.

  6. Ensure the trackRange parameter reflects the frame range you want. For FileIns, it takes the range of the clip. For Shake-generated elements you must provide a range, for example, 1-50.

  7. Press either the backward or forward track button to start the track processing (which ideally needs to be done once, but never really happens that way). The track generates animation curves.

Stabilize additions:

  1. Decide if you need one, two, or four-point tracking.

  2. Toggle the applyTransform parameter to on to stabilize the plate.

  3. If you are using Stabilize for match-moving, set the inverseTransform parameter to match.

MatchMove additions:

  1. Attach the foreground element to the first input of the MatchMove.

  2. Decide if you need one, two, or four-point tracking.

  3. Set the outputType parameter to Over (or another compositing operation).

  4. Toggle the applyTransform parameter to active to match the motion.

  5. If you are four-point tracking and you want to attach four arbitrary points on the foreground image to the background, press the BG/FG button to toggle it to FG . Then place the four corners on the Foreground element. These represent the corners that get plugged into the four tracking points.

  6. Toggle back to BG .

  7. You may have to switch the outputType parameter back to Over, or the compositing mode you set in step 3.

 

Tracking Onscreen controls:

All tracking nodes share the following interface features:

A tracker consists of three onscreen controls: The search region, the reference pattern, and the track point. The outer box is the search region, the inner box is the reference pattern. The cross hair is the track point.


 

 

 

One limitation with MatchMove is that it does not handle the pass-through of upstream onscreen controls, so those controls are in their un-transformed state.

 


Tracking Viewer Buttons:

Track Backward/Forward. Click to start the tracking process. All visible trackers attempt to lay down new tracking keyframes. Tracking continues until one of the trackRange frame limits is reached – the upper limit if you are tracking forward , or the lower limit if you are tracking backward or until your correlation falls below your failureTolerance.

Offset Track - Off. The track search region and the tracking point are linked. If you move one, the other follows.

Offset Track - On. The track search region and the tracking point are offset from each other. If your original sample pattern becomes obscured, offsett the search region to a different area. Your keyframes are still saved relative to the original spot.

Reset Track. Returns an offset track search region back to the track point.

Track Display. These buttons display all trackers, curves and keyframes, just the trackers and keyframes, or just the trackers. You can also control visibility of individual trackers as described below.

These buttons appear on the MatchMove node when four-point tracking. They allow you to toggle between adjusting the track points on the background (BG) or the corners of the foreground (FG) that get pinned to the track points.

Next to each track name is a Color Picker and a Visibility toggle. The Color Picker selects a different color for the display of that tracker. The Visibility toggle turns on and off the display of a tracker. When enabled, it retracks each time you press the Track button. If the tracker is invisible, it is not processed next time you press the Track buttons.





Tracking Parameters

All trackers share the following parameters:

Parameter
Notes
trackRange
The trackRange parameter is the potential frame range limit of your tracking. By default, tthe range is set to the clip range. For generated elements such as RGrad, it will take a range of 1. You can set new limits using Shake's standard range description, for example., 10-30x2. If you stop tracking and start up again, it starts from the current frame until it reaches the lower or upper limit of your trackRange, depending on if you are tracking forwards or backwards.
subPixelResolution

The subPixelResolution parameter determines the resolution of your track. The smaller the number, the more precise track.

Possible values:

1 Area is sampled at every pixel. Not very accurate or smooth, but very fast.
1/4 Area is sampled at every .25 pixels (16 times more than with a sampling of 1).
1/16 Area is sampled at every .0625 pixels (256 times more than with a sampling of 1).
1/32 Area is sampled at every .03125 pixels (1024 times more than with a sampling of 1).
1/64 Area is sampled at every .015625 pixels (4096 times more than with a sampling of 1).
matchSpace The pixels are matched according to the correlation between the selected color space, either luminance, hue, or saturation. When an image has roughly the same luminance, but contrasting hues, you should switch to hue-based tracking.
referenceTolerance A tracking correlation of 1 is a perfect score – there is an exact match between the original reference frame and the sampled area. When you lower the referenceTolerance, you accept greater inaccuracy in your track. If tracked keyframes are between the referenceTolerance and the failureTolerance, they are highlighted in the Viewer. Also, in some cases, referenceBehavior is triggered if the tracking correlation is below the referenceTolerance.
referenceBehavior

This behavior dictates the tracking area reference sample. By default, the reference pattern is the first frame at which you start the track, not necessarily the first frame of the trackRange. The last two behaviors in the referenceBehavior list measure the tracking correlation and match it to the referenceTolerance to decide an action.

use start frame The new samples are compared to the reference pattern from the first frame of the track. If you stop tracking midway, and start again at a later frame, the later frame is used as the reference sample.

update every frame

The source sample is updated from the previous frame. This usually creates an inherent drift in the track, as tiny errors accumulate. This method is for movements that have drastic changes in perspective and scale.
update from keyframes If you are using a failureBehavior of predict location and don't create keys or don't predict location, a keyframe is not necessarily saved every frame. In this case, you may only want to update from the last frame that had a valid keyframe.
update if above reference tolerance This updates the reference sample from the previous frame if the correlation is above the referenceTolerance. The intent is to update every frame unless you know the point is obscured. If you use a predict mode and know there are obstructions, it keeps the reference area from updating if the point is completely obscured.
update if below reference tolerance This updates the reference sample from the previous frame if the correlation is below the referenceTolerance. This is basically saying "If I can't get a good match, then resample." This is excellent for gradual perspective and scale shifts in the tracking area.

failureTolerance If the correlation of a track falls below this value, it initiates the failureBehavior.
failureBehavior

What occurs when the correlation drops below the failureTolerance:

stop The tracker stops if the correlation is less than the failureTolerance. You can also press any button to manually stop tracking.
predict location and create key. If a failure is detected, then the tracker predicts the location of the keyframe based on a vector of the last two keyframess, and continues tracking in the new area.
predict location and don't create key Same as above, but it merely predicts the new search area and does not create new keyframes until a high correlation is obtained. This is excellent for tracked objects that pass behind foreground objects.
don't predict location In this case, the tracker merely sits in the same spot looking for new samples. New keys are not created.
use existing key to predict location This setting allows you to manually create keys along your track path. You then return to the start frame and start tracking. The search pattern starts looking where the pre-existing motion path is.
limitProcessing This creates a Domain of Definition (DOD) around the bounding boxes of all active trackers. Only that portion of the image is loaded from disk when tracking, so the track is faster. This has no effect on the final output image.
trackNName The name of the track. To change the name, click in the text field.
trackNX/Y The actual track point in X and Y. This is what you would use if you were linking a parameter to a track point.
trackNCorrelation The correlation value of that key to the original sample. A score of 1 is a perfect score. 0 is a very, very, very, very bad score.
trackNWindow Parameters These multiple parameters control the windowing of the tracking box, and are not relevant to exported values.

 



Tracking Right Mouse Menu (in a trackName text field)

Click the Right mouse button in the text field of a trackName to show the special menu to manipulate your tracks:


Function
Notes
Copy/Paste
These are the standard copy and paste commands.
Load Track Pops up a list of all currently existing tracks. Select one, and a copy of that track loads into both the X and Y parameters of the current tracker.
Link Track Pops up a list of all currently existing tracks. Select one, and a link is made from the current tracker to the tracker you select in the pop-up list. Therefore, if you modify the tracker selected with the Link command, the current tracker is updated. If you delete the tracker you linked to, you loose your tracking data.
Average Tracks Displays a list of all tracks. Select up to four tracks that you want to average together, and press OK. An expression is entered into the current trackX and Y fields linking back to your averaged tracks. Note that you cannot choose to average your current track - it is deleted by this action.
Smooth Tracks Displays a slider to execute a blur function on your tracking curves, with the smooth value being the number of keys considered into the smoothing operation. To view the curves themselves, open the trackName subtree and click the load parameter button to load the parameters into the Curve Editor. If you use Smoothing and Averaging operations as your standard workflow, it is recommended to generate your tracks first with the Tracker, and then link them into a MatchMove or a Stabilize.
Load Track File Loads a Shake-formatted track file from disk. See the end of the tracking overview for the format of a track file.
Save Track File Saves a Shake-formatted track file to disk.
Clear Track Resets the current tracker. To reset the entire function, right-click an empty area of the parameters tab and select Reset All Values.



How a Tracker Works

The tracker works by taking a small snapshot of pixels from one frame, called the reference pattern. This is represented by the inner box on the onscreen tracker. The tracker then advances to the next frame, and samples the area inside the larger tracker box, called the search region. When sampling inside the search region, the tracker positions a box the same size as the reference pattern at the pixel in the first row, first column, and takes a sample. It then advances to the next pixel (or subpixel) column in the search region and takes a second sample. For every sample the tracker takes, it assigns a correlation value by comparing it to the reference pattern. When it has taken all of the samples, it assigns the new tracking point to the sample with the highest correlation value. It repeats this on the following frames until the track range is complete.

The referenceBehavior controls if and when the reference pattern is updated. By default, the reference pattern is the set to use start frame, meaning the first frame you start tracking, so even the last frame compares the samples to the first frame. Other modes can change this behavior.

 

The amount of samples taken in the search region is determined by the subPixelResolution parameter. A subPixelResolution of 1 positions the reference pattern box at every pixel to find a sample. This isn't accurate, because most movement occurs at the subpixel level – the movement is more subtle than one pixel across, and therefore gets factored in with other objects in the calculation of that pixel's color. The next resolution down in subPixelResolution, 1/4, advances the reference pattern box in .25 pixel increments, and so is therefore more accurate. The example here is a theoretical 1/2 resolution since the pattern is advanced in .5 pixel increments. Keep in mind that the lower the number, the more samples it takes. At 1/4 resolution, it takes 16 times more samples per pixel than at a resolution of 1. At 1/64, it takes 4096 times more samples per pixel.

 

It is because of this that most trackers don't handle significant rotational movement very well – they (Shake's included) only test for panning changes, not rotational. If they did did, it would have to multiply the amount of panning samples by the amount of degrees for the number of samples to take, which would be prohibitively costly at this stage. If you are faced with a situation with rotational movement, try using a referenceBehavior set to update every frame. This means that the reference pattern is updated at every frame, so you are only comparing a frame with the frame before it, and not the first frame.

Also keep in mind that manual adjustments are standard for tracking problems.



Tracking Strategies

Despite all of the demos we love to see (and show), tracking is rarely a magic bullet that works on the first attempt. These are some strategies to help you get accurate tracks.


Pick a Good Reference Pattern

The ideal reference pattern is one that doesn't change perspective, scale or rotation, and does not go offscreen or get obscured by other objects. It doesn't change overall brightness or color, and is very high contrast. It also is distinct from other patterns in the same neighborhood. Meanwhile, in the real world, we have to contend with all of these factors in our footage (think about that next time you watch a tracking demo, by the way).

In this example, there is the corner at area A, or anywhere along the line near B. Area A is the better choice, since B can be matched anywhere along the horizontal black line, and probably on the dark line that is below the letter B. One rule is to avoid similar patterns horizontally or vertically from your candidate pattern. If you can easily find one, so can the tracker.


 

Another potential candidate in the above example is the word Alivia in the sign. However, as the clip advances, the text becomes an indecipherable blur. The A and B points also move closer together – the clip has significant scaling on the X axis and some scaling in the Y due to the perspective shift. Although the overall brightness has dropped, contrast remains relatively high at the A point. This is a good candidate for tracking with referenceBehavior set to update if below reference tolerance or update every frame.

 

The reference sample should also be relatively constant and unique over time as well. Flickering lights would be bad for example, but if they were regular enough, you could try setting your trackRange to match the flicker, that is, 1-5, 10-15, 20-25, and so on. Granted, this is awkward. A better solution is to set your failureBehavior to don't create key or predict position and don't create key.

The following example shows a track marker placed on a TV screen so the client could place some silly tripe on the TV. The default tracker reference pattern is unnecessarily large. The only interesting detail is the black cross in the middle. Otherwise, most of the pattern matches up with most of the rest of the image – green. To adjust for this, limit the reference pattern to more closely match the black crosshair.

 

Pick a Good Search Region

You should also suit your search region to match your movement and the patterns near your reference pattern. The bus example doesn't track the lower left corner of the sign very well with the default settings because the middle black horizontal line can easily be matched up with the black line at the very base of the search region in later frames of the clip. This is because the X axis is squeezed so much that vertical details disappear. Additionally, since the bus is moving to the right, there is no point in wasting cycles to the left of the point. Remember, the larger the search region, the more samples the tracker has to take.

 

The corrected search region is now high enough to not include the lower black line, and extends minimally to the left.

 

Manually Coax Your Track

Another technique you can use is to manually insert keyframes of your track. For example, if you have 100 frames to track, you might put in a keyframe every 5 or 10 frames with the AutoKey feature. A trick to help you to do this is to set an increment of 5 or 10 in the Time Bar. Press the Left or Right arrow key to jump by the increment amount.

Once you keyframes are manually entered, return to frame 1, and put your failureBehavior on use existing keys to predict location. The tracker searches along the tracker's pre-existing motion path to find matching patterns.


Identify the Channel with the Highest Contrast

The tracker works best with a high contrast reference pattern. The human eye sees contrast as represented by value. However, you may sometimes have higher contrast in saturation or hue, so switch over to a different color space with the matchSpace parameter. A shot may also have a higher contrast in a specific RGB channel than in the others, for example the blue channel might have a bigger range than the red or green. In such a case, apply a Color - Reorder bbb to the image, and then track with luminance as your matchSpace.


Delog your Logarithmic Cineon files

Because the log to lin conversion increases contrast, you may have better results on linear data than on log data. Of course, maybe not. Nice.


Don't Reduce Image Quality

Ideally, you should track an image with the most amount of raw data. This means if you put a Brightness of .5 on your image, you have lost half of the color information. Therefore, track the image before the Brightness operator is applied.

 

Don't Track Proxies

Don't ever track with proxy settings. These are bad for two reasons. The first is that you are filtering the image so detail is lost. The second reason is that you are automatically throwing data away because of data round-off. For example, if you are using 1/4 proxy, you are automatically throwing away four pixels of data in any direction, which means an 8x8 grid of potential inaccuracy.

 

Don't Track the Image at its Best Quality

Yup, that's right, the exact opposite of what was just stated above. Isn't compositing fun? It often pays off to apply a Color - Monochrome node onto your image and drop the blue channel out if you have particularly grainy footage. Another good strategy is to apply a Filter - Blur to one or more channels to get rid of grain irregularities.

In some cases, you may want to modify your images to improve contrast in the reference pattern, either with a ContrastLum or ContrastRGB. Since you use this image to generate tracks, you are not obliged to keep the contrast image for the rest of your composite.


Tracking Images with Perspective, Scale, or Rotational Shifts

For images with significant change in size and angle, you can try two different referenceBehaviorsupdate if below reference tolerance or update every frame. The second one is the more drastic choice because you get an inherent accumulation of tiny errors if you update every frame. Therefore, try update if below reference tolerance first.

Another strategy is to jump to the midpoint frame of the clip and track forwards to the end frame of the clip. Then return to the midpoint frame and track backwards to the beginning of the clip.

A second strategy is to apply two Stabilize nodes. The first can be considered a rough stabilize. The second stabilize then works off of the first, and will therefore have a much better chance of finding acceptable patterns. Since the Stabilize nodes concatenate, no quality is lost.


Tracking Points that Get Obscured or Go Off-Frame.

There are two basic techniques to correct track points that are obscured by moving offscreen or an object passing in front of them.

The first strategy is to use a different failureBehavior, either predict location and create key or predict location and don't create key. The first setting is good for nice linear behavior – it continues to lay down keyframes following the vector of the last two valid keyframes it made, so the two frames prior to the pattern become obscured. It is excellent for points that go offscreen and never reappear. The second setting is a little nicer for patterns that reappear because it continues to search on a vector, but only creates a keyframe if it finds another acceptable pattern. You have an even interpolation between the frame before the pattern was obscured, and the frame after it is revealed again.

The above strategies only work if you have nice linear movement. The second strategy is to use the Offset button on the Viewer. When offset, you can split the reference pattern and search region away from the track point. In this example, the track is obscured by a lamp post, so the search region (not the point, just the region) is moved to a nearby pattern and tracking continues until the original pattern reappears. Even though one region is examined, the points get saved in another. The second tracking pattern should travel in the same direction as your original pattern.

 


Modifying Tracks

A track can modified in several ways to massage the data. You can manually modify a track in the Viewer or in the Curve Editor, average tracks together, smooth tracks to remove noise, or remove jitter to smooth out a camera movement.


Manually modifying tracks

At any time, enable the Autokey button in the Viewer to manually adjust a tracking point onscreen. You can use the + and - keys (next to the Backspace key) to zoom in and out. The zooming follows the cursor, so place the cursor on the key point in the Viewer and zoom in. Press the Home key or the Home button to return you to your normal view.

You can also adjust a tracking curve in the Curve Editor. Open the trackName subtree and click on the clock icons to load a parameter into the Editor. In this example, track1X (only the X parameter) is loaded into the Editor.



Averaging Tracks Together

A common technique is to track forward from the first frame to the last, create a second track and track backwards from the last frame to the first. These two tracks are then averaged together to (hopefully) derive a more accurate track. If you plan to use this method, it is recommend to use the Tracker node, and then load your tracks into Stabilize or MatchMove with the Load or Link Track functions (they haven't moved from the right mouse menu of trackName text field).

To Average Tracks:


This works by creating an expression in both the track3X and track3Y parameters. The expression for the X parameter looks like this:

(Tracker1.track2X+Tracker1.track1X)/2

Because these are linked to track1 and track2 on the Tracker1 node, don't delete these. For more information on linking, see below.

You can average up to four tracks at one time, but you can of course continue to manipulate your tracks with further functions, including Average Tracks.


Smoothing Track Curves

You can smooth a track with the Smooth Tracks function in the Tracker Parameters. Prior to smoothing the curve, you may want to copy the track to another tracker with the Load Track function on the second tracker.

To Smooth a Track Curve:

The default is 5, which means that 5 track points centered on the currently evaluated point are used to compute the current point's new, smoothed, value. This is a standard Gaussian (bell-curve type) filter. In other words, if you leave it at 5, when it computes the value of frame 12, it takes into account frames 10, 11, 12, 13, and 14. If you set it to 3, it uses 11, 12, and 13. The larger the smoothValue, the more points are taken into account (and thus more calculations done) for every point in the curve. Even values for smoothValue use the next largest odd number of frames, but the end ones don't contribute as much.

As an example, here is a noisy track curve prior to smoothing:

 

 

After the track curve is smoothed:

 

:

Linking to Tracking Data

Referencing track point data works similarly to referencing any other parameter within Shake. The twist here is that since you can rename the track point, you can change the name of the parameter to which you are referring. For example, if you have a Tracker node named Tracker1 and you've left a track point at its default name "track1." To reference the X track data use: Tracker1.track1X. If you change the name of the track point to "lowerleft", then the reference is changed to Tracker1.lowerleftX. This applies to the Y data as well.

You can also use the Link Track button to link one track to another track, simultaneously linking both the X and Y curves simultaneously.

 

Removing Jitter on a Camera Move

The following technique is useful when you have a camera move that you want to keep, but there is a lot of jitter in the shot. You need to stabilize the shot, but only by the small amount that is the actual jitter. To do this, combine the techniques mentioned above.







 

Tracking File format

A sample saved track file for use with Save or Load Track File.

TrackName track1
   Frame             X             Y   Correlation
    1.00       462.000       210.000         1.000 
    2.00       405.000       192.000         1.000 
    ...etc...


This is different from Load Expression which is available on the right mouse menu for any text field, not just trackers. That command looks for preformatted Shake expressions. For example, to load the above information into a Move2D, you have to load two files, one for xPan and one for yPan. Their formats are something like this:

Linear(0,0462@1, 405@2, ....)

and

Linear(0,210@1, 192@2, ...)