About Tracking, Stabilizing and MatchMoving

This discusses tracking in depth, listing interface features, workflow issues, as well as tips on getting a successful track and manipulating the data. For specific formats of each node, jump to the appropriate reference page. For a tutorial on how to use the trackers, jump to the Tracking Tutorial.

Shake has three tracking nodes, each of which can do their own tracking, or refer to tracks made by other tracking nodes:


Tracking Workflow

This is a general overview of the steps you take to generate a track. Each of these is detailed in the following sections.

General Tracking Workflow:

  1. Play through your background clip several times to determine a good tracking point.
  2. Attach your tracking (Transform - Tracker, Stabilize, or MatchMove) node to the background clip.
  3. Make sure the onscreen controls are visible in the Viewer
  4. Go to the frame you want to start tracking at.
  5. Position the tracker at the point you want to track and adjust the reference pattern and the search region.
  6. Make sure the trackRange parameter reflects the range you want. For FileIns, it will take the range of the clip. For Shake-generated elements (i.e., wobbly Text), you will have to provide a range, for example, 1-50.
  7. Hit either the backward or forward track button . This will start the tracking processing, which ideally needs to be done once. After it is finished, you have tracking animation curves.

Stabilize additions:

  1. Toggle the applyTransform parameter to on to stabilize the plate.
  2. If you are using Stabilize to do match-moving, hit the inverseTransform button.

MatchMove additions:

  1. Attach the Foreground element to the second input of the MatchMove.
  2. Select the outputType parameter to be Over (or another compositing operation).
  3. Toggle the applyTransform parameter to on to match the motion.
  4. If you are doing 4 point tracking and you want to attach four arbitrary points on the FG image to the Background, hit the BG/FG button to toggle it to FG . You can then place the four corners on the Foreground element. These represent the corners that get plugged into the four tracking points.
  5. Toggle back to BG .
  6. You may have to switch the outputType parameter back to Over, or whatever compositing mode you were using.

 

Tracking Onscreen controls:

All tracking nodes share the following interface features:

A tracker consists of three onscreen controls: The search region, the reference pattern, and the actual track point. The outer box is the search region, the inner box is the reference pattern. The cross hair is the track point.

To move the tracker, click anywhere on a blank spot inside of the tracking region, or on the track point and drag. To resize either the tracking region or the sample pattern, grab a corner and the boxes will uniformly scale in X or Y.

The larger the search region, the slower the track will be.

If you grab an edge of the search region, you can scale it non-uniformly. This is good for "leading" the track point. For example, in the bus clip in the tutorial project, the bus moves to the right. Therefore, it doesn't make sense to scan to the left of the current track pattern for a pattern match.

In the case of 4 point MatchMove and Stabilize, the trackers should be positioned in a counterclockwise order starting at the lower left corner. This will ensure the proper alignment of your element when the transformation is applied.

If you want to offset the tracking area from where the keyframe is actually saved (for example the tracking point goes offscreen or gets obscured), toggle the Offset Tracker button.. This will allow you to move the search pattern without moving the track key. This is good for when something goes offscreen, so you grab something else to keep tracking on. To reset the search area back to the key, use the Reset Tracker button:

You can turn off a specific tracker by toggling its visibility by the track name inside the track node parameter list, or by turning off the display of all onscreen controls . All visible trackers will be processing when you hit the track button, regardless of whether they have previously tracked keys or not.

One limitation with MatchMove and Stabilize is that they will not handle the pass-through of upstream onscreen controls, so those controls will be in their un-transformed state.


Tracking Viewer Buttons:

Button
Function

Track Backwards/Forwards. Hitting either of these buttons will start the tracking process. All visible trackers will try to lay down new tracking keys. The tracking will continue until one of the trackRange frame limits is reached, either the upper limit if you are tracking forwards , or the lower limit if you are tracking backwards or until your correlation falls below your failureTolerance.

Offset Track. The track search area and the tracking point are linked. If you move one, the other follows.

Offset Track. The track search area and the tracking point are offset from each other. If your original sample pattern gets obscured, try offsetting the search box to a different area. Your keys will still be saved in the original spot.

Reset Track. Once you have offset a track search region, you can return it back to the track point by hitting this button.


Track Display. These buttons display all trackers, curves and keyframes, just the trackers and keyframes, or just the trackers. You can also control visibility of individual trackers as described below.

These buttons appear on the MatchMove node when doing 4 point tracking. They allow you to toggle between adjusting the track points on the Background (BG) or the corners of the Foreground (FG) that get pinned to the track points.

Beside each track name is a Color Picker and a Visibility toggle. Press the Color Picker to select a different color for the display of that tracker. The Visibility toggle will turn on and off the display of a tracker. If it is on, it will re-track each time you hit the Track button. If the tracker is invisible, it will not be processed next time you hit the Track buttons.




Tracking Parameters

All trackers share the following parameters:

Parameters
Type
Default
trackRange
string
1
This is the potential frame range limit of your tracking. If you read in a clip, the range will be set to the clip range. For generated elements like RGrad, it will take a range of 1. You can set new limits using Shake's standard range description, i.e., 10-30x2. If you stop tracking and start up again, it will start from the current frame until it reaches the lower or upper limit of your trackRange, depending on if you are tracking forwards or backwards.

subPixelResolution string 1/16

The resolution of your track. The smaller the number, the more precise your tracking will become.

Possible values:

1 area is sampled at every pixel. Not very accurate or smooth, but very fast.
1/4 area is sampled at every .25 pixels, i.e., 16 times more than with a sampling of 1.
1/16 area is sampled at every .0625 pixels, i.e., 256 times more than with a sampling of 1.
1/32 area is sampled at every .03125 pixels, i.e., 1024 times more than with a sampling of 1.
1/64 area is sampled at every .015625 pixels, i.e., 4096 times more than with a sampling of 1.
matchSpace string luminance
The pixels are matched according to the correlation between the selected color space, either luminance, hue, or saturation. If an image has roughly the same luminance, but contrasting hues, the user would switch over to hue-based tracking.

referenceTolerance float .75
A tracking correlation of 1 is a perfect score, meaning there is an exact match between the original reference frame and the sampled area. By lowering the referenceTolerance, you are accepting greater inaccuracy in your track. If tracked keys are between the referenceTolerance and the failureTolerance, they will be highlighted in the Viewer. Also, in some cases, referenceBehavior will be triggered if the tracking correlation is below the referenceTolerance.

referenceBehavior string use start frame

This behavior dictates what reference sample a tracking area should reference. By default, the reference pattern is the first frame at which you started the track, not necessarily the first frame of the trackRange. The last two behaviors measure the tracking correlation and match it to the referenceTolerance to decide an action.

use start frame
The new samples are compared to the reference pattern from the first frame at which you started tracking. If you stop tracking midway, and start up again at a later frame, the later frame will be used as the reference sample.

update every frame
The source sample is updated from the previous frame. This will usually give you an inherent drift in your track, as tiny errors will accumulate. This method is for movements that have drastic changes in perspective and scale.

update from keyframes
If you are using a failureBehavior of predict location and don't create keys or don't predict location, a keyframe will not necessarily be saved every frame. In this case, you may only want to update from the last frame that had a valid keyframe.

update if above reference tolerance
This will update the reference sample from the previous frame if the correlation is above the referenceTolerance. The intent of this is update every frame unless you know the point is obscured. If you have a predict mode and know there are obstructions, it keeps the ref area from updating if the point is completely obscured.

update if below reference tolerance
This will update the reference sample from the previous frame if the correlation is below the referenceTolerance. This is basically saying "If I can't get a good match, then resample". This is excellent for gradual perspective and scale shifts in your tracking area.

failureTolerance float .5
If the correlation of a track falls below this value, it will initiate the failureBehavior.

failureBehavior string stop

stop
The tracker stops if the correlation is less than the failureTolerance. You can also hit Esc to manually stop tracking.

predict location and create key.
If a failure is detected, then the tracker predicts the location of the key based a vector of the last two keys, and continues tracking in the new area.

predict location and don't create key.
As above, but new keys are not created, it merely predicts the new search area. This is excellent for tracked objects passing behind foreground objects.

don't predict location
In this case, the tracker merely sits in the same spot looking for new samples. New keys are not created.

use existing key to predict location
This setting allows you to manually create keys along your track path. You then go back to the beginning to track and start tracking, and the search pattern will start looking at where the pre-existing motion path is.

limitProcessing int 1
This will create a Domain of Definition (DOD) of the bounding box of all active trackers. Only that portion of the image will be loaded from disk when tracking, therefore it will go more quickly. This has no effect on the final output image.

trackNName string trackN
The name of the track. You can change this.

trackNX/Y float NA
The actual track point in X and Y. This is what you would use if you were linking a parameter to a track point.

trackNCorrelation float NA
The correlation value of that key to the original sample. A score of 1 is a perfect score. 0 is a very very very very bad score.

trackNWindow Parameters float NA
These multiple parameters control the windowing of the tracking box, and are not relevant to exported values.



Tracking Right Mouse menu (in a trackName textfield)

In addition, if you click the right mouse in the textfield of a trackName, you will get a special menu to manipulate your tracks:

Function
Notes
Copy/Paste
These are the standard text editing Copy and Paste commands.

Load Track This will pop up a list of all currently existing tracks. Select one, and a copy of that track will loaded into both the X and Y parameters of the current tracker.

Link This will pop up a list of all currently existing tracks. Select one, and a link will be made from the current tracker to the one you have just selected. Therefore, if you modify the tracker you selected with the Link command, the current tracker will be updated. If you delete the tracker you linked to, you will of course loose your tracking data.

Average Tracks When this is called up, you get a list of all tracks. Select up to four tracks that you want to average together, and hit OK. An expression will be entered into the current trackX and Y fields that links back to your averaged tracks. Note that you cannot choose to average your current track - it will in fact be deleted by this action. See below for using this function.

Smooth Tracks This will present a slider to execute a blurring function on your tracking curves, with the number being the number of keys considered into the smoothing operation. To see the curves themselves, open the trackName subtree, and load the trackNX and Y parameters into the Curve Editor. If you use Smoothing and Averaging operations as your standard workflow, we recommend generating your tracks first with the Tracker, and then linking them into a MatchMove or a Stabilize. See below for using this function.

Load Track File This will load a Shake-formatted track file from disk. See below for a format of a track file.

Save Track File This will save a Shake-formatted track file to disk.

Clear Track This will reset the current tracker. To reset the entire function, go to the top part of the parameters page and select with the right mouse the Reset Values function.



How a Tracker Works

The tracker works by taking a small snapshot of pixels from one frame, which we call the reference pattern. This is represented by the inner box on the onscreen tracker. The tracker then advances to the next frame, and samples everywhere inside the larger tracker box, which we call the search region. When sampling inside the search region, it positions a box the same size as the reference pattern at the pixel in the first row, first column, taking a sample. It then advances to the next pixel (or subpixel) column in the search region and takes a second sample. For every sample it takes, it assigns a correlation value by comparing it to the reference pattern. When it has taken all of the samples, it assigns the new tracking point to the sample with the highest correlation value. It then repeats this on the following frames until the tracking is complete.

Your referenceBehavior controls if and when the reference pattern gets updated. By default, the reference pattern is the set to use start frame, meaning the first frame you start tracking at, so even the last frame will compare the samples to the first frame. Other modes can change this behavior.

 


The amount of samples taken in the search region is determined by the subPixelResolution parameter. If you have a resolution of 1 (not very accurate), it positions the reference pattern box at every pixel to find a sample. This isn't accurate, because most movement occurs at the subpixel level, i.e., the movement is less subtle than one pixel across, and therefore gets factored in with other objects in the calculation of that pixel's color. The next resolution down in subPixelResolution, 1/4, advances the reference pattern box in .25 pixel increments, and so is therefore more accurate. The example here is a theoretical 1/2 resolution since the pattern is advanced in .5 pixel increments. Keep in mind that the lower the number, the more samples it takes. At 1/4 resolution, it takes 16 times more samples per pixel than at a resolution of 1. At 1/64, it takes 4096 times more samples per pixel.

It is because of this that most trackers don't handle significant rotational movement very well - they (Shake's included) only test for panning changes, not rotational. If you did, you would have to multiply the amount of panning samples by the amount of degrees, which would be prohibitively costly at this stage. If you are faced with a situation with rotational movement, try using a referenceBehavior of update every frame. This means that the reference pattern is updated at every frame, so you are only comparing a frame with the frame before it, and not the first frame.




Tracking Strategies

Despite all of the demos we love to see (and show), tracking is rarely a magic bullet that works on the first attempt. These are some strategies to help you get accurate tracks.


Pick a good reference pattern

The ideal pattern is one that doesn't change perspective, scale or rotation, and does not go offscreen or get obscured by other objects. It doesn't change overall brightness or color, and is very high contrast. It also is distinct from other patterns in the same neighborhood. Meanwhile, in the real world, we have to contend with all of these factors in our footage. Think about that next time you watch a tracking demo, by the way.

In this example, we have the corner at area A, or anywhere along the line near B. A would be the better choice, since B can be matched anywhere along the horizontal black line, and probably on the dark line that is below the letter B. Therefore, one principle is to avoid similar patterns horizontally or vertically from your candidate pattern. If you can easily find one, so can the tracker.

 

Another potential candidate is the word Alivia in the sign. However, as we examine the clip, you can see that the text becomes an indecipherable blur. We can also see how close our A and B points get together- the clip has significant scaling on the X axis and some scaling in the Y due to the perspective shift. Although the overall brightness has dropped, our contrast remains relatively high at the A point. This would be a good candidate for tracking with referenceBehavior of update if below reference tolerance or update every frame.

The reference sample should also be relatively constant and unique over time as well. Flickering lights would be bad for example, but if they were regular enough, you could try setting your trackRange to match the flicker, i.e., 1-5, 10-15, 20-25, etc.

 

 

This example shows a track marker that was placed on a TV screen so the client could place whatever they wanted on TV in the shot. The default tracker reference pattern is unnecessarily too large. The only interesting detail is the black cross in the middle. Otherwise, most of the pattern matches up with most of the rest of the image - green.

To adjust for this, we limit the reference pattern to more closely match the black crosshair.




Pick a good search region

You should also suit your search region to match both your movement, and the patterns near your reference pattern. The bus example doesn't track the lower left corner of the sign very well with the default settings because the middle black horizontal line can easily be matched up with the black line at the very base of the search region in later frames of the clip. This is because the X axis is squeezed so much that vertical details disappear. Additionally, since the bus is moving to the right, there is no point in wasting cycles to the left of the point. Remember, the larger the search region, the more samples it has to take.


This image shows a corrected search region, which is now high enough to not include the lower black line, and extends minimally to the left.


Manually coax your track

Another technique you can use is to manually insert keyframes of your track. For example, if you have 100 frames to track, you might put in a key every 5 or 10 frames by using the AutoKey feature. A trick to help you to do this is to put an increment of 5 or 10 in the Time Bar. When you hit the left or right arrow key, you will jump by the increment amount.

Once you are done manually putting in your keys, go back to frame 1, and put your failureBehavior on use existing keys to predict location. This means the tracker will search along the tracker's pre-existing motion path to find matching patterns.


Identify the channel with the highest contrast

The tracker works best with a high contrast reference pattern. To the human eye, we see contrast as represented by value. However, you may sometimes have higher contrast in saturation or hue, so switch over to a different color space with the matchSpace parameter. A shot may also have a higher contrast in a specific RGB channel than in the others, for example the blue channel might have a bigger range than the red or green. In that case, you could put a Color - Reorder bbb on the image, and then track with luminance as your matchSpace.

Delog your logarithmic Cineon files

Because the log to lin conversion increases contrast, you will probably have better results on linear data than on log data.


Track the image at its best quality

Ideally, you should be tracking an image with the most amount of raw data. This means if you put a Brightness of .5 on your image, you have lost half of the color information. Therefore, track the image before the Brightness operator is applied.

Be careful with proxyScale and proxyRatio settings. These are generally bad for two reasons. The first is that you are filtering the image which means detail is lost. The second reason is that you are automatically throwing data away. For example, if you are using 1/4 proxy, you are automatically throwing away four pixels of data in any direction, which means an 8x8 grid of potential inaccuracy.

In some cases, you may in fact want to modify your images to improve contrast in the reference pattern, either with a ContrastLum or ContrastRGB. Since you are just using this image to generate tracks, you are not obliged to keep the contrast image for the rest of your composite.

When working with grainy film footage, a good strategy is to strip out the red or green channel (usually green as it has the highest luminance value) because the blue channel holds the most grain and is therefore likely to disturb its pattern every frame. You also might try adding a Blur of a few pixels or a Median filter to reduce the grain before you track.

Finally, sometimes you may be having problems with random film grain being too severe, and your reference pattern becomes useless. Apply a Filter - Median or Blur before you track to reduce the effects of grain. Yes, I know this contradicts the second suggestion. Welcome to the exciting world of tracking.


Tracking images with perspective, scale, or rotational shifts

If your image has significant changes in size and angle, you should try two different referenceBehaviors, update if below reference tolerance or update every frame. The second one is the more drastic choice because you will get an inherent accumulation of tiny errors if you update every frame. Therefore, try update if below reference tolerance first.

Another strategy is to jump to the midpoint frame of the clip and track forwards to the end frame of the clip. Then return to the midpoint frame and track backwards to the beginning of the clip. See below on more techniques to refine and average your tracks.

A second strategy is to do two Stabilize nodes. The first can be considered a rough stabilize. The second stabilize then works off of that, and will therefore have a much better chance of finding acceptable patterns. Since the Stabilize nodes will concatenate, no quality is lost.


Tracking points that get obscured or go off-frame.

There are two basic techniques for correcting for points that get obscured by either going offscreen or by having an object passing in front of them.

The first strategy is to use different failureBehavior, either predict location and create key or predict location and don't create key. The first is good for nice linear behavior, as it will continue to lay down keys following the vector of the last two valid keys it made, meaning the two frames prior to the pattern becoming obscured. It is excellent for points that go offscreen and never reappear. The second setting is a little nicer for patterns that reappear because it will continue to search on a vector, but will only create a key if it finds another acceptable pattern. You will have an even interpolation between the frame before the pattern was obscured, and the frame after it is revealed again.

These only work if you have nice linear movement. The second strategy is to use the Offset button on the Viewer. When these are offset, you can split the reference pattern and search region away from the track point. In this example, our point is obscured by a lamp post, so we switch the search region (not the point, just the region) to a nearby pattern and continue tracking until our original pattern reappears. Even though we are examining one region, the points get saved in another. The second tracking pattern should be traveling in the same direction as your original pattern.


Modifying Tracks

Once you have a track, you can modify it in several ways to massage the data. You can manually modify a track either in the Viewer or in the Curve Editor, you can average tracks together, and you can smooth tracks to remove noise. There is also a trick to stabilizing a camera move that keeps the move but removes the jitter.


Manually modifying tracks

At any time, you can turn on the Autokey button in the Viewer and manually adjust a tracking point by simply grabbing it and putting it where you need. You can use the + and - keys by the Backspace key to zoom in and out. The zooming follows the cursor, so just place the cursor on the key point in the Viewer and zoom in. Hitting the Home key or the Home button will return you to your normal view.

You can also adjust a tracking curve in the Curve Editor. Go to the trackName and open the subtree. Click on the clock icons to load a parameter into the Editor. In this example, track1X is loaded into the Editor. We have also turned on autokey so that we can numerically alter the track by hand.



Averaging Tracks together

A common technique is to track forwards from the first frame to the last, and then create a second track, tracking backwards from the last frame to the first. These two tracks are then averaged together to hopefully derive a more accurate track. If you are going to be using this method, we recommend you track with the Tracker node, and then load your tracks into Stabilize or MatchMove using the Load or Link Track functions (they haven't moved from the right mouse menu of trackName textfield).

As an example of how to average tracks, create a Tracker, and create two extra trackers for a total of three. Create tracks on track1 and track2. Then hit the right mouse menu on track3 and select Average Tracks. This will pop up a window giving you up to four input trackers to average together. Select Tracker1.track1 and Tracker1.track2 in the first two popups, leaving the last two at none. Hit OK, and you will see that track3 is midway between the first two tracks.


This works by creating an expression in both the track3X and track3Y parameters. The expression for the X parameter looks like this:

(Tracker1.track2X+Tracker1.track1X)/2

Since these are linked back to track1 and track2 on the Tracker1 node, you shouldn't delete these. For more information on linking, see below.

You can average up to four tracks at once, but you can of course continue to manipulate your tracks with further functions, including Average Tracks.

Note that the Smoothing and Averaging tools that are available within Tracker/Stabilize/Matchmove are provided as somewhat of an interim solution for the 2.2 release. The eventual plan is to have this sort of functionality moved directly into the Curve Editor, with a much greater degree of control over the degree and extent of smoothing (see below) that is applied. It is hoped that the current version will be at least somewhat useful, even though the feature is fairly limited at this point.


Smoothing Track Curves

You can smooth a track with the Smooth Tracks function in the right-mouse menu. Again, this is a temporary workaround until our next release, but here is how it works for now.

Select the track you want to smooth. Although you can of course Undo, you might want to copy the track to another tracker using the Load Track function on the second tracker. Select Smooth Tracks, and a window will appear, asking you for a smoothValue parameter.

The default is five, which means that 5 points centered on the current one will be used to compute the current one's new, smoothed, value. This is a standard Gaussian (bell-curve type) filter. In other words, if you leave it at 5, when it computes the value of frame 12, it will take into account frames 10, 11, 12, 13, and 14. If you set it to 3, it will use 11, 12, and 13. For frame 13, it would use 11, 12, 13, 14, 15 for frame 13 (smoothValue = 5) and 12, 13, 14 for frame 13 (smoothValue=3). Just a warning: in this beta there isn't any sign that Shake is working, but just have patience for a few minutes, although you can load the curves into the Curve Editor. The larger the smoothValue the more points are taken into account (and thus more calculations done) for every point in the curve. Even values for smoothValue use the next largest odd number of frames but the end ones don't contribute as much.

As an example, here I have a noisy track:

After the smoothing, it looks like this

:

Linking to Tracking Data

Referencing track point data works similarly to referencing any other parameter within Shake. The twist here is that since you can rename the track point, you can change the name of the parameter to which you are referring. For example, say I have a Tracker node named Tracker1 and I've left a track point at its default name "track1". To reference the X track data I would use: Tracker1.track1X. If I change the name of the track point to "lowerleft", then the reference would be to Tracker1.lowerleftX. This follows for the Y data as well.

An important difference with standard linking: Since the MatchMove and Stabilize nodes use a Tracker internally, I must reference their tracking data with the assumption that there is a tracker inside of it. As an example, lets say I've named the track point "upper_right". I would use MatchMove1.tracker.upper_rightX. Similarly, to get the Y value of a track point named "bottom" in a Stabilize node named Stabilize5, I would use Stabilize5.tracker.bottomY. Here are some examples in context:

 

You can copy or link to any tracker with any other tracker by using Load or Link Track. You can also link to them using Shake's standard expressions.Open the subtree on a trackName, you will see the X and Y coordinates listed out. Therefore, to link to them, you use NodeName.tracker.tracknameX and Y

 

For example, here I have linked the Scroll node to a tracker so that I don't loose any data. If the Tracker node for the above illustration was called Tracker1, this is how you would link to the X and Y parameters.

Tracker1.tracker.track1X 

If I want to invert the transformation, i.e., turn 1 point tracking data into stabilization data, negate the expression:

-Tracker1.tracker.track1X 
         

For 2 point MatchMove and Stabilize, Shake will derive the scaling factor and rotation angle. They can be viewed under the applyScale and applyRotation subtrees, and of course linked to as well.

        

Removing Jitter on a Camera Move

This technique is useful when you have a camera move that you want to keep, but there is a lot of jitter in the shot. You therefore need to stabilize the shot, but only by the small amount that is the actual jitter. To do this, we combine the above techniques.

 

Tracking File format

A sample saved track file for use with Save or Load Track File.

TrackName track1
   Frame             X             Y   Correlation
    1.00       462.000       210.000         1.000 
    2.00       405.000       192.000         1.000 
    ...etc...


This is different from Load Expression which is available on the right mouse menu for any textfield, not just trackers. That command looks for preformatted Shake expressions, i.e., to load the above information into a Move2D, you would have to load two files, one for xPan and one for yPan. Their formats would be something like this:

Linear(0,0462@1, 405@2, ....)

and

Linear(0,210@1, 192@2, ...)