About Tracking, Stabilizing and MatchMoving |
This discusses tracking in depth, listing interface features, workflow issues, as well as tips on getting a successful track and manipulating the data. For specific formats of each node, jump to the appropriate reference page. For a tutorial on how to use the trackers, jump to the Tracking Tutorial.
Shake has three tracking nodes, each of which can do their own tracking, or refer to tracks made by other tracking nodes:
This is a general overview of the steps you take to generate a track. Each of these is detailed in the following sections.
General Tracking Workflow:
Stabilize additions:
MatchMove additions:
All tracking nodes share the following interface features:
A tracker consists of three onscreen controls: The search region, the reference pattern, and the actual track point. The outer box is the search region, the inner box is the reference pattern. The cross hair is the track point. To move the tracker, click anywhere on a blank spot inside of the tracking region, or on the track point and drag. To resize either the tracking region or the sample pattern, grab a corner and the boxes will uniformly scale in X or Y. The larger the search region, the slower the track will be. |
![]() |
If you grab an edge of the search region, you can scale it non-uniformly. This is good for "leading" the track point. For example, in the bus clip in the tutorial project, the bus moves to the right. Therefore, it doesn't make sense to scan to the left of the current track pattern for a pattern match. In the case of 4 point MatchMove and Stabilize, the trackers should be positioned in a counterclockwise order starting at the lower left corner. This will ensure the proper alignment of your element when the transformation is applied. If you want to offset the tracking area from where the keyframe is actually
saved (for example the tracking point goes offscreen or gets obscured),
toggle the Offset Tracker button. You can turn off a specific tracker by toggling its visibility One limitation with MatchMove and Stabilize is that they will not handle the pass-through of upstream onscreen controls, so those controls will be in their un-transformed state. |
![]() |
Button
|
Function |
![]() ![]() |
Track Backwards/Forwards. Hitting either of these buttons
will start the tracking process. All visible trackers will try to
lay down new tracking keys. The tracking will continue until one of
the trackRange frame limits is reached, either the upper limit
if you are tracking forwards |
![]() |
Offset Track. The track search area and the tracking
point are linked. If you move one, the other follows. |
![]() |
Offset Track. The track search area and the tracking
point are offset from each other. If your original sample pattern
gets obscured, try offsetting the search box to a different area.
Your keys will still be saved in the original spot. |
![]() |
Reset Track. Once you have offset a track
search region, you can return it back to the track point by hitting
this button.
|
![]() ![]() ![]() |
Track Display. These buttons display all trackers, curves and keyframes, just the trackers and keyframes, or just the trackers. You can also control visibility of individual trackers as described below. |
![]() ![]() |
These buttons appear on the MatchMove node when doing 4 point tracking. They allow you to toggle between adjusting the track points on the Background (BG) or the corners of the Foreground (FG) that get pinned to the track points. |
![]() |
Beside each track name is a Color Picker and a Visibility toggle.
Press the Color Picker to select a different color for the display
of that tracker. The Visibility toggle will turn on and off the display
of a tracker. If it is on, it will re-track each time you hit the
Track button. If the tracker is invisible, it will not be processed
next time you hit the Track buttons. |
All trackers share the following parameters:
Parameters
|
Type
|
Default | ||||||||||
trackRange |
string
|
1 | ||||||||||
This is the potential frame
range limit of your tracking. If you read in a clip, the range will
be set to the clip range. For generated elements like RGrad,
it will take a range of 1. You can set new limits using Shake's standard
range description, i.e., 10-30x2. If you stop tracking and start up
again, it will start from the current frame until it reaches the lower
or upper limit of your trackRange, depending on if you are tracking
forwards or backwards. |
||||||||||||
subPixelResolution | string | 1/16 | ||||||||||
The resolution of your track. The smaller the number, the more precise your tracking will become. Possible values:
|
||||||||||||
matchSpace | string | luminance | ||||||||||
The pixels are matched according to the correlation
between the selected color space, either luminance, hue,
or saturation. If an image has roughly the same luminance, but
contrasting hues, the user would switch over to hue-based tracking. |
||||||||||||
referenceTolerance | float | .75 | ||||||||||
A tracking correlation of 1 is a perfect
score, meaning there is an exact match between the original reference
frame and the sampled area. By lowering the referenceTolerance,
you are accepting greater inaccuracy in your track. If tracked keys
are between the referenceTolerance and the failureTolerance,
they will be highlighted in the Viewer. Also, in some cases, referenceBehavior
will be triggered if the tracking correlation is below the referenceTolerance.
|
||||||||||||
referenceBehavior | string | use start frame | ||||||||||
This behavior dictates what reference sample a tracking area should
reference. By default, the reference pattern is the first frame at
which you started the track, not necessarily the first frame of the
trackRange. The last two behaviors measure the tracking correlation
and match it to the referenceTolerance to decide an action. update every frame |
||||||||||||
failureTolerance | float | .5 | ||||||||||
If the correlation of a track falls below
this value, it will initiate the failureBehavior. |
||||||||||||
failureBehavior | string | stop | ||||||||||
stop predict location and create key. don't predict location use existing key to predict location |
||||||||||||
limitProcessing | int | 1 | ||||||||||
This will create a Domain of
Definition (DOD) of the bounding box of all active trackers. Only that
portion of the image will be loaded from disk when tracking, therefore
it will go more quickly. This has no effect on the final output image.
|
||||||||||||
trackNName | string | trackN | ||||||||||
The name of the track. You can
change this. |
||||||||||||
trackNX/Y | float | NA | ||||||||||
The actual track point in X and
Y. This is what you would use if you were linking a parameter to a track
point. |
||||||||||||
trackNCorrelation | float | NA | ||||||||||
The correlation value of that
key to the original sample. A score of 1 is a perfect score. 0 is a
very very very very bad score. |
||||||||||||
trackNWindow Parameters | float | NA | ||||||||||
These multiple parameters control
the windowing of the tracking box, and are not relevant to exported
values. |
In addition, if you click the right mouse in the textfield of a trackName, you will get a special menu to manipulate your tracks: | ![]() |
Function
|
Notes
|
Copy/Paste
|
These are the standard text editing Copy and Paste commands.
|
Load Track | This will pop up a
list of all currently existing tracks. Select one, and a copy of that
track will loaded into both the X and Y parameters of the current tracker. |
Link | This will pop up a
list of all currently existing tracks. Select one, and a link will be
made from the current tracker to the one you have just selected. Therefore,
if you modify the tracker you selected with the Link command, the
current tracker will be updated. If you delete the tracker you linked
to, you will of course loose your tracking data. |
Average Tracks | When this is called
up, you get a list of all tracks. Select up to four tracks that you want
to average together, and hit OK. An expression will be entered into the
current trackX and Y fields that links back to your averaged
tracks. Note that you cannot choose to average your current track - it
will in fact be deleted by this action. See below for using this function. |
Smooth Tracks | This will present
a slider to execute a blurring function on your tracking curves, with
the number being the number of keys considered into the smoothing operation.
To see the curves themselves, open the trackName subtree, and load
the trackNX and Y parameters into the Curve
Editor. If you use Smoothing and Averaging operations as your standard
workflow, we recommend generating your tracks first with the Tracker,
and then linking them into a MatchMove or a Stabilize. See
below for using this function. |
Load Track File | This will load a Shake-formatted
track file from disk. See below
for a format of a track file. |
Save Track File | This will save a Shake-formatted
track file to disk. |
Clear Track | This will reset the
current tracker. To reset the entire function, go to the top part of the
parameters page and select with the right mouse the Reset
Values function. |
The tracker works by taking a small snapshot of pixels from one frame, which we call the reference pattern. This is represented by the inner box on the onscreen tracker. The tracker then advances to the next frame, and samples everywhere inside the larger tracker box, which we call the search region. When sampling inside the search region, it positions a box the same size as the reference pattern at the pixel in the first row, first column, taking a sample. It then advances to the next pixel (or subpixel) column in the search region and takes a second sample. For every sample it takes, it assigns a correlation value by comparing it to the reference pattern. When it has taken all of the samples, it assigns the new tracking point to the sample with the highest correlation value. It then repeats this on the following frames until the tracking is complete. Your referenceBehavior controls if and when the reference pattern gets updated. By default, the reference pattern is the set to use start frame, meaning the first frame you start tracking at, so even the last frame will compare the samples to the first frame. Other modes can change this behavior.
|
|
The amount of samples taken in the search region is determined by the subPixelResolution parameter. If you have a resolution of 1 (not very accurate), it positions the reference pattern box at every pixel to find a sample. This isn't accurate, because most movement occurs at the subpixel level, i.e., the movement is less subtle than one pixel across, and therefore gets factored in with other objects in the calculation of that pixel's color. The next resolution down in subPixelResolution, 1/4, advances the reference pattern box in .25 pixel increments, and so is therefore more accurate. The example here is a theoretical 1/2 resolution since the pattern is advanced in .5 pixel increments. Keep in mind that the lower the number, the more samples it takes. At 1/4 resolution, it takes 16 times more samples per pixel than at a resolution of 1. At 1/64, it takes 4096 times more samples per pixel. It is because of this that most trackers don't handle significant rotational movement very well - they (Shake's included) only test for panning changes, not rotational. If you did, you would have to multiply the amount of panning samples by the amount of degrees, which would be prohibitively costly at this stage. If you are faced with a situation with rotational movement, try using a referenceBehavior of update every frame. This means that the reference pattern is updated at every frame, so you are only comparing a frame with the frame before it, and not the first frame. |
![]() |
Despite all of the demos we love to see (and show), tracking is rarely a magic
bullet that works on the first attempt. These are some strategies to help you
get accurate tracks.
Pick a good reference pattern
The ideal pattern is one that doesn't change perspective, scale or rotation, and does not go offscreen or get obscured by other objects. It doesn't change overall brightness or color, and is very high contrast. It also is distinct from other patterns in the same neighborhood. Meanwhile, in the real world, we have to contend with all of these factors in our footage. Think about that next time you watch a tracking demo, by the way.
In this example, we have the corner at area A, or anywhere along the line near B. A would be the better choice, since B can be matched anywhere along the horizontal black line, and probably on the dark line that is below the letter B. Therefore, one principle is to avoid similar patterns horizontally or vertically from your candidate pattern. If you can easily find one, so can the tracker.
|
![]() |
Another potential candidate is the word Alivia
in the sign. However, as we examine the clip, you can see that the text
becomes an indecipherable blur. We can also see how close our A and
B points get together- the clip has significant scaling on the X
axis and some scaling in the Y due to the perspective shift. Although the
overall brightness has dropped, our contrast remains relatively high at
the A point. This would be a good candidate for tracking with referenceBehavior
of update if below reference tolerance or update every frame.
|
![]() |
The reference sample should also be relatively constant and unique over time as well. Flickering lights would be bad for example, but if they were regular enough, you could try setting your trackRange to match the flicker, i.e., 1-5, 10-15, 20-25, etc.
|
|
This example shows a track marker that was placed on a TV screen so the client could place whatever they wanted on TV in the shot. The default tracker reference pattern is unnecessarily too large. The only interesting detail is the black cross in the middle. Otherwise, most of the pattern matches up with most of the rest of the image - green. To adjust for this, we limit the reference pattern to more closely match
the black crosshair. |
![]() ![]() |
Pick a good search region
You should also suit your search region to match both your movement,
and the patterns near your reference pattern. The bus example doesn't
track the lower left corner of the sign very well with the default settings
because the middle black horizontal line can easily be matched up with
the black line at the very base of the search region in later frames of
the clip. This is because the X axis is squeezed so much that vertical
details disappear. Additionally, since the bus is moving to the right,
there is no point in wasting cycles to the left of the point. Remember,
the larger the search region, the more samples it has to take. |
![]() ![]() |
This image shows a corrected search
region, which is now high enough to not include the lower black line, and
extends minimally to the left. |
![]() |
Another technique you can use is to manually insert keyframes of your track. For example, if you have 100 frames to track, you might put in a key every 5 or 10 frames by using the AutoKey feature. A trick to help you to do this is to put an increment of 5 or 10 in the Time Bar. When you hit the left or right arrow key, you will jump by the increment amount.
Once you are done manually putting in your keys, go back to frame 1, and put
your failureBehavior on use existing keys to predict location.
This means the tracker will search along the tracker's pre-existing motion path
to find matching patterns.
The tracker works best with a high contrast reference pattern. To the human
eye, we see contrast as represented by value. However, you may sometimes have
higher contrast in saturation or hue, so switch over to a different color space
with the matchSpace parameter. A shot may also have a higher contrast
in a specific RGB channel than in the others, for example the blue channel might
have a bigger range than the red or green. In that case, you could put a Color
- Reorder bbb on the image, and then track with luminance
as your matchSpace.
Because the log to lin conversion increases contrast, you will probably have better results on linear data than on log data.
Ideally, you should be tracking an image with the most amount of raw data. This means if you put a Brightness of .5 on your image, you have lost half of the color information. Therefore, track the image before the Brightness operator is applied.
Be careful with proxyScale and proxyRatio settings. These are generally bad for two reasons. The first is that you are filtering the image which means detail is lost. The second reason is that you are automatically throwing data away. For example, if you are using 1/4 proxy, you are automatically throwing away four pixels of data in any direction, which means an 8x8 grid of potential inaccuracy.
In some cases, you may in fact want to modify your images to improve contrast in the reference pattern, either with a ContrastLum or ContrastRGB. Since you are just using this image to generate tracks, you are not obliged to keep the contrast image for the rest of your composite.
When working with grainy film footage, a good strategy is to strip out the red or green channel (usually green as it has the highest luminance value) because the blue channel holds the most grain and is therefore likely to disturb its pattern every frame. You also might try adding a Blur of a few pixels or a Median filter to reduce the grain before you track.
Finally, sometimes you may be having problems with random film grain being
too severe, and your reference pattern becomes useless. Apply a Filter
- Median or Blur before you track to reduce the effects of grain.
Yes, I know this contradicts the second suggestion. Welcome to the exciting
world of tracking.
If your image has significant changes in size and angle, you should try two different referenceBehaviors, update if below reference tolerance or update every frame. The second one is the more drastic choice because you will get an inherent accumulation of tiny errors if you update every frame. Therefore, try update if below reference tolerance first.
Another strategy is to jump to the midpoint frame of the clip and track forwards to the end frame of the clip. Then return to the midpoint frame and track backwards to the beginning of the clip. See below on more techniques to refine and average your tracks.
A second strategy is to do two Stabilize nodes. The first can be considered
a rough stabilize. The second stabilize then works off of that, and will therefore
have a much better chance of finding acceptable patterns. Since the Stabilize
nodes will concatenate, no quality is lost.
There are two basic techniques for correcting for points that get obscured by either going offscreen or by having an object passing in front of them.
The first strategy is to use different failureBehavior, either predict location and create key or predict location and don't create key. The first is good for nice linear behavior, as it will continue to lay down keys following the vector of the last two valid keys it made, meaning the two frames prior to the pattern becoming obscured. It is excellent for points that go offscreen and never reappear. The second setting is a little nicer for patterns that reappear because it will continue to search on a vector, but will only create a key if it finds another acceptable pattern. You will have an even interpolation between the frame before the pattern was obscured, and the frame after it is revealed again.
These only work if you have nice linear movement. The second strategy
is to use the Offset button |
![]() |
Once you have a track, you can modify it in several ways to massage the data.
You can manually modify a track either in the Viewer or in the Curve
Editor, you can average tracks together, and you can smooth tracks to
remove noise. There is also a trick to stabilizing a camera move that keeps
the move but removes the jitter.
At any time, you can turn on the Autokey button
in the Viewer and manually adjust a tracking point by simply grabbing it and
putting it where you need. You can use the + and - keys by the Backspace
key to zoom in and out. The zooming follows the cursor, so just place the cursor
on the key point in the Viewer and zoom in. Hitting the Home key or the
Home button
will return you to your normal view.
You can also adjust a tracking curve in the Curve Editor. Go to the trackName and open the subtree. Click on the clock icons to load a parameter into the Editor. In this example, track1X is loaded into the Editor. We have also turned on autokey so that we can numerically alter the track by hand. | ![]() |
A common technique is to track forwards from the first frame to the last, and then create a second track, tracking backwards from the last frame to the first. These two tracks are then averaged together to hopefully derive a more accurate track. If you are going to be using this method, we recommend you track with the Tracker node, and then load your tracks into Stabilize or MatchMove using the Load or Link Track functions (they haven't moved from the right mouse menu of trackName textfield).
As an example of how to average tracks, create a Tracker,
and create two extra trackers for a total of three. Create tracks on track1
and track2. Then hit the right mouse menu on track3 and select
Average Tracks. This will pop up a window giving you up to four input
trackers to average together. Select Tracker1.track1 and Tracker1.track2
in the first two popups, leaving the last two at none. Hit OK,
and you will see that track3 is midway between the first two tracks. |
![]() |
This works by creating an expression in both the track3X and track3Y parameters. The expression for the X parameter looks like this: (Tracker1.track2X+Tracker1.track1X)/2 Since these are linked back to track1 and track2 on the Tracker1 node, you shouldn't delete these. For more information on linking, see below. You can average up to four tracks at once, but you can of course continue to manipulate your tracks with further functions, including Average Tracks. Note that the Smoothing and Averaging tools that are available within
Tracker/Stabilize/Matchmove are provided as somewhat
of an interim solution for the 2.2 release. The eventual plan is to have
this sort of functionality moved directly into the Curve Editor,
with a much greater degree of control over the degree and extent of smoothing
(see below) that is applied. It is hoped that the current version will
be at least somewhat useful, even though the feature is fairly limited
at this point. |
![]() |
You can smooth a track with the Smooth Tracks function in the right-mouse menu. Again, this is a temporary workaround until our next release, but here is how it works for now.
Select the track you want to smooth. Although you can of course Undo, you might want to copy the track to another tracker using the Load Track function on the second tracker. Select Smooth Tracks, and a window will appear, asking you for a smoothValue parameter.
The default is five, which means that 5 points centered on the current one will be used to compute the current one's new, smoothed, value. This is a standard Gaussian (bell-curve type) filter. In other words, if you leave it at 5, when it computes the value of frame 12, it will take into account frames 10, 11, 12, 13, and 14. If you set it to 3, it will use 11, 12, and 13. For frame 13, it would use 11, 12, 13, 14, 15 for frame 13 (smoothValue = 5) and 12, 13, 14 for frame 13 (smoothValue=3). Just a warning: in this beta there isn't any sign that Shake is working, but just have patience for a few minutes, although you can load the curves into the Curve Editor. The larger the smoothValue the more points are taken into account (and thus more calculations done) for every point in the curve. Even values for smoothValue use the next largest odd number of frames but the end ones don't contribute as much.
As an example, here I have a noisy track:
After the smoothing, it looks like this
:
Referencing track point data works similarly to referencing any other parameter within Shake. The twist here is that since you can rename the track point, you can change the name of the parameter to which you are referring. For example, say I have a Tracker node named Tracker1 and I've left a track point at its default name "track1". To reference the X track data I would use: Tracker1.track1X. If I change the name of the track point to "lowerleft", then the reference would be to Tracker1.lowerleftX. This follows for the Y data as well. An important difference with standard linking: Since the MatchMove and Stabilize nodes use a Tracker internally, I must reference their tracking data with the assumption that there is a tracker inside of it. As an example, lets say I've named the track point "upper_right". I would use MatchMove1.tracker.upper_rightX. Similarly, to get the Y value of a track point named "bottom" in a Stabilize node named Stabilize5, I would use Stabilize5.tracker.bottomY. Here are some examples in context:
|
|
You can copy or link to any tracker with any other tracker by using Load or Link Track. You can also link to them using Shake's standard expressions.Open the subtree on a trackName, you will see the X and Y coordinates listed out. Therefore, to link to them, you use NodeName.tracker.tracknameX and Y
|
![]() |
For example, here I have linked the Scroll node to a tracker so that I don't loose any data. If the Tracker node for the above illustration was called Tracker1, this is how you would link to the X and Y parameters. Tracker1.tracker.track1X |
![]() |
If I want to invert the transformation, i.e., turn 1 point tracking data into stabilization data, negate the expression: -Tracker1.tracker.track1X For 2 point MatchMove and Stabilize, Shake will derive
the scaling factor and rotation angle. They can be viewed under the applyScale
and applyRotation subtrees, and of course linked to as well. |
![]() |
This technique is useful when you have a camera move that you want to keep, but there is a lot of jitter in the shot. You therefore need to stabilize the shot, but only by the small amount that is the actual jitter. To do this, we combine the above techniques.
A sample saved track file for use with Save or Load Track File.
TrackName track1 Frame X Y Correlation 1.00 462.000 210.000 1.000 2.00 405.000 192.000 1.000 ...etc...
This is different from Load Expression which is available on the right
mouse menu for any textfield, not just trackers. That command looks for preformatted
Shake expressions, i.e., to load the above information into a Move2D,
you would have to load two files, one for xPan and one for yPan.
Their formats would be something like this:
Linear(0,0462@1, 405@2, ....)
and
Linear(0,210@1, 192@2, ...)