About Tracking, Stabilizing and MatchMoving |
This section discusses tracking in depth, including interface features, workflow issues, and tips on successful tracking and manipulation the data. For specific formats of each node, jump to the appropriate reference page. For a tutorial on how to use the trackers, jump to the Tracking Tutorial.
There are three tracking nodes available in Shake Tracker, Stabilize, and MatchMove.
This is a general overview of the steps you take to generate a track. Each of these is detailed in the following sections.
General Tracking Workflow:
Play your background clip several times to determine a good tracking point.
Attach your tracking (Transform - Tracker, Stabilize, or MatchMove) node to the background clip.
Make sure the onscreen controls are visible in the Viewer .
Go to the frame that you want to start the track.
Position the tracker on the point you want to track and adjust the reference pattern and the search region.
Ensure the trackRange parameter reflects the frame range you want. For FileIns, it takes the range of the clip. For Shake-generated elements you must provide a range, for example, 1-50.
Press either the backward or forward track button
to start the track processing (which ideally needs to be done once, but
never really happens that way). The track generates animation curves.
Stabilize additions:
Decide if you need one, two, or four-point tracking.
Toggle the applyTransform parameter to on to stabilize the plate.
If you are using Stabilize for match-moving, set the inverseTransform parameter to match.
MatchMove additions:
Attach the foreground element to the first input of the MatchMove.
Decide if you need one, two, or four-point tracking.
Set the outputType parameter to Over (or another compositing operation).
Toggle the applyTransform parameter to active to match the motion.
If you are four-point tracking and you want to attach four arbitrary points
on the foreground image to the background, press the BG/FG button to toggle
it to FG .
Then place the four corners on the Foreground element. These represent the
corners that get plugged into the four tracking points.
Toggle back to BG .
You may have to switch the outputType parameter back to Over, or the compositing mode you set in step 3.
All tracking nodes share the following interface features:
A tracker consists of three onscreen controls: The search region, the reference pattern, and the track point. The outer box is the search region, the inner box is the reference pattern. The cross hair is the track point.
To offset the tracking area from where the keyframe is actually saved (for
example, the tracking point moves offscreen or is obscured), toggle the
Offset Tracker button .
This allows you to move the search pattern without moving the track key.
When your tracking reference pattern goes offscreen, grab something else
to keep tracking on. To reset the search area back to the key, use the Reset
Tracker button:
One limitation with MatchMove is that it does not handle the pass-through of upstream onscreen controls, so those controls are in their un-transformed state.
![]() ![]() |
Track Backward/Forward. Click to start the tracking process.
All visible trackers attempt to lay down new tracking keyframes. Tracking
continues until one of the trackRange frame limits is reached
the upper limit if you are tracking forward |
![]() |
Offset Track - Off. The track search region and the
tracking point are linked. If you move one, the other follows. |
![]() |
Offset Track - On. The track search region and the
tracking point are offset from each other. If your original sample
pattern becomes obscured, offsett the search region to a different
area. Your keyframes are still saved relative to the original spot. |
![]() |
Reset Track. Returns an offset track search
region back to the track point.
|
![]() ![]() ![]() |
Track Display. These buttons display all trackers, curves and keyframes, just the trackers and keyframes, or just the trackers. You can also control visibility of individual trackers as described below. |
![]() ![]() |
These buttons appear on the MatchMove node when four-point tracking. They allow you to toggle between adjusting the track points on the background (BG) or the corners of the foreground (FG) that get pinned to the track points. |
Next to each track name is a Color Picker and a Visibility toggle. The Color Picker selects a different color for the display of that tracker. The Visibility toggle turns on and off the display of a tracker. When enabled, it retracks each time you press the Track button. If the tracker is invisible, it is not processed next time you press the Track buttons.
All trackers share the following parameters:
Parameter
|
Notes
|
||||||||||
trackRange |
The trackRange parameter is the potential frame
range limit of your tracking. By default, tthe range is set to the
clip range. For generated elements such as RGrad, it will take
a range of 1. You can set new limits using Shake's standard range
description, for example., 10-30x2. If you stop tracking and start
up again, it starts from the current frame until it reaches the lower
or upper limit of your trackRange, depending on if you are
tracking forwards or backwards.
|
||||||||||
subPixelResolution |
The subPixelResolution parameter determines the resolution of your track. The smaller the number, the more precise track. Possible values:
|
||||||||||
matchSpace | The pixels are matched according to the correlation between the selected color space, either luminance, hue, or saturation. When an image has roughly the same luminance, but contrasting hues, you should switch to hue-based tracking. | ||||||||||
referenceTolerance | A tracking correlation of 1 is a perfect score there is an exact match between the original reference frame and the sampled area. When you lower the referenceTolerance, you accept greater inaccuracy in your track. If tracked keyframes are between the referenceTolerance and the failureTolerance, they are highlighted in the Viewer. Also, in some cases, referenceBehavior is triggered if the tracking correlation is below the referenceTolerance. | ||||||||||
referenceBehavior |
This behavior dictates the tracking area reference sample. By default, the reference pattern is the first frame at which you start the track, not necessarily the first frame of the trackRange. The last two behaviors in the referenceBehavior list measure the tracking correlation and match it to the referenceTolerance to decide an action.
|
||||||||||
failureTolerance | If the correlation of a track falls below this value, it initiates the failureBehavior. | ||||||||||
failureBehavior |
What occurs when the correlation drops below the failureTolerance:
|
||||||||||
limitProcessing | This creates
a Domain of Definition (DOD) around the bounding boxes of all active
trackers. Only that portion of the image is loaded from disk when tracking,
so the track is faster. This has no effect on the final output image.
|
||||||||||
trackNName | The name of the track. To change the name, click in the text field. | ||||||||||
trackNX/Y | The actual track point in X and Y. This is what you would use if you were linking a parameter to a track point. | ||||||||||
trackNCorrelation | The correlation value of that key to the original sample. A score of 1 is a perfect score. 0 is a very, very, very, very bad score. | ||||||||||
trackNWindow Parameters | These multiple parameters control the windowing of the tracking box, and are not relevant to exported values. |
Click the Right mouse button in the text field of a trackName
to show the special menu to manipulate your tracks:
Function
|
Notes
|
Copy/Paste
|
These are the standard copy and paste commands.
|
Load Track | Pops up a list of
all currently existing tracks. Select one, and a copy of that track loads
into both the X and Y parameters of the current tracker. |
Link Track | Pops up a list of
all currently existing tracks. Select one, and a link is made from the
current tracker to the tracker you select in the pop-up list. Therefore,
if you modify the tracker selected with the Link command, the current
tracker is updated. If you delete the tracker you linked to, you loose
your tracking data. |
Average Tracks | Displays a list of
all tracks. Select up to four tracks that you want to average together,
and press OK. An expression is entered into the current trackX
and Y fields linking back to your averaged tracks. Note that you
cannot choose to average your current track - it is deleted by this action.
|
Smooth Tracks | Displays a slider
to execute a blur function on your tracking curves, with the smooth value
being the number of keys considered into the smoothing operation. To view
the curves themselves, open the trackName subtree and click the
load parameter button to load the parameters into the Curve Editor.
If you use Smoothing and Averaging operations as your standard workflow,
it is recommended to generate your tracks first with the Tracker,
and then link them into a MatchMove or a Stabilize. |
Load Track File | Loads a Shake-formatted
track file from disk. See the
end of the tracking overview for the format of a track file. |
Save Track File | Saves a Shake-formatted
track file to disk. |
Clear Track | Resets the current tracker. To reset the entire function, right-click an empty area of the parameters tab and select Reset All Values. |
The tracker works by taking a small snapshot of pixels from one frame, called the reference pattern. This is represented by the inner box on the onscreen tracker. The tracker then advances to the next frame, and samples the area inside the larger tracker box, called the search region. When sampling inside the search region, the tracker positions a box the same size as the reference pattern at the pixel in the first row, first column, and takes a sample. It then advances to the next pixel (or subpixel) column in the search region and takes a second sample. For every sample the tracker takes, it assigns a correlation value by comparing it to the reference pattern. When it has taken all of the samples, it assigns the new tracking point to the sample with the highest correlation value. It repeats this on the following frames until the track range is complete.
The referenceBehavior controls if and when the reference
pattern is updated. By default, the reference pattern is the set to use
start frame, meaning the first frame you start tracking, so even the last
frame compares the samples to the first frame. Other modes can change this
behavior.
The amount of samples taken in the search region is determined
by the subPixelResolution parameter. A subPixelResolution of
1 positions the reference pattern box at every pixel to find a sample. This
isn't accurate, because most movement occurs at the subpixel level
the movement is more subtle than one pixel across, and therefore gets factored
in with other objects in the calculation of that pixel's color. The next resolution
down in subPixelResolution, 1/4, advances the reference pattern box
in .25 pixel increments, and so is therefore more accurate. The example here
is a theoretical 1/2 resolution since the pattern is advanced in .5 pixel
increments. Keep in mind that the lower the number, the more samples it takes.
At 1/4 resolution, it takes 16 times more samples per pixel than at
a resolution of 1. At 1/64, it takes 4096 times more samples per pixel.
It is because of this that most trackers don't handle significant rotational movement very well they (Shake's included) only test for panning changes, not rotational. If they did did, it would have to multiply the amount of panning samples by the amount of degrees for the number of samples to take, which would be prohibitively costly at this stage. If you are faced with a situation with rotational movement, try using a referenceBehavior set to update every frame. This means that the reference pattern is updated at every frame, so you are only comparing a frame with the frame before it, and not the first frame.
Also keep in mind that manual adjustments are standard for tracking problems.
Despite all of the demos we love to see (and show), tracking is rarely a magic
bullet that works on the first attempt. These are some strategies to help you
get accurate tracks.
Pick a Good Reference Pattern
The ideal reference pattern is one that doesn't change perspective, scale or rotation, and does not go offscreen or get obscured by other objects. It doesn't change overall brightness or color, and is very high contrast. It also is distinct from other patterns in the same neighborhood. Meanwhile, in the real world, we have to contend with all of these factors in our footage (think about that next time you watch a tracking demo, by the way).
In this example, there is the corner at area A, or anywhere along the line near B. Area A is the better choice, since B can be matched anywhere along the horizontal black line, and probably on the dark line that is below the letter B. One rule is to avoid similar patterns horizontally or vertically from your candidate pattern. If you can easily find one, so can the tracker.
Another potential candidate in the above example is the word Alivia
in the sign. However, as the clip advances, the text becomes an indecipherable
blur. The A and B points also move closer together the
clip has significant scaling on the X axis and some scaling in the Y due to
the perspective shift. Although the overall brightness has dropped, contrast
remains relatively high at the A point. This is a good candidate for
tracking with referenceBehavior set to update if below reference tolerance
or update every frame.
The reference sample should also be relatively constant and unique over time as well. Flickering lights would be bad for example, but if they were regular enough, you could try setting your trackRange to match the flicker, that is, 1-5, 10-15, 20-25, and so on. Granted, this is awkward. A better solution is to set your failureBehavior to don't create key or predict position and don't create key.
The following example shows a track marker placed on a TV screen so the client
could place some silly tripe on the TV. The default tracker reference pattern
is unnecessarily large. The only interesting detail is the black cross in the
middle. Otherwise, most of the pattern matches up with most of the rest of the
image green. To adjust for this, limit the reference pattern to more
closely match the black crosshair.
Pick a Good Search Region
You should also suit your search region to match your movement and the patterns
near your reference pattern. The bus example doesn't track the lower left corner
of the sign very well with the default settings because the middle black horizontal
line can easily be matched up with the black line at the very base of the search
region in later frames of the clip. This is because the X axis is squeezed so
much that vertical details disappear. Additionally, since the bus is moving
to the right, there is no point in wasting cycles to the left of the point.
Remember, the larger the search region, the more samples the tracker has to
take.
The corrected search region is now high enough to not include the lower black
line, and extends minimally to the left.
Another technique you can use is to manually insert keyframes of your track. For example, if you have 100 frames to track, you might put in a keyframe every 5 or 10 frames with the AutoKey feature. A trick to help you to do this is to set an increment of 5 or 10 in the Time Bar. Press the Left or Right arrow key to jump by the increment amount.
Once you keyframes are manually entered, return to frame 1, and put your failureBehavior on use existing keys to predict location. The tracker searches along the tracker's pre-existing motion path to find matching patterns.
The tracker works best with a high contrast reference pattern. The human eye sees contrast as represented by value. However, you may sometimes have higher contrast in saturation or hue, so switch over to a different color space with the matchSpace parameter. A shot may also have a higher contrast in a specific RGB channel than in the others, for example the blue channel might have a bigger range than the red or green. In such a case, apply a Color - Reorder bbb to the image, and then track with luminance as your matchSpace.
Because the log to lin conversion increases contrast, you may have better results on linear data than on log data. Of course, maybe not. Nice.
Ideally, you should track an image with the most amount of raw data. This means if you put a Brightness of .5 on your image, you have lost half of the color information. Therefore, track the image before the Brightness operator is applied.
Don't ever track with proxy settings. These are bad for two reasons. The first is that you are filtering the image so detail is lost. The second reason is that you are automatically throwing data away because of data round-off. For example, if you are using 1/4 proxy, you are automatically throwing away four pixels of data in any direction, which means an 8x8 grid of potential inaccuracy.
Yup, that's right, the exact opposite of what was just stated above. Isn't compositing fun? It often pays off to apply a Color - Monochrome node onto your image and drop the blue channel out if you have particularly grainy footage. Another good strategy is to apply a Filter - Blur to one or more channels to get rid of grain irregularities.
In some cases, you may want to modify your images to improve contrast in the reference pattern, either with a ContrastLum or ContrastRGB. Since you use this image to generate tracks, you are not obliged to keep the contrast image for the rest of your composite.
For images with significant change in size and angle, you can try two different referenceBehaviors update if below reference tolerance or update every frame. The second one is the more drastic choice because you get an inherent accumulation of tiny errors if you update every frame. Therefore, try update if below reference tolerance first.
Another strategy is to jump to the midpoint frame of the clip and track forwards to the end frame of the clip. Then return to the midpoint frame and track backwards to the beginning of the clip.
A second strategy is to apply two Stabilize nodes. The first can be
considered a rough stabilize. The second stabilize then works off of the first,
and will therefore have a much better chance of finding acceptable patterns.
Since the Stabilize nodes concatenate, no quality is lost.
There are two basic techniques to correct track points that are obscured by moving offscreen or an object passing in front of them.
The first strategy is to use a different failureBehavior, either predict location and create key or predict location and don't create key. The first setting is good for nice linear behavior it continues to lay down keyframes following the vector of the last two valid keyframes it made, so the two frames prior to the pattern become obscured. It is excellent for points that go offscreen and never reappear. The second setting is a little nicer for patterns that reappear because it continues to search on a vector, but only creates a keyframe if it finds another acceptable pattern. You have an even interpolation between the frame before the pattern was obscured, and the frame after it is revealed again.
The above strategies only work if you have nice linear movement. The second
strategy is to use the Offset button
on the Viewer. When offset, you can split the reference pattern and search region
away from the track point. In this example, the track is obscured by a lamp
post, so the search region (not the point, just the region) is moved to a nearby
pattern and tracking continues until the original pattern reappears. Even though
one region is examined, the points get saved in another. The second tracking
pattern should travel in the same direction as your original pattern.
A track can modified in several ways to massage the data. You can manually
modify a track in the Viewer or in the Curve Editor, average tracks together,
smooth tracks to remove noise, or remove jitter to smooth out a camera movement.
At any time, enable the Autokey button
in the Viewer to manually adjust a tracking point onscreen. You can use the
+ and - keys (next to the Backspace key) to zoom in and out. The zooming
follows the cursor, so place the cursor on the key point in the Viewer and zoom
in. Press the Home key or the Home button
to return you to your normal view.
You can also adjust a tracking curve in the Curve Editor. Open the trackName subtree and click on the clock icons to load a parameter into the Editor. In this example, track1X (only the X parameter) is loaded into the Editor.
A common technique is to track forward from the first frame to the last, create a second track and track backwards from the last frame to the first. These two tracks are then averaged together to (hopefully) derive a more accurate track. If you plan to use this method, it is recommend to use the Tracker node, and then load your tracks into Stabilize or MatchMove with the Load or Link Track functions (they haven't moved from the right mouse menu of trackName text field).
To Average Tracks:
Create a Tracker, and click Add twice (to have a total of
three trackers).
You could also potentially use tracks from any other Stabilize, MatchMove
or Tracker node.
Create tracks on track1 and track2.
Right-click on track3 and select Average Tracks from the menu.
Select Tracker1.track1 and Tracker1.track2 as the first two inputs, respectively, leaving the last two at none. The following illustration shows Stabilize1 to remind you that any tracking node can be a track source.
This works by creating an expression in both the track3X and track3Y parameters. The expression for the X parameter looks like this:
(Tracker1.track2X+Tracker1.track1X)/2
Because these are linked to track1 and track2 on the Tracker1 node, don't delete these. For more information on linking, see below.
You can average up to four tracks at one time, but you can of course continue to manipulate your tracks with further functions, including Average Tracks.
You can smooth a track with the Smooth Tracks function in the Tracker Parameters. Prior to smoothing the curve, you may want to copy the track to another tracker with the Load Track function on the second tracker.
To Smooth a Track Curve:
The default is 5, which means that 5 track points centered on the currently evaluated point are used to compute the current point's new, smoothed, value. This is a standard Gaussian (bell-curve type) filter. In other words, if you leave it at 5, when it computes the value of frame 12, it takes into account frames 10, 11, 12, 13, and 14. If you set it to 3, it uses 11, 12, and 13. The larger the smoothValue, the more points are taken into account (and thus more calculations done) for every point in the curve. Even values for smoothValue use the next largest odd number of frames, but the end ones don't contribute as much.
As an example, here is a noisy track curve prior to smoothing:
After the track curve is smoothed:
:
Referencing track point data works similarly to referencing any other parameter within Shake. The twist here is that since you can rename the track point, you can change the name of the parameter to which you are referring. For example, if you have a Tracker node named Tracker1 and you've left a track point at its default name "track1." To reference the X track data use: Tracker1.track1X. If you change the name of the track point to "lowerleft", then the reference is changed to Tracker1.lowerleftX. This applies to the Y data as well.
You can also use the Link Track button to link one track to another track, simultaneously linking both the X and Y curves simultaneously.
The following technique is useful when you have a camera move that you want to keep, but there is a lot of jitter in the shot. You need to stabilize the shot, but only by the small amount that is the actual jitter. To do this, combine the techniques mentioned above.
Track the plate with a Tracker node (assume in this case it is called Tracker1).
Create a second tracker inside the same node, and load track1 into track2 using Load Track,.
Smooth track2 using Smooth Track.
At this point, we have a track, and a smoothed version of it. The following example shows the Y curves of the two tracks:
A sample saved track file for use with Save or Load Track File.
TrackName track1 Frame X Y Correlation 1.00 462.000 210.000 1.000 2.00 405.000 192.000 1.000 ...etc...
This is different from Load Expression which is available on the right
mouse menu for any text field, not just trackers. That command looks for preformatted
Shake expressions. For example, to load the above information into a Move2D,
you have to load two files, one for xPan and one for yPan. Their
formats are something like this:
Linear(0,0462@1, 405@2, ....)
and
Linear(0,210@1, 192@2, ...)