About Premultiplication and Compositing

This section is perhaps one of the most important of the Overview, because it talks about compositing and color correcting process on a fundamental level. It will detail the process that makes standard compositing operators work, and about what the hell the term premultiplication actually means and why you should know about it, regardless of what compositing software you are using.

We think about premultiplication when we have modified a Foreground element and it is then comped over a Background image. By "modify", we classically mean color-correction, and especially nodes that raise the black level like Add, Clamp, ColorMatch, ColorCorrect, Compress, and Contrast, but there are consequences with filters as well.

For those of you who don't feel like reading a long-winded dissertation on the mathematics of premultiplication, here is the abridged version:

Rule Number 1: Color Correct unpremultiplied images


Rule Number 2: Filter and Transform premultiplied images



Egghead definition of Premultiplied

Let's start with a definition of what a premultiplied image is:

"An image that has its RGB channels multiplied by it's alpha channel."

Typically, images coming out of a 3D render are premultiplied, meaning the transparent areas have black both in the RGB areas and in the same areas of the alpha channel. A side effect of a premultiplied image is that the RGB channels never have a higher value than the alpha channel. The beauty of all this is that it has absolutely no meaning without understanding how compositing works, which is why this page is very long. So, let's continue...

Problems Caused by Ignoring Premultiplication

Aside from being ostracized by your friends and perhaps detecting a faint musty smell (I'm just kidding here - this is a commonly misunderstood concept. Or, as certain Presidents might say, "misunderestimated"), here are two typical problems that occur because of ignoring the premultiplied state of an image.

In doc/pix/premult, read in the image bg.jpg and munch_pre.sgi. These images come from us courtesy of our fine friends at Oddworld Inhabitants for the game "Munch's Oddysee". Note for the purposes of illustration, these differ from the final composite, which looks much better than what we are playing with here.

The second image, munch_pre.sgi, is rendered out of Maya, is premultiplied (the blacks in the alpha match the blacks in the RGB) and has the embedded alpha channel. This image, by the way, is of Munch. We aren't sure what Munch is. He's got this cool conical tongue, though, and can shoot lightning bolts out of his head, so that should count for something.

munch_pre.sgi munch_pre.sgi, alpha bg.jpg

 

Put munch_pre Over bg.jpg:

Tree Over1

 

Let us really concentrate hard and imagine we want to color-correct Munch by inserting a ContrastLum onto munch_pre and setting the value to .5. If you zoom into the edges, you will see a nasty white edge around Munch. That would be problem #1.

Tree Over1 detail

Now, select the ContrastLum node in the Node View and replace it with an Add node by Ctrl+clicking on the Add button in the Color tab, and then boost up your red/green/blue to something like .4 - you can do this by holding down the V key over the Add color picker and drag the mouse to the right. In this example, you can see that we have brightened the entire image, even though we attempted to just color-correct Munch. This would be problem #2.

Tree Over1

By ignoring premultiplication status, we can have either problems with edges, or with global levels being brought up. Many people have seen these types of errors, and assumed it was a mask problem, so they pinch in the mask a bit. Even more extreme people have erroneously assumed you can't color-correct premultiplied images. Happily for our competitive benefit, these problems are easily solved through proper management of premultiplication, which, I know, I still haven't explained.

 

The Math of Over and KeyMix

To understand why these problems occur, it is best to cut straight to the heart of compositing by understanding how a standard composite operator (foreground through a mask over a background) works. This has nothing to do with Shake, but in fact was worked out in the 1970s, mainly by two clever but no doubt fun fellows named Porter and Duff.

Here is the math they came up with. In this equation, A is the foreground's alpha channel:

Comp = (Fg * A) + ((1-A)*Bg)

Rather than go into this too deeply, I will briefly point out the signficant part, Fg*A, which is the definition of premultiplication - remember "RGB multiplied by it's alpha"? Since I don't want to sweat about the math, we will alternatively understand this by actually building the composite with other nodes, in effect reconstructing an Over node from scratch.

Read in two more images from doc/pix/premult - munch_unpremult.jpg and munch_mask.iff: Technically speaking, the mask should be a single channel image - this is a two channel image, but I've gone to extreme pains of copying the alpha into the RGB channels of the munch_mask image. The munch_unpremult image is a non-premultiplied image - it has no mask, so there is no correspondence between black in the RGB channels and the mask to describe transparency of the image.

munch_unpremult.jpg munch_mask.iff

So, going back to the formula, the first step is (Fg*A). This is simple enough to do by attaching the Layer - IMult to munch_unpremult and munch_mask. This gives a result identical to munch_premult - the RGB is multiplied by the mask:

Tree IMult1

 

The next step in the formula is to invert the Foreground alpha channel: (1-A). This is done with the Color - Invert function and attaching it to munch_mask:

Tree Invert1

The next step in the formula, ((1-A)*Bg), calls for us to multiply the Background by the inverted alpha. Again, we use IMult:

Tree IMult2

Now, the crucial step, we add the two results together with an IAdd node

Comp = (Fg * A) + ((1-A)*Bg)

and we end up with exactly the same result as an Over node:

Tree IAdd1

By punching a hole in the background, the alpha has determined what is transparent when the two plates are added together. The key concept here is that because you are adding, anything that is black (a value of 0) means it won't show up in the composite.

IMult1 IMult2

The KeyMix and Over operators do all of this math for you so you don't have to create four nodes to do a composite. What is the difference between Over and KeyMix? Over assumes that the Foreground is premultiplied, i.e, indentical to IMult1. KeyMix is only for non-premultiplied images, i.e., identical to munch_unpremult. Therefore, strictly speaking, the math actually breaks down like this:

KeyMix = (Fg * A) + ((1-A)*Bg)
Over = Fg + ((1-A)*Bg)

In this example, the Fg of Over is already multiplied by the Foreground alpha channel, thus the term premultiplied - it isn't multiplied in the composite because it was previously multiplied, usually by the 3D renderer.

Unpremultiplying an Image

With this knowledge, we can go back and start to understand why we had errors when doing our ContrastLum and our Add using the Over. We placed an Add on the premultiplied Munch. To do the same thing here, you would place it on IMult1, and of course you will get the same error when you boost the Add to .5 - the entire image gets brighter, not just Munch. This is bad.

Tree with Add inserted IAdd2

The clue is in looking at Add2. You can see that there is no longer an exact correlation between the black areas of munch_mask and the black areas of the RGB channels:

Add2 munch_mask

So, being clever people, we know that the alpha mask is determining the transparent areas. We have to reset the relationship of the RGB channels with the alpha channel by multiplying the RGB by the mask, otherwise you will be adding .5 to the entire background image, which is in fact what happened and it made the entire background brighter. When we multiply any color by black (0), that color will turn to black, so by multiplying Add2 by the mask, all of the grey should disappear.

This is where things get a little odd. Insert another IMult after Add2 and attach it to munch_mask. This will re-premultiply the image and appear to work:

Tree with IMult3 inserted IAdd2

I say "appear" to work, because mathematically speaking, we have an error. We need to have this:

Comp = (Fg * A) + ((1-A)*Bg)

but in fact, we multiply the Fg two times (there are two IMults), so we have this:

Comp = (Fg * A * A) + ((1-A)*Bg)

You are probably thinking right now - "So what, big deal, it looks fine". Ok, Smarty-pants, to test to see if it actually does look fine in all cases, switch the Add to ContrastLum and set the value to .5 again. If you look very closely at the edge, you will see a dark rim around it:

Add switched to ContrastLum IAdd1 detail

If we ignore IMult3 (select it and hit I), we end up with the same nasty edges we had before, so obviously don't do that.

IAdd1 detail with IMult3 ignored

So, what are we to do? Now, going back to our Fourth Grade math class, we recall that if we divide something by a number and then multiply by the same number, it is the same as multiplying by 1, or the equivalent of no change at all. Therefore, if we are multiplying the Foreground by the mask one too many times, we can balance it out by dividing it by the alpha. Mathematically, this looks like this, if you care:

Comp = (Fg * A / A * A) + ((1-A)*Bg)

or, abbreviating it, we return to the proper formula, since the two extra A's cancel out:

Comp = (Fg * A) + ((1-A)*Bg)

Translating this into our node tree, we insert a Layer - IDiv before the color correction, attaching it to IMult1. If you look at the edge now, you can see that it is clean. To test this, select the IDiv node and hit I repeatedly to ignore the node:

IDiv inserted IAdd1 detail

When we insert the IDiv, by dividing by the mask, we have un-premultiplied the image, and it is ready for color correction. Therefore, we can conclude that color correction should be done on an unpremultiplied image.

Now, here comes the kicker. We could have bypassed the insertion of IDiv1 and IMult3 entirely by merely switching the ContrastLum up in between munch_unpremult and IMult1, because munch_unpremult is already an unpremultiplied image. Delete IDiv1 and IMult3 and switch ContrastLum1 up and you will have exactly the same clean result:

IDiv and IMult3 deleted, ContrastLum moved up IAdd1 detail

We can therefore derive this little rule:

Rule Number 1: Color Correct unpremultiplied images



Managing Premultiplication

Those steps have all been an elaborate illustration to understand Over, KeyMix and premultiplication. First of all, the basic difference between KeyMix and Over:

KeyMix
Over
For non-premultiplied Foreground images. The alpha can be anywhere, including in the Foreground image. For premultiplied Foreground images, but can also turn on premultiplication if necessary for unpremultiplied images. The alpha is in the Foreground element.

 

Let us duplicate our current complex tree with simplified versions.

Here is the version using KeyMix, which handles un-premultiplied Foreground elements, the Background, and a mask to split between the two. In this case, I hit invert in the KeyMix parameters (you could also switch the Foreground and Background inputs):

Identical tree using KeyMix

What if you are using a premultiplied image straight out of a 3D render? In that case, we have special tools, Color - MDiv and MMult, respectively to unpremultiply and premultiply the RGB channels by the alpha. "MDiv" stands for "Mask Divide", "MMult" for "Mask Multiply". MDiv turns the image into an unpremultiplied image, we do the color correction, and then we premultiply again. Using the premultiplied image as the foreground, the tree looks like this

Identical tree with a premultiplied Foreground

Over has a shortcut for premultiplying in its parameters. In this tree, we can omit the MMult entirely by turning on the preMultiply flag in Over1:

Identical tree with premultiplication handled by Over1: Over1 parameters

 

You don't need to apply an MDiv for each color-correction. You can stack as many up as you want:

Multiple corrections on one MDiv

 

Filters and Premultiplication

Now to our next step, which is using filters, like Blur. Here is Rule #2:

Rule Number 2: Filter and Transform premultiplied images

 

Let's see what happens if we filter an unpremultiplied image. Place a Blur on munch_unpremult, and we see that the mask has clipped off any soft gradation:

Blur on an unpremultiplied image KeyMix1 detail

Oops. Copy Blur1, and clone it by hitting Ctrl+Shift+V. If you hit Ctrl+E, you will see that Blur2 is slaved to Blur1. Apply Blur2 to munch_mask, and you can see that we are introducing a glowing highlight around our image.

Blur on an unpremultiplied image KeyMix1 detail

For there to be no glow, we have to do it on an image that will have nothing added to the rim, i.e., on a black background, since black has a value of 0 and will therefore add nothing to the filtering: Therefore, we need to fundamentally change the compositing order by premultiplying the Foreground by the mask with a SwitchMatte node and then using an Over node to composite - KeyMix is only for unpremultiplied images:

Blur on a premultiplied image Over2 detail

Here is the same tree using the premultiplied image. Remember to set the preMultiply parameter to off in Over1, or else you will get a black edge around the image:

Blur on a premultiplied image Over1 detail

Since Transforms do a bit of filtering, technically speaking you should do Transforms on a premultiplied image as well, although your tolerances are a little looser.

Typically, therefore, you ensure your image is unpremultiplied by inserting an MDiv onto premultiplied images, do your color correction, appy an MMult, apply transforms and filters, and then composite.

 

Nodes Affecting Premultiplication

The following nodes generally are able to switch premultiplication status of an image, although anything which operates on the mask differently from the RGB will affect this status.

Node Details
ColorCorrect This color corrector has an option under its Misc tab to indicate if the incoming image is premultiplied or not. If it is, set it to "yes" - an MDiv and MMult will be internally inserted into the computation.
Reorder This allows you to switch a channel into the alpha channel, so therefore could disrupt your premultiplication status. However, it isn't often used on images in the normal chain of color-correct, position, composite, but is more of a utility node.
DilateErode This node is typically used to add or chew away the matte by a few pixels. Set your channels to just "a". Since you therefore are modifing the matte separately from the RGB, you have made the image unpremultiplied.
LumaKey, DepthKey, DepthSlice All three of these keyers have the option to immediately set your image to premultiplied or not.
KeyLight This keyer has the ability to output either a premultiplied or unpremultiplied version, or just the alpha channel with the RGB untouched.
Primatte This keyer allows you to output either just the alpha or a premultiplied version. Setting it to unpremultiplied requires an external MDiv.
AddMix This is a modified Over operator that allows you to manually break the premultiplied relationship by tweaking curves of how the matte multiplies the foreground and background images. We've inherited it from another package, where it was used extensively because of the fact that this particular system had no control over premultiplication.
SwitchMatte This copies a channel from the second image to the first image's alpha channel. You have the option to immediately premultiply the image if you want - it is on be default.
DropShadow This should be done on premultiplied images only.