Introduction
Today compositing of parts of several image sources is extremely easy. Yet there is one exception: Adobe Photoshop. Photoshop is one of the rare major 2D applications in which you almost certainly will have a big problem called “The Unwanted Line” when you try to do 2D compositing. This might not account for all purposes that exist but it is nonetheless true for 3D rendered images that come with an Alpha Channel.
This tutorial will show you not only the wrong approach and explain the reasons for failure but will also show a technique/workaround with which you finally can use the big Photoshop for your compositing needs.
The normal approach to delete a background
There are many possibilities with which one can remove or mask a background from an image. From the eraser tool, quick mask mode to channel selection Photoshop offers many tools for each purpose. Yet, in this case (a 3D image with an alpha) all techniques will ultimately fail. The following process is therefore used as an example.
After you have rendered an image in your 3D application of choice you can load it into Photoshop. In your channels list you should have the RGBs and one additional channel, the Alpha. Now, the Alpha Channel is the most important thing for this process because this basically tells everyone which parts of an image are opaque or transparent.
As you can see on the left part of the following image, I have created a fairly simple 3D scene of two objects: one sphere and one podium below it. Everything else is basically emptiness which is shown by the red color that fills the rest of the image (normally the background color would be black but I changed it to red for this tutorial). And as you can see on the right part of the image the Alpha channel is stating the same, because in this case white means opaque and black means transparent (or non-existent):
So to remove the red parts of the image I use my favorite process in which I simply use the Alpha Channel to do the selection work:
- In the Layers window I double-click on the Background Layer of the image to change it to a layer
- Then I go to the Channels List and select the Alpha Channel. The image now shows blacks and whites
- I click on “Load Channels as Selection” which can be found in the lower end of the channels window and looks like a circle. This causes Photoshop to select everything that is white
- Next I invert the selection (Ctrl+Shift+I) because I want to delete the background, not the foreground
- Lastly I go back to the RGB view of the image and hit “Delete” on my keyboard.
Voilá, the background is deleted and in theory the objects should now be placed perfectly in front of my new background layer which I created afterwards. But, no. There is a small problem:
The objects that I masked have a small red line surrounding them that more or less ruins the whole effort since it is impossible to use this “thing” for a good compositing.
Short Insert: The Remove Black/White Matting Problem
As I already wrote Photoshop does indeed have many ways to accomplish one goal. Besides the process I have used above there is a rather simple approach to seemingly solve the line problem. It can be found in Layer -> Matting and is called Remove Black/White Matting.
Although this tool does its job quite well it in fact also creates a big problem for complex compositing. Since this tool simply deletes all pixels with either black or white color in them it as a result reduces the size of the image parts you want to use. As this is something no true 3D professional should want to achieve I wouldn’t recommend using this technique.
Ok, let’s head back to the image with the red lining around the objects.
The Reason for this Failure
So, what has happened? Why is this red line showing up although everything was seemingly done correctly?
To clarify this strange behavior it is a good idea to look a little bit closer to the 3D image.
By extremely zooming in to this red line it is now possible to see that the pixels in question don’t have all a constant opacity. In fact we have a thick “border” (with red at its highest peak in the middle) while the outer transparency increases with each pixel. This means that the selection was not the problem as it correctly interpreted all brightness values of the Alpha Channel.
But why is the line showing up? Well, when we go back some steps and look at the original image you can see that the border of the sphere itself is blurred. This also applies to the Alpha Channel. Both the RGB part and the Alpha part share the exact same blurred borders.
Photoshop doesn’t know that the red is in fact some kind of transparency or a background. We know this but Photoshop doesn’t. Photoshop only sees pixels and it couldn’t care less about their color values. So when I load the Alpha as a selection and hit delete, Photoshop does exactly this: it deletes all pixel that matter BUT applies some transparency to those pixels that are not “fully” selected as the selection itself – because of the blurring/anti-aliasing in the Alpha – wasn’t at 100% everywhere. It is like working in the Quick Mask Mode.
In other words what I have here is a foreground plate that has some “transparency”/blur (it fades into the red background) and an Alpha Channel that also has some “transparency”/blur (it fades into black). It is a double game. So, when I for example delete a slightly red pixel with a half selection I get a slightly red pixel that has a 50% transparency. By taking all pixels into account it is obvious what I get: a red line.
Premultiply
By going from Photoshop to your 3D application of choice it is possible to exactly spot the source of the red line problem that is called premultiply.
I have to admit that I myself don’t fully understand the technical aspects of this process but my current understanding is this (please correct me if I am wrong):
understanding is this (please correct me if I am wrong):
When you hit the render button, the engine normally calculates the image with all the lights, colors…and anti-aliasing. This in itself is no problem. But when the engine wants to calculate the anti-aliasing on a border between an object and a background (in other words: nothing) it gets quite problematic. This is where the premultiply comes into play. The engine now simply renders the border of the objects without any anti-aliasing and then calculates the Alpha/transparency of the border. After this is finished the engine multiplies those two values for each pixel and voila you have a nice smooth border.
Ok, this one doesn’t sound very convincing yet the problem solver is nevertheless at hand and which you can find right in your 3D application of choice.
The working and fast solution
Since the render engine does add some anti-aliasing to the border of the objects and which gives us all the headaches as it adds “transparency”, the most simply and straightforward solution is this:
Search in the rendering options of your 3D application of choice (“Premultiply with Alpha”, “Alpha Premultiply”, “Premultiply“) and turn it off.
Yes, that’s it. That’s all you have to do. (Quite a lot of text to tell you this, eh?)
So, when I go back to my 3D example scene and turn off Premultiply I get this result:
As you can see the objects now have some very jaggy outer edges yet the anti-aliasing is still working everywhere else. In a way you can say that this is their true appearance on the selected/native pixel level resolution – at least on the borders next to nothing.
When I now use the described process from above to mask this image (so I can use the result for compositing) I will get this:
Now that is what I call a good result. As you can see the border became smooth and there is no red to be found anywhere.
And you now know why: because there is no precalculated blurring that faded the edges with the red background.
