Layer Rendering Compositing

[looksoon-sun]

This is a repost from Maya rendering - but I think the solution might be in comping, thus the repost.

Essentially I'm not sure the best method of rendering/compinig an object that passes in front an object and behind it, or objects that are in front of one part of an object and behind another. See below.

OK, so I'm working on a project in which this particular issue will arise several times, so I want to try to plan ahead the best way to render/composite.

As an example, say I have a deer standing in a field of tall grass (fur). Because of the animation the grass is one fur system on a single plane. Imagine a camera looking at the deer from aabout a 30 degree side angle. For most of the image, the grass is behind the deer object. However, in front of the deer's hooves/legs some of the grass is in front of the deer.

Ideally I would like to render the deer and the grass separately (render layers). However, I can't figure out if I can do this and composite it so that the single fur system is both behind the deer and in front of it in particular regions. I could easily do this if it was a single frame, but I'm not sure how to go about this (if it's even possible) in after effects.

What I initially thought was possible with render layers, but have come to discover probably isn't, is the ability to render out a render layer (say the grass layer) but have the deer's shape mask out the part of the grass that the deer is in front of (without the deer actually being rendered. Does this make sense? Is there a common work around that I am unaware of?

Another example is say I have an object orbiting another. Can I render the objects separately but have the orbiting object be masked from the render layer when it passes behind the other object?

Or do I have to have two instances of the rendering within after effects and have one in front only during the frames that the object is forward and one instance in back during the frames when the orbiter moves behind the object? (I have done this in the movie you will see in my next post) But this solution won't really work for the grass example above.

Thanks in advance for any help or advice.

[y030052]

Yes, you can do both of the (slightly different) things you mention.

Simple layer rendering is easier, and it's easy to create a "holdout" or "black hole" matte for objects which should occlude the parts of your scene you're rendering. I prefer to do this by applying a Use Background shader to the objects you DON'T want to render into the RGB, but which you want to mask out the objects you DO want. Another way to do this is to change the Matte Opacity setting of the shaders on the objects you want to create a mask (set to black hole, I think).

As for your deer/grass situation, that's a little more complex as you have grass both in front of and behind the deer (not to mention that with fur you're not longer dealing with a simple shader situation so it's intrinsically a bit more convoluted). The answer to this situation is probably to render a depth pass, which creates a grayscale image with the luminance or brightness of your objects being determined by their distance from the camera. You can then take this depth pass into a compositing program and use it to determine layer order on a per-pixel basis.

There's a good free tutorial at Gnomon Online that covers a very similar situation you should check out to give you an understanding of how this process works. It's probably in the rendering section.