Utilizing Baked Shader Outputs In Your Textures

[Chosen Few]

Someone sent me a PM this morning, asking about Autodesk Turtle's various shader output options.  I thought I'd share my reply here, in case others have similar questions.

If you don't have Turtle, that's OK.  You still may find this post useful, for general information.  The principles are universal to all (good) renderers.  If you're using Mental Ray, or what have you, you'll be able to do all the same things; you'll just have to push a few different buttons. 

My point here is not to say how great Turtle is (although it is), but rather to express that baking each of your shader attributes to a separate map can be a really powerful way to create great textures.   It happens Turtle makes this supremely easy to do, since it's the only renderer that was was designed specifically for baking, from the ground up, and as such, its interface happens to be incredibly baker-friendly.  If you're using something else, you may need to hunt and peck for some of the options I'll be talking about, but assuming your renderer is worth its salt, they'll all be in there somewhere.

Below is an image of Turtle's Baking tab, with the Shader Outputs checklist exposed:

 

 

 

As you can see, just check the box(es) for whatever type of map(s) you want, and Turtle will spit them right out for you.  It's really slick.  Again, if you're using a different renderer, it might take a few more clicks, but at least now you know the terminology to look for.

The primary use for this kind of mapping is for 3D engines that are more sophisticated than SL.  In those engines, you can create shaders that can use literally every single one of these maps as inputs to control their respective attributes, for spectacular dynamic effects.  In SL, however, we're limited to just three kinds of maps: diffuse (color), normal, and specularity.  So, if we want to use any of the others here, we have to fake it, by baking them into our textures.  That's what we'll be talking about here.

One way to do it is by compositng the maps together in Photoshop.  While PS won't necessarily yield fully accurate results, since it's not technically a material compositng program, you can still use the maps together in various ways to create good looking textures for SL.

 

The specific questions the person asked in regard to this were, "What should the precise order of these layers be when assembled in Photoshop? Also, what are their blending modes?"

As worded, the questions seem to imply the author is under the impression that there's a strict set of rules one must adhere to in order to get good results.  That's not actually the case.  If you bake the shader outputs to separate maps, there's no specific order in which they must be composited back together, and there are no set rules about what blending mode must be used with each type of map.  These things wll vary every time, depending on the particular imagery, and how you want it to look.

To understand how to use the various shader output maps, it's of course helpful first to know what they all are.  Here's a quick definition of each:

Full Shading - Adds up all the other outputs, to produce a single full color image, including all lighting effects, colors, etc.  This is the fastest and simplest way to bake.

Illumination - Maps only the effects of direct lights shining onto the model's surface.  For example, shine a spotlight onto the middle of plane, and you'll end up with a white circle in the middle of an otherwise black map (assuming that's the only direct light source in the scene).

Indirect Illumination - Maps only the effects of indirect lighting on the model's surface.  This includes things like global illumination and final gather.

Albedo - Maps the amount of light that can be bounced off of a surface.  In RL, when light hits a surface, some of it gets absorbed, and some of it gets bounced back.  A perfect mirror would have an albedo value of 1 (pure white), since all the light that hits it would reflect right off of it.  On the opposite end of the spectrum, a black hole has an albedo of zero (pure black), since no light can reflect off it at all.  Everything else in the universe falls somewhere in between (shades of gray).

Diffuse - Maps the actual color(s) of the surface, without any lighting/shading effects.  You can think of this as equivalent to full bright in SL.

Specular - Maps the intensity of specular highlights on the surface.   

It's important not to confuse specularity and reflectivity. A lot of people in SL tend to talk about them as if they're the same thing, but they're not.  They both contribute to "shininess" in different ways.

A good example for noting the difference is brushed metal.  It tends to be very specular, but not very reflective.  Shine a light on a brushed metal surface, and you'll see a nice concentrated highlight shining back at you.  That's a specular highlight, one of the defining characteristics of all metalic surfaces.  Because brushed metal is not reflective, you won't see your face in it like you would in a mirror, but because it is specular, you do see concentrated highlights.

Ambient  - Maps the amount of ambient light that can escape from surface.

Incandescence - Maps the amount of light emitted from self-illuminating surfaces.  A red-hot metal rod, for example, would have a high incandescence, since it lights up all by itself, regardless of external lighting.

SSS - Maps the sub-surface scattering of light in a semi-translucent surface.  A good example of sub-surface scattering in real life can be seen in human skin.  When light hits your skin, it passes through the outward surface, and then gets bounced around at irregular angles, before finally exiting the surface at a different point.  This is what causes skin to have that nearly indifinible quality about it that is so hard to replicate in artwork.

In any 3D application, SSS requires special shaders, specifically designed to utlize it.  Needless to say SL is not nearly sophisticated enough for that. 

SSS is of little to no value in Photoshop, since it's a dynamic effect that requires three-dimensionality in order to work.

Reflections - Maps the reflectivity of the surface.  Pretty self explanatory.

Refractions - Maps how far light is bent as it passes through a surface.  For an easy real life example, put a straw in a glass of water, and take a good look at it.  The part of the straw that is below the water's surface will appear to be at a different angle from the part above, as if the straw is bent.  This is because water bends light.

SL does not support refractions.  This is one of the reasons it's very difficult to create realistic glass, ice, etc. in SL.  In RL, we recognize that those things have depth because of the way they refract.  But in SL, that never happens, so it always looks unrealistic.

It's unlikely you'll find refraction maps to be of any use in Photoshop.  By definition, refraction requires three-dimensionality.

Custom - This would map any custom shader properties you've created for use both in Maya and in whatever game engine you're designing for.  If you want to get super creative, you could probably come up with custom properties that you could utilize in Photoshop in interesting ways, but I'm not going to get into that here.  Learn the existing stuff first, before you go making up your own.

For most of the outputs, alpha mapping logic applies.  White represents the maximum value of each property, black represents the minimum value, and all shades of gray are in between.  Some, of course, use full color.

 

As for what blending modes in Photoshop to use with each type of map, that depends on what kinds of effects you want to achieve. Remember, the maps weren't strictly meant to be composited in this fashion.  This is just one of the hacky little things we do to sort of work around the fact that SL's graphics engine is so limited.  Thus there really are no right or wrong answers.

That said, now that you know what the various outputs all are, some the common ways to use them are probably already coming to mind for you.  Again, there are no hard rules, but here are some general tips:

 

• In most cases, you'll want your base layer to be a diffuse map, since that's the one with the base colors in it.
• To add shading from ambient occlusion, use one of the darkening modes (Darken, Multiply, Color Burn, Linear Burn, or Darker Color).
• To add lighting from illumination, specularity, incandescence, etc.,  use one of the lightening modes (Lighten, Screen, Color Dodge, Linear Dodge, Lighter Color), or Luminosity.
• To control overall intensity via albedo, use one of the contrast modes (Overlay, Soft Light, Hard Light, Vivid Light, Linear Light, Hard Mix), or Saturation.

In all cases, you can control the intensity of the effect via layer opacity.

 

Don't feel you have to limit yourself to just one blending mode per map, by the way.  You can combine them in all sorts of ways, for additional effects.  And don't forget you can mask them any way you want, for precise per-pixel control. 

Also, don't limit yourself just to blending modes.  You can also use any of the maps as alpha channels for Photoshop's lighting effects, and for various filters.

The sky's the limit with this stuff, so experiment.

 

Whatever you do, don't fall into the trap of overthinking it so much that it prevents you from simply taking action. There's certainly no rule stating that you have to learn this stuff dry, before you even start.  The best way to learn is by doing.  So play with it, and have fun.

There's also no rule that says you have to get complicated in order to get good looking results.  There's nothing wrong with just doing a full shading bake, and calling it a day, if you like how it looks.

 

I hope people find this helpful.  I'm still out in the cold, so to speak, thanks to the new TOS, but the occasional forum post like this is still fun to write. :)

[IvanBenjammin]

Nice post, Chosen. I use Xnormal for all my baking needs, but I might have to check out Turtle. Is that included in Max and Maya?

Just wanted to add a point or two, hope you don't mind:

 

Specularity/Reflectivity:

You can control both reflectivity and specularity in SL. Reflectivity is faked in with the Environment setting, with the alpha channel in your spec map acting as a reflectivity map. Environment is only reflecting the sky so its not 'true reflection' by any stretch, but works fine for just about anything except an actual mirror.

You can also have a glossiness map (glossiness being how sharp specularity is) with an alpha channel in your normal map. You can control glossiness for the whole surface with a single value - the alpha channel is only useful if you have a material with differing values - lets say you have a mirror with a wooden frame - you want the glass to be highly glossy and reflective, with the wooden frame having a softer spec. Obviously can also be done with two different materials.

Incandescence:

One of the new alpha channel modes is 'Emissive mask', which maps to the glow value of the material. Assuming you don't need the alpha channel in your diffuse texture for something else, you can use the Incandescence bake here. If doing this, be aware however that  people viewing your stuff without advanced materials will see a partially transparent surface as they can only interpret the alpha in the default manner.

 

 

So by working with the maps you can bake out of turtle, combining some into your diffuse maps while using others for spec maps and alpha channels you can get some pretty good results.

 

[Chosen Few]

Thanks for the attitudinal points, Ivan.  Good stuff.

To answer your question, Turtle is strictly for Maya, not for Max.  It now comes with Maya 2014, which is nice.  For Maya 2011, 2012, and 2013, it was only included if you bought Maya with the Creativity Suite.  Prior to 2011, it was available separately from Illuminate Labs, before Autodesk bought the company.

Why on Earth Autodesk didn't just start bundling Turtle with Maya as soon as they acquired it, I have no idea.  It took them four years to start doing the right thing.

[IvanBenjammin]

Thanks Chosen, might be time for me to grab the latest trial version of Maya and see what's changed since the last time I used it - 2012 I think.

[Chosen Few]

Heh, I'm still using Maya 2009.  So, your 2012 has me beat!

[IvanBenjammin]

Naa, you still win. I used 2012 for a couple of weeks as part of a job test, prior to that the last version I'd used was 4.5. I have no idea how many version numbers there were between that and them changing it to year numbers....

It was the first 3d package I learnt to use, but never got to use it professionally. To this day though, a lot of Maya shortcuts and hotkeys are still muscle memory, so I customize whatever app I'm using to be as close to Maya as I can.