Artefacts in baked normal maps

[Drongle McMahon]

I need some help with improving baked normal maps.

I came across this problem while baking normal maps high poly->low poly, Selective->Active, in Blender. It's shown in the first picture by the blue arrows. The vertical edges of the iron bars in the high poly mesh are flat quads. They should have constant angle to light and camera, and thus show no variation in specular or environmental reflection, but as you can see, they have a triangular shaped reflective discontinuity. Looking at the normal map, you can see that this is actually encoded there. So it's not an SL problem, it's a baking problem. I thought it was due to some adjacent smooth shading causing difefrent normals at the corners, but making the whole mesh flat shaded still left the same (or a similar) artefact.

Note: these are not actaully the same mesh. They are used they show the effects most clearly.

I tried doing the baking in xNormal and got a similar result. So it's a general problem, not specific to Blender.

So I looked for a simple case to examine the cause. This is shown in the second picture. The mesh at the top (a), just two quads, is the high poly mesh. It is sloping at 60 degrees towards the low poly mesh, which is a simple plane quad. If the UV map has exacty the same angles as the mesh (b), then the baked normal is constant over both quads. However, when the angles are distorted by moving vertices along the vertical axis of the UV map, the right hand quad gets the wrong normals (c-e). The symetrical distortion (c) is similar to that in the normal mapping of the edges of the crate, and has the similar effect of making two triangles with different normals.

However, if the UV map is rotated, so that the vertex displacements are now in the horizontal axis, then the identical distortions don't introduce incorrect normals at all (f-g). In the crate, this is evident in that only the vertical edges have the artefact. The horizontal edges have acceptably constant reflections, in spite of some of their edges being smooth shaded. However, the baked normal of the undistorted rectangle is not the same as before the rotation. All this suggests  to me that the artefacts are somehow the result of the way the tangent is calculated when the UV have distortions to to avoid having too many seams. If the tangent is different in the two triangles, then perhaps a different normal, relative to it, is required to represent the same affective direction. My understanding of the calculations concerned is totallyn insufficient to explain this better, and I would be glad if someone could provide, or point to, a better explanation.
 
Meanwhile, what can be done to avoid this kind of artefact? One option is to use very fragmented UV maps in which all angles are preserved, but that is expensive in SL because seams increase LI, and it makes the use of efficient general purpose tileable textures impossible. Perhaps there are ways of limiting the distortions to one axis while avoiding over fragmentation, but that seems to me too complicated to be bearable. Does anyone have any other solutions? Is there a simple way of altering the calculationparameters instead of altering the UV maps? Have I missed something that should be obvious?

ETA: I did a whole lot more distortions. It's definately not the angles. Distortions that leave the angles unaltered can give the artefact. Others that change the angles produce no artefact. The obkly constsitent thging seems to be that moving any vertex hirizontally in the UV map has no effect on the baked normal. Moving any vertex vertically produces the artefact, unless it's an even vertical stretching and.or tranflation of the whole thing. Rotation causes the whole thing to change together, without differences between triangles.

[Scooter Hollow]

I don't know if you have more of a workload that would make it unfeasible, but for something like the example picture I'd probably just fix it in gimp. There's no details in the problem area, just a slope, so it'd be relatively easy to get the correct colors off a simpler image and fill them in.

[Drongle McMahon]

Yes. In fact that's what I did. Just some clone tool work on the red channel.  It was fairly laborious though. So I was just interested in understanding what's going on and maybe how to avoid it. Here's the before-and-after. This is a standard prim cube with the maps on it. You'll notice I also rotated the top texture ... there's another story there!

[Chic Aeon]

Not surprisingly you lost in along the way .

I DID want to say however that if you find this repeating, it may not be YOU at all. In Cloud Party (rest its virtual soul) which had tremendous tech there was a normals flipping thing going on in the uploader depending on how you produced your normal map. You needed to flip some of the cooridinants to get the angles correctly. I did this in ShaderMap2.

I have no idea if this has anything to do with your issue but since it was similar thought I would mention. It might simply be the uploader's confusion. Wouldn't be the first time .

 

 

[Drongle McMahon]

The exact effect does depend on the triangulation, which is different in Blender and the uploader, and on smooth vs flat shading. However it is still there whichever of these is. Also, the effect is seen in Blender as well as inworld. So it can't be (only) the uploader, ad it can't depend on the tangent basis (as far as I recall, that is the same in Blender and SL, but I can't find a reference.)

[MIstahMoose]

It might be worth posting this to blenderartist.org so you don't have to laboriously go through that process again. I personally do not know enough about normals to suggest what I am thinking without sounding...silly.
So I recommend asking people much more familiar with blender and its normals...And of course posting back here if you do find out!

[Astrid Kaufmat]

indeed shadermap2 is really great program for normal maps It also allows you to do some usage of it even in not licensed version.

another program you may use is free and is x-normals (it's a program used for game engine so it's studied and made specifically for that purpose)

The top would be Zbrush (compared to other programs the way it makes displacement would make your pc work less so it's really good on old machines).

Mudbox from autodesk also makes easy normalmaps

ndo2 if you have photoshop is a good tool too especially combined with xnormals,tho it's not free.

Blender did many steps forward compared to the first versions I tried, but it's still a not mature program that would make you feel the lack of things once that you've got the taste of better prorams. All another world no words to describe the difference.

[Dora Gustafson]

It looks to me like you give the answer yourself in the simplified example.
A quad in blender does not necessarily have all 4 corners/vertexes in the same plane.
It is 'flat' only in special cases.
The quads tend to become more and more distorted the more you work on a model

When a quad is not 'flat' the vertexes define two 'flat' triangles and I guess this is what you are up against

:smileysurprised::smileyvery-happy:

[Aquila Kytori]

Hi

Llike Chic I didn't understand what you were doing in the second image of your first post.

 I  Googled "triangular shaped reflective discontinuity" , seems your the only person in the history of humankind to have ever have used that combination of words;   

But I did come across this tutorial which explains some problems that can appear when baking normal maps and he does explain something about "Discontinuities". Don't know if they are the same as yours though .

http://www.chrisalbeluhn.com/Normal_Map_Tutorial.html

 

I Don't understand either why you had to hand paint (clone) the Normal map to get those inner surfaces right , couldn't you have just set those faces to flat shading and rebake ? I think i'm missing what you are trying to explain .

This where we really miss not having Chosen Few around these days so  Probably best to follow MistahMooses advice on this .

 

 

[MIstahMoose]

Might even just read through this thread here: http://www.polycount.com/forum/showthread.php?t=81154

 

Tons and TONS of information on normal maps. I just remembered it today when I woke up
I think I'll take a gander at it and refresh my normal map knowledge before my next project

 

ETA: Also it could be beneficial to look into a post-normal map program such as NDO http://quixel.se/ndo/

Thus you can bake, and proceed to fix and add as much as you'd like post-normal bake.  Albeit it is a pretty penny to obtain, very valuable if you make a lot of normal maps. time is money and money is time

[MIstahMoose]

---

Astrid Kaufmat wrote:

indeed shadermap2 is really great program for normal maps It also allows you to do some usage of it even in not licensed version.

another program you may use is

free

and is

x-normal

s

(it's a program used for game engine so it's studied and made specifically for that purpose)

The top would be Zbrush (compared to other programs the way it makes displacement would make your pc work less so it's really good on old machines).

Mudbox from autodesk also makes easy normalmaps

ndo2 if you have photoshop is a good tool too especially combined with xnormals,tho it's not free.

Blender did many steps forward compared to the first versions I tried, but it's still a not mature program that would make you feel the lack of things once that you've got the taste of better prorams. All another world no words to describe the difference.

---

I do not believe this problem occurs due to what program he is baking with.

 

I'll have to disagree with your statement about Zbrush. It really isn't that great of a normal map baking program, especially when you need to bake high to low. Yes it is easy on your computer with how it renders polygons, no it is probably not going to get you the best results. You'll notice that most professionals don't actually bake in the programs they model in, because most of those programs are focusing on the modeling more than the baking (I don't know why they don't do both)

Mudbox, like Zbrush is more focused on modeling, it can produce nice normals but isn't optimal.

Blender does an okay job, benefits that you can bake normals from textures (though this is features in most programs anyway). Another Pro is that it is easier to import a high and low poly to work with and line up to bake high-low normals. Zbrush fails this and mudbox fails this as well (Fails as in you cannot bake a high to low map in zbrush nor mudbox (unless they added it into mudbox recently or I was unaware.) )

I have no experience with shadermap2

 

I do recommend ndo2, Great program. AMAZING post editing

 

I do recommend Xnormal, amazing normal bakes and was the industry standard for awhile until Ndo2 and substance designer hit the world. Now no one knows which is the more preferred.

[Drongle McMahon]

Thanks for answers everyone...

Astrid: As I said, xNormal gives the same effect with the same UV map on the high LOD mesh and just using ray projection. So it's not just a Blender problem. From what I find below, it seems to be something to do with a difference in projection into the UV plane and onto the low poly surface. So using a cage in xNormal might well do a better job, and Blender doesn't have cages for baking (I think). I didn;t try that yet, but I probably should.

Dora: The geometry in the simple example is completely flatm with both quads in the same plane, It's only the UV map that changes. So, although what you say is true (and the source of many problems) it doesn't apply here.

Aquila: But I want it to be smooth. Also, the simple example is all flat shading. If I set it to smooth, the artefacys still appear, but with fuzzy gradients instead of sharp trasitions. In the crate model I still get the artefact with flat shading throughout. That surprised me too. If you make the UV map by projrct from view in the same direction as the rays will go for the bake, then the artefacts disappear. Perhaps your UV map was made that way, or is close to the same thing? (Yes. Losing Chosen was one of the worst effects of the new ToS!).

MIstahMoose 1: Excellent thread. I read a lot of other stuff at polycount too. I think it's close to the same problem. The edges of my bars were sloped so that they didn't fail to project, but in retrospect, they weren't sloped enough. Because they are too steep, UV mapping them with the same projection that the rays will take in the bake completely removes the artefact, but it leaves them with very thin striips of the UV and normal maps. That means the bars look very thin and the specular reflection is very narrow (see pic below). The compromise would be to make the slope more gentle, so they look thicker, sacrificing the correctness of the angle for specular reflection.

MIstahMoose 2: Although I do have photoshop, I prefer not to use it for SL because I am intersted in seeing what can be done with free tools here. Unfortunately, that excludes most of the nicest normal map tools, including nDo2. One day I will probably give in.

As detailed above, I changed the UV map so that it was the same projection of the surface that was going to be used in the bake. That results in very thin bars, but avoided the arteface. Using more goemetry in the low poly, so that the bars had real thickness, would look much better, but the object here was to make csomething applicable to a prim cube. Here is the result I got (I also flat shaded the top surface of the bars to make them thicker. It will have to ndo for now.

My original purpose here was to see how to make it look metallic under local light and daylight. This involves playing with the alpha channels of the normal and specular maps. I think I am getting nearer.The other thing I did was to add some noise to the parts of the normal map representing the bars, in the hope of getting some specular refelection over a wider range of angles.

Meanwhile, my mathematical curiosity males me still want to understand the behaviour in the simple example. Maybe I need to head over to BlenderArtists.

[Dora Gustafson]

---

Drongle McMahon wrote:

 

So I looked for a simple case to examine the cause. This is shown in the second picture. The mesh at the top (a), just two quads, is the high poly mesh. It is sloping at 60 degrees towards the low poly mesh, which is a simple plane quad. If the UV map has exacty the same angles as the mesh (b), then the baked normal is constant over both quads. However, when the angles are distorted by moving vertices along the vertical axis of the UV map, the right hand quad gets the wrong normals (c-e). The symetrical distortion © is similar to that in the normal mapping of the edges of the crate, and has the similar effect of making two triangles with different normals.

 

However, if the UV map is rotated, so that the vertex displacements are now in the horizontal axis, then the identical distortions don't introduce incorrect normals at all (f-g). In the crate, this is evident in that only the vertical edges have the artefact. The horizontal edges have acceptably constant reflections, in spite of some of their edges being smooth shaded. However, the baked normal of the undistorted rectangle is not the same as before the rotation. All this suggests  to me that the artefacts are somehow the result of the way the tangent is calculated when the UV have distortions to to avoid having too many seams. If the tangent is different in the two triangles, then perhaps a different normal, relative to it, is required to represent the same affective direction. My understanding of the calculations concerned is totallyn insufficient to explain this better, and I would be glad if someone could provide, or point to, a better explanation.

---

One thing I don't understand is why you make the UV map from the high poly mesh?

As I understand it, it should be an UV map for the low poly mesh

The result is supposed to go on the low poly mesh in the end anyway

I can not reproduce the error when I use UV map for the low poly mesh

The UV map in this case becomes one simple rectangle

I am a beginner on this and like to learn

The tutorials I followed both made the UV map from the low poly mesh and that is why I ask why you do it differently

:smileysurprised::smileyvery-happy:

[Kwakkelde Kwak]

---

Dora Gustafson wrote:

One thing I don't understand is why you make the UV map from the high poly mesh?

As I understand it, it should be an UV map for the low poly mesh

The result is supposed to go on the low poly mesh in the end anyway

---

For simple normal maps you don't need any UV mapping on the high poly mesh. If you want to project more than just normals (specular, diffuse etc), or if you want to project a high poly mesh with bump, displacement or normal maps applied to it, you certainly need to UV map the high poly too.

[IvanBenjammin]

Artifacts like this are usually a result of the high poly mesh lacking enough normal data to bake correctly. Often, the quick fix is to fix it in PS/Gimp, but sometimes (depending on the topology of your HP model) you can simply increase your subdivisions to fix the issue. Alternatively, you can do it the arduous way of beveling/chamfering all the relevant edges.

Here's a really basic example. Its similar to your mesh I think - hopefully it illustrates what's going on

The High poly mesh, without subdivs is just a plane with an inset and extruded face in the middle. The Low poly is a single plane. All baking done in Xnormal.

The 'hard' and 'smooth' bakes are self explanatory. The artifacts are even worse on my mesh than yours, but completely disappear with hard edges.

'Smooth w/ Beveling' has smooth shaded normals, with a simple double edge bevel around all internal edges. This seems to eliminate some artifacts, but makes others more obvious.

'Smooth w/ Subdiv' is the original mesh without beveling, subdivided several times without smoothing. Looks better, but the shading in the corners is still a little ill-defined.

And finally, 'Smooth w/ Beveling and Subdiv' is the beveled mesh, subdivided without smoothing. Its not a great example actually because the bevelling isn't strong enough (needs more than 2 edges to really define the hard corners), but artifacts are gone.

I should mention that I'm not using Blender so I can't help with how to smooth and bevel stuff, though I'm sure its straightforward.

[Drongle McMahon]

AHA! You are right. I've been baking them the wrong way round without realising it. I had obviously completely misunderstood the meanings of "selected" and which was "active", so that in my mind they were the other way round!  (i.e. Selected was the one you were working on) Amazing it worked at all! I should have realised because I knew the high poly UV should not matter. As Kwak said, I had mapped it to add some roughness with a texture feeding into the normals. Now I have to admit to being thoroughly stupid. :matte-motes-sick: The simple experiment all makes sense now.

Of course, it works perfectly when it's done the right way round ...

But why did xNormal give the same artefact? Never mind, I don't want to know!

[Drongle McMahon]

Thanks Ivan. As you see, it was all just my stupidity. I just have to hope hope all the information provided by others will be as useful to a wider audience as it was to me. I did learn a lot, even what "selected" means in Blenderese!

[IvanBenjammin]

As long as someone learns something, its all worthwhile

 

[Indigo Mertel]

@MistahMoose, can you elaborate on Substance Designer on how it fares with normal maps? I have the program but I am still learning how to use it. As far as I am aware SD doesn't let you bake normal maps from high to low poly meshes.

[MIstahMoose]

Export both your low poly and high poly as .fbx and then drag and drop them into the program. Right click the low poly and hit bake model information. Select normal map from mesh. Then click the "High definition mesh" under that menu and select your high poly mesh model. And a cage file if you have one as well.

Jerc just added 8 new tutorial videos to the Allegorithmic channel(like almost literally yesterday). Check those out, Visual is better than my 6am groggy explanation

http://www.youtube.com/channel/UC-toy9WMImypmLAiU9h_SzQ

Also: Substance Designer Master Thread at polycount has a lot of useful information scrambled inside of it

http://www.polycount.com/forum/showthread.php?t=129560

ETA

I like baking in substance designer because I find it easier to handle than most other programs (cough -xnormal- cough)

It also bakes with higher quality and precision than I've seen in a lot of programs as well, I think the Quixel programs and of course Xnormal are the only two that can compare. Though Quixel's ddo and ndo are really really laggy :\ And Xnormal's UI makes me cringe

[Indigo Mertel]

Wow! Thanks a lot for this information. I really was not aware I could bake normal maps from high to low poly models. This adds even more value in my investiment in SD...

I know about the new videos, I closely follow SD news and even share them on social networks as part of my "social curator duties" but I have only watched the first video so far. Thanks for the tip. The only gripe I have with SD is that the learning curve is steep and the filters are not adequately documented. In fact, I suggested the folks at Allegorithmic to create a wiki open to the user contribution as we have here for SL. They expressed a mild interest in it so hopefully we''ll see something in the future.

I know what you mean with xNormal. The interface is... erm, weird.

[Indigo Mertel]

Ok, this thread picked my interest as I've also run into some problems in baking normal maps with Blender. I did a search to find more information on the web and happened to find this video tutorial. Can anyone tell me how effective is the technique explained in the video in relation to SL? Is this the proper way to resolve shading issues? Does this technique increases the LI of models?

Please note the tutorial is only 9 min. long despite the 18 min reported. The author complained the length was doubled when the video was uploaded to YouTube.

[Kwakkelde Kwak]

---

Indigo Mertel wrote:

 Is this the proper way to resolve shading issues? Does this technique increases the LI of models?

---

Which shading issues? The ones that are the result of trying to bake rounded edges somewhere not on the edge of UV islands?

The technique of using smoothing groups instead of split edges makes no difference at all. A vertex that's shared between two smoothing groups has two vertex normals, one for each group. SL doesn't support multiple vertex normals, so the vertex is split by the uploader. For example, a mesh cube in SL is not a cube at all, it's six squares pushed together at the edges.

[Masami Kuramoto]

If your lowpoly mesh has split edges, you have to split its UV layout along those edges as well, but that doesn't mean that you have to split the model if there are multiple UV islands. What the guy did in the video is exactly backwards.

The most important thing to understand about baking in general is that the projection from highpoly to lowpoly always occurs along the interpolated vertex normal of the lowpoly mesh. Interpolation stops at split edges, which is why you can't get a seamless bake on a model with split edges unless you use a cage or a proxy that is entirely smooth. If your model has flat areas, you have to insert additional control loops to keep the projection flat as well, otherwise there will be distortions towards the edges of those flat areas.

If you use xNormal for baking, you have to insert those control loops into both the lowpoly mesh and the cage (since the topology of the cage must match the topology of the lowpoly mesh). If you bake in Blender, only the proxy mesh needs those control loops, so you can save a few vertices in your lowpoly model.

[arton Rotaru]

To get nice shading from Blender it's somewhat necessary to mark edges sharp, and/or add control loops. This can increase the Download Weight in SL though. However, if you have your sharp edges where you have UV splits anyway, the hard edges comes with no extra cost, because the vertex is already duplicated from the UV split. Though, to get nice shading you might want to mark edges sharp which aren't usually on UV splits. These will increase the Download Weight in SL then. (And, like Masami mentioned, you have to split the UV's on these hard edges as well then.)

Even if the SL shaders  tangent basis would exactly match the one used by your baking program, which "could" make hard edges obsolete altogether, there's still the problem of people who don't have Advanced Light Model turned on. So those wouldn't see the shading from the normal map, but just an overall smooth shaded "something" anyway. Depending on what it is, it can look quite bad then. So we should take care of proper shading ourselfs, even if it adds a little extra cost.

And yes, you should bake using a cage.