Weird download impact calculations on mesh upload

[arton Rotaru]

3 minutes ago, Toastfan123 said:

but I honestly do like the extra detail for closeups ^^

Stone p*rn? 😀
 

[JackTheOtterFox]

2 minutes ago, arton Rotaru said:

Stone p*rn? 😀

Well, I mean, it's my first proper creation, so I might as well give it all I can

[ChinRey]

18 minutes ago, Toastfan123 said:

Yeah, I thought it would be, well, more worth it, but the actual influence of the bump and specular maps are almost unnoticable, so I'll drop it down to 8 textures
And I could have probably made the textures 512x512, but I honestly do like the extra detail for closeups ^^
(And also I don't want to upload them all again :P)

Just promise you'll only use them yourself and not distribute them.

I signle copy of an overtextured item in a private place with not much other content isn't doing any harm.

The problem is only when texture (and triangle) heavy content is aggressively marketed, sold all over the grid and placed in scenes with lots of other equally heavy content problems arise and people start complaining about lag, blaming it on everything and everybody except those objects.

That being said, if you really mean 24 1024x1024 textures, 24 normal maps and 24 specular maps, that adds up to 288 MB of VRAM. In other words, four obejcts as texture heavy as that are enough to overload a regular home computer, 15 of them are enough to overload an average (according to Steam) game computer. That's food for thought.

Remember what I said about good old repeating textures?

There is a good rule of thumb here: half the details you add to the mesh and the textures baecause they seem so important while you build, you will never notice again after the object is uplaoded and put in place. Three quarters of the details will never be noticed by anybody else.

Edited February 27, 2019 by ChinRey

[ChinRey]

30 minutes ago, arton Rotaru said:

• Trying to drop a texture onto an object, but put it on a plants alpha plane, which happened to be infront of your object. And this plant is No Copy, and isn't your own creation, so you don't have the original texture to fix it afterwards.

I can't do that, I only use my own plants. But trying to remember which texture I used and where I stored it, can be frustrating enough.

[JackTheOtterFox]

1 minute ago, ChinRey said:

Just promise you'll only use them yourself and not distribute them.

I signle copy of an overtextured item in a private place with not much other content isn't doing any harm.

The problem is only when texture (and triangle) heavy content is aggressively marketed and sold all over the grid and placed in scenes with lots of other equally heavy content problems arise and people start complaining about lag and blame it on everything and everybody except the content.

That being said, if you really mean 24 1024x1024 textures, 24 normal maps and 24 specular maps, that adds up to 288 MB of VRAM. In other words, four obejcts as texture heavy as that are enough to overload a regular home computer, 15 of them are enough to overload an average (according to Steam) game computer. That's food for thought.

Remember what I said about good old repeating textures?

I meant 8 Sets of Diffuse, Normal and Specular ^^
So that's 3 channels each for the Diffuse, 3 channels for the Normal and 1 channel for the Specular.
With compression that should be something around ~3000KB per Texture set, so ~24.000KB or 24MB in total, which is still fine since the textures will be instanced for all parts of the walkway I suppose ^^

If I'd distribute something like that, I would probably make multiple versions, like one with 512x512 and one with repeating textures, allthough they would look quite bad on my current model since it's not designed for that.

[arton Rotaru]

3 minutes ago, ChinRey said:

I can't do that, I only use my own plants. But trying to remember which texture I used and where I stored it, can be frustrating enough.

It's even more fun if you do it on someone elses plants, to whom you have edit rights to.

Edited February 27, 2019 by arton Rotaru

[ChinRey]

8 minutes ago, Toastfan123 said:

I meant 8 Sets of Diffuse, Normal and Specular ^^
So that's 3 channels each for the Diffuse, 3 channels for the Normal and 1 channel for the Specular.
With compression that should be something around ~3000KB per Texture set, so ~24.000KB or 24MB in total

That's not how VRAM works though. For a start, there is no such thing as 1 channel graphics files in SL, it's either 3 channel, 24 bit (without alpha) or 4 channel, 32 bit (with alpha). And for some reason, according to Firestorm's VRAM readout at least, textrues are always assigned enough VRAM for an alpha channel whether they need it or not. As for compression, there's no such things in the VRAM. It can't be. It can't be, the whole idea is to have an ultrafast memory directly conencted to the gpu. There's no time for decompression there.

Edited February 27, 2019 by ChinRey

[JackTheOtterFox]

Just now, ChinRey said:

That's not how VRAM works though. For a start, there is no such thing as 1 channel graphics files in SL, it's either 3 channel, 24 bit (without alpha) or 4 channel, 32 bit (with alpha). And for some reason, according to Firestorm's VRAm radout at least, textrues are always assigned enough VRAM for an alpha channel whether they need it or not. As for compression, there's no such things in the VRAM. It can't be. It can't be, the whole idea is to have an ultrafast memory directly conencted to the gpu. THee's no time for decompression there.

Wait, if I upload a texture with only one channel it won't add more channels to it, will it? I mean, there is absolutely no reason to have more than one channel for a specularity map, and that's not new, that will be the case ever since SL introduced specularity maps, whenever that was. Does it really just inflate the textures? o.o

[ChinRey]

11 minutes ago, Toastfan123 said:

Wait, if I upload a texture with only one channel it won't add more channels to it, will it?

I'm not sure if JPEG 2000 even supports 8 bit color pallettes but I'm farily sure SL doesn't.

All image files are converted to JPEG2000 with lossless compression btw, regardless of what format you upload them in.

SL does support colored specular maps btw. It's potentially useful for metallic surfaces. I often simply use the texture as a specular map for those, since they usually have the right shading and color.

Edited February 27, 2019 by ChinRey

[JackTheOtterFox]

5 minutes ago, ChinRey said:

I'm not sure if JPEG 2000 even supports 8 bit color pallettes but I'm farily sure SL doesn't.

All image files are converted to JPEG2000 with lossless compression btw, regardless of what format you upload them in.

How can I check how much memory (or video memory) my textures are actually consuming?
You mentioned some VRAM readout?

[ChinRey]

1 minute ago, Toastfan123 said:

How can I check how much memory (or video memory) my textures are actually consuming?
You mentioned some VRAM readout?

It's a fairly new Firestorm only feature. Right-click on object and select "Detais" - or was it "Object details".

I'm not sure how correct that number is. As I said, it does show all textures as being 4 channel which does seem a bit strange. @Beq Janus is a Firestorm developer so it's possible she knows even though this wasn't ehr project.

[JackTheOtterFox]

Okay, just found it. It tells me my object's textures are taking up 32686KB of VRAM, which honestly seems to be okay for me. Is that including all textures, so diffuse, normal and specular?

[JackTheOtterFox]

1 minute ago, ChinRey said:

It's a fairly new Firestorm only feature. Right-click on object and select "Detais" - or was it "Object details".

I'm not sure how correct that number is. As I said, it does show all textures as being 4 channel which does seem a bit strange. @Beq Janus is a Firestorm developer so it's possible she knows even though this wasn't ehr project.

Okay, updating my prior statement. That window tells me my object is taking up 90.112KB of VRAM, which is a lot more...
(It's a really nice feature though, I'm learning more and more about SL :D)

[ChinRey]

1 minute ago, Toastfan123 said:

Okay, just found it. It tells me my object's textures are taking up 32686KB of VRAM, which honestly seems to be okay for me. Is that including all textures, so diffuse, normal and specular?

Not if you still have 24 of them.

If you know the textures you used, it's easy to calculate yourself. Each 1024x1024 is 4 MB.

[JackTheOtterFox]

2 minutes ago, ChinRey said:

Not if you still have 24 of them.

If you know the textures you used, it's easy to calculate yourself. Each 1024x1024 is 4 MB.

Textures are still instanced though, right?
So having that object or others using the same textures 20x in the scene wouldn't increase the VRAM used, am I correct with that assumption?

[ChinRey]

1 minute ago, Toastfan123 said:

Textures are still instanced though, right?
So having that object or others using the same textures 20x in the scene wouldn't increase the VRAM used, am I correct with that assumption?

Yes, you are right.

[JackTheOtterFox]

1 minute ago, ChinRey said:

Yes, you are right.

Okay, that's something at least...
I suppose the smartest thing would be to drop the texture resolution to 512x512, which should 1/4 the VRAM usage of the object, which is probably a good start.

Looks like you are right, at least from my few tests it looks like it is inflated up to 3 channels, which is kind of not so great (especially because it's a specular map...) but oh well. I don't really want to completely remove bump and specular map, but that leaves me with no option to further decrease the intensity.

If I'd use tiled textures it wouldn't match the UV mapping and I'd see seams, which is something I absolutely want to avoid at all cost.
So for now I suppose that's it, at least if noone has better suggestions.

(And I was actually planning to re-purppouse these assets as something to be soled on the marketplace, but I suppose I'll just get kicked in the butt if I do so now...)

[Beq Janus]

12 hours ago, Toastfan123 said:

12 hours ago, ChinRey said:

That's not how VRAM works though. For a start, there is no such thing as 1 channel graphics files in SL, it's either 3 channel, 24 bit (without alpha) or 4 channel, 32 bit (with alpha). And for some reason, according to Firestorm's VRAm radout at least, textrues are always assigned enough VRAM for an alpha channel whether they need it or not. As for compression, there's no such things in the VRAM. It can't be. It can't be, the whole idea is to have an ultrafast memory directly conencted to the gpu. THee's no time for decompression there.

Wait, if I upload a texture with only one channel it won't add more channels to it, will it? I mean, there is absolutely no reason to have more than one channel for a specularity map, and that's not new, that will be the case ever since SL introduced specularity maps, whenever that was. Does it really just inflate the textures? o.o

While the viewer has support for various image formats with varying numbers of channels, by the time we are rendering it such things are lost. I'd have to dig around a lot to prove that we can even upload a single channel grey scale image these days. In theory, you can, but in practice, I am less certain, it may well get converted to 24-bit RGB colour space irrespective of what you give us.

However,  the statement that "its a specular map so it is just one channel" needs some clarifying, because that's not at all how it works, and never has been.

In second life there are 5 channels that control specularity, First we have the 4 channels of what we call the spec map. The first 3 channels (RGB) control the per texel colour reflection these get combined with the overall tint specified in the specularity settings for the material, then we have environmental reflectivity, which or more or less the equivalent of traditional "shininess" and is controlled by the alpha channel, this gets combined with the environment intensity setting. The 5th channel is stored as the alpha map of the normal texture and is more or less a roughness map. It controls the sharpness of reflections (i.e. how much light is scattered by that surface) it gets combined with the per material glossiness value. There is no way to control just an individual channel, which does cause overhead for some types of material, but that's how materials work.

This link explains it fully http://wiki.secondlife.com/wiki/Material_Data#Texture_Channel_Encoding

 

[JackTheOtterFox]

2 minutes ago, Beq Janus said:

While the viewer has support for various image formats with varying numbers of channels, by the time we are rendering it such things are lost. I'd have to dig around a lot to prove that we can even upload a single channel grey scale image these days. In theory, you can, but in practice, I am less certain, it may well get converted to 24-bit RGB colour space irrespective of what you give us.

However,  the statement that "its a specular map so it is just one channel" needs some clarifying, because that's not at all how it works, and never has been.

In second life there are 5 channels that control specularity, First we have the 4 channels of what we call the spec map. The first 3 channels (RGB) control the per texel colour reflection these get combined with the overall tint specified in the specularity settings for the material, then we have environmental reflectivity, which or more or less the equivalent of traditional "shininess" and is controlled by the alpha channel, this gets combined with the environment intensity setting. The 5th channel is stored as the alpha map of the normal texture and is more or less a roughness map. It controls the sharpness of reflections (i.e. how much light is scattered by that surface) it gets combined with the per material glossiness value. There is no way to control just an individual channel, which does cause overhead for some types of material, but that's how materials work.

This link explains it fully http://wiki.secondlife.com/wiki/Material_Data#Texture_Channel_Encoding

Oh wow, that is extremely useful to know, thank you a lot for this information!
I will definitely experiment around with that. When I read "Specular" in the viewer I was assuming it was just a grayscale texture representing the value 0-1, I didn't even think about the fact that the other channels might actually serve a purpouse. It's really nice to know that my assumption was wrong and that the channels aren't just wasted. I was too quick to come to a conclusion (and also quite tired), so my deepest apologies for that.

[OptimoMaximo]

17 hours ago, Toastfan123 said:

So that's 3 channels each for the Diffuse, 3 channels for the Normal and 1 channel for the Specular.

That would be the case IF it were a Specular Level, however SL uses what is called a Specular Color. This comes from the attributes inherited from a blinn-phong shader model, in order to give metals a proper reflection: in that shader model, metals have no diffuse component (black) and their resulting diffuse color (what you see once rendered) comes from the specular color. I did a short explanation of this in this other thread, not long ago 

 

Obviously the link pointed by @Beq Janus contains the information needed for SL, and she's absolutely correct in doing so. However i think some extra, "external" info isn't a bad idea to have 

Edited February 28, 2019 by OptimoMaximo
needless to say... TYPOS!

[Beq Janus]

1 hour ago, Toastfan123 said:

It's really nice to know that my assumption was wrong and that the channels aren't just wasted. I was too quick to come to a conclusion (and also quite tired), so my deepest apologies for that.

No apologies needed, we all learn every day if we don't then we probably weren't trying anything new.

[Ana Stubbs]

21 hours ago, ChinRey said:

That's one of the requirements for becoming a true SL builder. A few more things left to do before you're properly qualified:

• Accidentally delete your work platform 1000 m or more up in the air
• Fall off the platform whilst being so preoccupied with your build you don't notice
• Retexture a huge linkset when you only intended to change one little face of it
• Accidentally unlink a 255 part linkset
• Send you build off so far into the distance it's lost forever
• Upload a mesh with physics so heavy it overloads a sim all on its own

Did I miss anything?

Forget about autoreturn, and find your build has been returned in completely nonsensical coalescenced bundles.

Forget where you left your build, and spend hours looking for it.

Set your build to physical by mistake, and watch it explode all over the sim.

And the big one - forget to name your build. You know it's in your inventory somewhere, but so are 10,000 other entries for "Object".

[ChinRey]

2 hours ago, Ana Stubbs said:

(a lot)

Great. But this is turning into a digression and it's a topic so important it really needs its own thread.

So I made one for it: https://community.secondlife.com/forums/topic/433704-formal-requirements-for-becoming-a-real-builder/

 

[JackTheOtterFox]

Okay, just for the sake of completeness:
I'm done with this!

That was quite a journey, but I'm happy that it's done now. And this might be the first proper thing I created for SL, but at least in my opinion, it looks really good, and I'm really happy with it.

Things I learned on this adventure:
- Land impact calculation is weird and partially nonsensical
- The larger the object in actual dimensions the higher the download impact because it could in theory be seen by more people, so the server has more stress
- The more high-resolution textures there are on a mesh the higher the VRAM usage, however that does not impact the LI value
- Textures are instanced, which means that if one object shares the texture with another one, the texture is only once loaded into VRAM
- If your nose isn't directly glued to the floor you can't see the difference between 512x512 and 1024x1024 resolution textures for small objects like stepping stones
- If you try to cheat by making the LOD only a stone-shaped plane, make sure it's at the top of the model or it won't be rendered because it's clipping with the terrain
- Physics meshes are tricky - and can very quickly increase the LI. They shouldn't have unnessesary details
- Both LOD and Physics meshes will be "aligned" to the main mesh by stretching their dimensions until they fit the dimensions of the main mesh.
- To properly align these meshes, make sure their dimensions are equal by adding 0-scale triangles at two opposite corners of a box containing all vertices of all meshes
  (Blender 2.8's new 'edit multiple objects at once' feature comes in very handy for this)
- To actually use the physics mesh in-world, you need to first enable it in the object preferences
- If the dimension of the object is 0.5m or smaller in height the physics mesh will be ignored completely without a warning - whatever you do
- There is more to the specular and bump-maps than meets the eye - further investigation required
And finally, probably the most important lesson learned from this:
- Placing 115 stones in a neatly fashion can get very boring very quickly

The final model is actually made out of 5 smaller models, each of them having a LI of 1. That means that the total land-impact for the stepping stones is 5, which is a fantastic number in my opinion! It's also a number that is significantly smaller than 19, which was the original LI value that lead to me opening this thread. Optimizing models is extremely important, and can bring a lot of improvement to performance, visuals and LI - It's definitely always worth it.
I think I will actually make a marketplace item out of this, I don't think there are any ways of increasing the performance more without suffering in quality - Before I do that I will try to improve the textures though, I think especially with the new information from @Beq Janus and @OptimoMaximo I see a lot of room for improvements.

Thank you to everyone who helped me in the past few days, I really learned a lot thanks to you, and I enjoyed working on this a lot (at least for the most part), so I might continue to create assets in the future. (I'll probably wait until I can finally get onto the Beta Grid though, I don't think I want to know the total upload costs of the past few days... For that reason alone I should create a listing on the marketplace for sure xD)

Edited February 28, 2019 by Toastfan123

[Beq Janus]

28 minutes ago, Toastfan123 said:

Things I learned on this adventure:
- Land impact calculation is weird and partially nonsensical
- The larger the object in actual dimensions the higher the download impact because it could in theory be seen by more people, so the server has more stress

A great summary, and a worthwhile pilgrimage. I take a small exception to the two points quoted.

LI calculation is literal and well-defined, though it may not make sense without understanding the proper context, it is not easy to find out what it is and that gives it the air of mystique. That is just for the steaming cost, as that counts 10 times over for the physics cost element. One thing that is also very true is that the current LI calc is also old and what once held true is not quite so black and white now.

The size aspect maker perfect sense and little to nothing to do with the server. the server certainly has no additional stress due to the larger scale of an object. In the past assets were served by the region, and it could be argued that that caused more load, but that makes an assumption about people requesting simultaneously that is hard to quantify. In any case, this has not been the situation for a long time now, all mesh and texture assets are served via the CDN. However, the scale aspect makes sense without that. A large object is visible from further afield at high detail, thus a scene consisting of lots of large highly detailed objects is more complex to render than the same scene consisting of an equal number of smaller versions of the same objects because some of the smaller objects will have decayed to the lower LOD models. Fewer triangles to render means less load and faster rendering. In theory, this extends to streaming because an object that is far away only has the lower LOD model data requested. Thus the viewer downloads less data. In practice, and with modern networks, the use of CDN, and larger local cache this is of less value and that is what I think we'll see those metrics revamped under project ArcTan when it appears.

The size has an inverse relationship with physics. If you are using a triangular mesh physics shape then the smaller the triangles the more tests have to be done to determine collision. this DOES impact the server because that is where the physics engine runs. The reason for the 0.5 limit (which is rather arbitrary and possibly too large, but that's a separate argument) is to prevent the physics cost going asymptotic as the scale tends towards zero. I do struggle to justify (based on this argument here, the fixed cost for analysed hull physics but again, that's another matter for another day)

This animation tries to illustrate the problem with an extreme example. In the best case, the physics engine tests 2 triangles for intersection with the ball in order to decide if they collided. In the worst case example here it may have to test up to 1024 triangles, clearly is it a lot more work. (taken from http://beqsother.blogspot.com/2017/10/blue-sky-thinking-physics-view-explained.html)