Why do some sims utilize the GPU power and others not at all?

[Jackson Redstar]

I know this may have been discussed some in the past but lately been exploring SL 'sim hopping (on FS viewer ver 6.4.12) the EEP version, keeping shared environment on and with shaders and shadows on, setting in prefs set to High. Just seeing what's out thereand how different places look on EEP, if they are doing anything 'extraordinary' 

What I am finding and I know this is nothing 'new' but some sims you land in the GPU ramps up to a 60, 70& or more. CPU stays usually around 10-15%, and in this case the region FPS is usualy pretty good (20+ FPS often up to 40 or more) Other sims the CPU is roughly the same, but the GPU remains low, 20, 30% and FPS is rather dismal (sub 15fps) I am curious what is it that 'triggers' the GPU to ramp up and what triggers it to remain almost idle? Is it solely in SL code or is there something in windows as well that is a determining factor? Keep in mind these are almost always fairly empty sims as far as avatars go, maybe 2-4 at the most

my specs here:
 

Firestorm 6.4.12 (62831) Dec  3 2020 22:34:49 (64bit / SSE2) (Firestorm-Releasex64) with Havok support
Release Notes

You are at 146.1, 196.2, 22.0 in Free Spirit Farms located at ec2-52-26-37-138.us-west-2.compute.amazonaws.com
SLURL: http://maps.secondlife.com/secondlife/Free Spirit Farms/146/196/22
(global coordinates 205,714.0, 355,524.0, 22.0)
Second Life Server 2021-01-08.554811
Release Notes

CPU: Intel(R) Core(TM) i7-10700K CPU @ 3.80GHz (3792 MHz)
Memory: 32672 MB
OS Version: Microsoft Windows 10 64-bit (Build 19042.746)
Graphics Card Vendor: NVIDIA Corporation
Graphics Card: GeForce GTX 1080/PCIe/SSE2
Graphics Card Memory: 8192 MB

Windows Graphics Driver Version: 27.21.14.6140
OpenGL Version: 4.6.0 NVIDIA 461.40

RestrainedLove API: (disabled)
libcurl Version: libcurl/7.54.1 OpenSSL/1.0.2l zlib/1.2.8 nghttp2/1.40.0
J2C Decoder Version: KDU v8.0.6
Audio Driver Version: FMOD Studio 2.01.05
Dullahan: 1.8.0.202007261348
  CEF: 81.3.10+gb223419+chromium-81.0.4044.138
  Chromium: 81.0.4044.138
LibVLC Version: 2.2.8
Voice Server Version: Not Connected
Settings mode: Firestorm
Viewer Skin: Firestorm (Grey)
Window size: 1920x1080 px
Font Used: Deja Vu (96 dpi)
Font Size Adjustment: 0 pt
UI Scaling: 1
Draw distance: 208 m
Bandwidth: 2950 kbit/s
LOD factor: 2
Render quality: High (5/7)
Advanced Lighting Model: Yes
Texture memory: Dynamic (4096 MB min / 10% Cache / 10% VRAM)
VFS (cache) creation time (UTC): 2021-1-8T1:44:18 
Built with MSVC version 1916
Packets Lost: 101/160,135 (0.1%)
February 07 2021 07:26:29 SLT

[KjartanEno]

It could be any number of things. But with shadows enabled, and assuming also shadows from local lights, the likely culprit could be the number and area of influence of those local lights. I've rented parcels where I can be looking around with decent framerates, but when I faced a particular direction, my framerate could drop by 20 or more. One time, out of curiosity, I turned on the 'beacons' for lights and started to derender the ones that seemed to affect my framerate the most when camming in the low framerate area. Once I had a good idea of the problem lights, I relogged and derendered them permanently. As far as I know, shadows are handled by the CPU in the SL renderer. That's a lot of calculation to put into one thread, and all the while your GPU will seem to idle more.

But if you are saying your CPU AND GPU seem to idle in some sims, resulting in low framerates, I'm not so sure about the cause. I don't have that problem on Linux...

Edited February 7, 2021 by KjartanEno
more info

[Jackson Redstar]

1 hour ago, KjartanEno said:

It could be any number of things. But with shadows enabled, and assuming also shadows from local lights, the likely culprit could be the number and area of influence of those local lights. I've rented parcels where I can be looking around with decent framerates, but when I faced a particular direction, my framerate could drop by 20 or more. One time, out of curiosity, I turned on the 'beacons' for lights and started to derender the ones that seemed to affect my framerate the most when camming in the low framerate area. Once I had a good idea of the problem lights, I relogged and derendered them permanently. As far as I know, shadows are handled by the CPU in the SL renderer. That's a lot of calculation to put into one thread, and all the while your GPU will seem to idle more.

But if you are saying your CPU AND GPU seem to idle in some sims, resulting in low framerates, I'm not so sure about the cause. I don't have that problem on Linux...

ok, well, dont even get me started on local lights lol! But it would be nice to figure out why on some sims the GPU goes full beans, then on others it says 'nah, I'm on break buddy... good luck with your FPS, you're on your own here!"

[Chic Aeon]

My best guess (having nothing to do with EEP) is that the sims have a lot of textures -- as in many and big.  If you get into the habit of checking the triangle (or vertices) of the mesh and the textures on them you can easily see that some creators are using WAY over the amount of textures that will keep your computer happy.  I have been crashing a fair amount lately (black screen - reboot crashes) and often in the same sims.  NOW I have learned to derender everyone  (all those animesh pets and heavy wearables add to the issues) and that helps.  

 

While some sim owners have made new EEP skies or days, many haven't and aren't using an EEP viewer yet (and kicking and screaming about the time they will need to LOL).    So I am not too surprised that things don't seem different.  You not only need a viewer that sees EEP -- you need the assets for EEP.  I am sure you know that, just stating the obvious. 

 

Here is a screenshot from a current event showing the amount of textures on one décor (albeit large) item.  Along with the triangle count - NOTE THE TEXTURES.   That's a lot of work for your graphics card. 

 

 

And here is a whole house with individually baked textures inside and out.  Compare the two.   It isn't all that difficult to see when you multiply that on so lovely looking render mesh by hundreds of similar items, that your computer will NOT be happy.   My framerate is about five times better on my sim than it is on a typical sim and miles better than some places.  But folks want pretty and I get that. It however comes with a hefty price -- a real life one sometimes :D. 

 

Edited February 7, 2021 by Chic Aeon
spelling

[Profaitchikenz Haiku]

49 minutes ago, Chic Aeon said:

I have been crashing a fair amount lately (black screen - reboot crashes) and often in the same sims.

Does this still happen when you turn off ALM? I'm curious because it seems to me that having ALM off doesn't really affect what I'm seeing, but does seem to give me less trouble.

[Chic Aeon]

LOL. Don't know.  I have had ALM on since its inception. Just turned it off as a test and not willing to do that for more than two seconds. VERY ugly.   Turning it off WOULD keep the material maps from loading (at least it did when we did some tests so many years ago) so it would keep the texture loading down.  Most folks know more about all that than "I" do so I'll let them chime in with wisdom.   

 

There is a huge difference for me with ALM on and off.  It may depend on Windlight settingss.  

 

This is my typical usage settings.

[KjartanEno]

Turning off advanced lighting is a last resort, and something I didn't even do using Intel integrated graphics (Intel i3 6100 Skylake) because it looks ugly. Modern hardware and drivers are fully capable of using advanced lighting. When framerates dip to unusable levels, I leave advanced lighting enabled, turn off shadows, and reduce draw distance. Jackson's hardware is leagues beyond a 5 year old Intel Skylake dual core CPU with hyperthreading. His issue is more likely related to that 208m draw distance combined with local lights and shadows. In some sims most of his CPU power is idling while a single thread attempts to calculate all kinds of meshes, textures, and lighting. The difference between 128m and 208m is more than simply 80m of linear distance. It's also the additional volume that is being rendered along an expanding cone from the viewer's camera position. If you really want to view distance while keeping a usable framerate, one solution is to narrow the camera angle by pressing CTRL+0 as many times as needed. Return to the default field of view with CTRL+9.

[Chic Aeon]

1 hour ago, KjartanEno said:

 The difference between 128m and 208m is more than simply 80m of linear distance. It's also the additional volume that is being rendered along an expanding cone from the viewer's camera position. 

Agreed. And there is a setting in Firestorm that  says something like "render things only in my camera view" (I can't find it now but saw it a couple of nights ago).  I have that ON. I suspect that if it wasn't checked (or might not be in all viewers) that the computer would attempt to render items not even in camera range. Patch said something a couple of years ago when the Belli houseboats and Trads were unveiled that even the objects in neighboring regions would cause lag.  I was surprised as this didn't seem to be about 512 draw distance but just the way things were rendered generally). Again, not an expert on this.

ALSO while  the OP and I both have LOD set at 2  -- anyone having that up to 3 or 4 will definitely have more issues loading mesh. When I had mine set at 4 a few years ago I was constantly crashing in one sim  (not a reboot needed crash - the viewer just closed unexpectedly).   Not good. 

 

There are very few times that your REALLY need to see from one side of the sim to another.  I keep my draw distance at 120 at my place and sometimes turn it down very low depending on where I am headed.  It helps.  If I NEED to see further it is a very quick change. 

[arabellajones]

One thing on draw distances: the standard SL viewer has a minimum draw distance of 64m, while Firestorm has a minimum of 32m.

The LOD setting can be useful, but I reckon that if a creator says it needs increasing to see a product properly, they didn't know how LOD worked when they made it. But some stuff on the marketplace is old, and while you can get an idea of just how old it might be, and old is not automatically bad, it's not always clear.

[Jackson Redstar]

in the phototools panel under the Gen tab is the checkbox for 'Dynamically adjust level of detail' that is always on by default and can make a huge performance difference. Of course when shooting a photo or video, that needs to come off in many cases

But in all it still don't seem to explain why on some sims the GPU is mostly idle and other sims it is running 70% plus. I know Ive seen a perf boost on this much faster 10th gen i7 (though not really translated to a FPS boost), and all the sims I been to are on aws servers now

[Madelaine McMasters]

It's been ages since I experienced massive direction related FPS effects, and I didn't check GPU usage at the time, but I have the same feeling as Chic, that textures are potentially involved. I once rented a place on a sim that was near a store. The store was nothing more than a ginormous wall tiled with images of the items for sale. There were easily 500 such tiles, and each was a 1024x1024 texture. I got reasonable frame rates when looking anywhere but the store. The moment I cammed in that direction, the wheels came off my Mac.

Edited February 8, 2021 by Madelaine McMasters

[animats]

1 hour ago, Madelaine McMasters said:

There were easily 500 such tiles, and each was a 1024x1024 texture. I got reasonable frame rates when looking anywhere but the store. The moment I cammed in that direction, the wheels came off my Mac.

Yes, the renderer has real problems when even the close-range stuff won't fit in GPU memory. It's frantically trying to cope, but just doesn't have enough resources.

Don't build stores with giant walls of vendors all visible from one point.

[Ardy Lay]

10 hours ago, Madelaine McMasters said:

It's been ages since I experienced massive direction related FPS effects, and I didn't check GPU usage at the time, but I have the same feeling as Chic, that textures are potentially involved. I once rented a place on a sim that was near a store. The store was nothing more than a ginormous wall tiled with images of the items for sale. There were easily 500 such tiles, and each was a 1024x1024 texture. I got reasonable frame rates when looking anywhere but the store. The moment I cammed in that direction, the wheels came off my Mac.

Textures R Us?  I used to go there to blow up computers for fun.  It was a great place to shed stalkers!

Edited February 9, 2021 by Ardy Lay

[NiranV Dean]

Quite simple actually!

Lets establish some fundamentals:

Your GPU usage depends solely on its utilization through shaders.

Your CPU usage depends on basically everything else.

 

This means: Your CPU will always determine your framerate because it is always the bottleneck unless the GPU is overburdened and starts to become the bottleneck.

The most common GPU usage with shadows on are shadow drawing (not the actual shadow calculations, those are on CPU), the more shadow the shader has to draw, the longer the shader iterates through one pass to draw your final image, the more your GPU is used. This is commonly seen when shadows become longer as the sun sets, more shadows = more iterations = more GPU usage. This also means that more objects and larger objects casting shadows also directly result in more GPU usage as more area is being shadowed.

This means: Your FPS will solely depend on your CPU which always runs at its maximum usage (for SL) that SL is capable of handling. If there are a lot of things around that need to be done on your CPU, such as calculating shadows (a lot of grass for instance really wrecks shadow performance) you'll see your CPU usage stay the same (which is the maximum) but your framerate drop and depending on your environment and the amount of shadows needed to be drawn you'll also see your GPU usage drop (as the GPU needs less to do since it's waiting on the CPU). If you were to set the sun low or have a giant object cover the entire SIM you'd be raising your GPU usage, possibly to the point that your GPU will hit 100% and will start to bottleneck, resulting in your CPU waiting on the GPU and dropping CPU usage as result. This can go from anywhere to "seemingly" low CPU usage and low GPU usage and low FPS to the an even lower CPU usage but high GPU usage but still high FPS (due to the GPU doing most of the hard work but still not managing to keep up and starting to cut into your otherwise excellent framerate).

What this essentially boils down to is: Your CPU determines the baseline performance you can get based on the environment you are in and the necessary work to be done, any additional work cuts into your framerate, your GPU on the other hand is basically "free" resources which do not cut into your performance unless you start lacking GPU resources. Having a place with high framerate can mean that you can still get a high framerate even though your GPU is bottlenecking (due to not bottlenecking too hard, just enough to start lowering CPU usage) by simply "overdoing" GPU usage most commonly achieved with a lot of shadows.

Note that i tried to simplify this as much as possible, i'm not a graphics programmer so it might not be 100% accurate, this is solely based on some basic fundamentals that i know (such as code running on CPU under normal circumstances and shaders running on GPU) and a lot of observation over the past 10+ years.

Edited February 9, 2021 by NiranV Dean

[Jackson Redstar]

15 hours ago, NiranV Dean said:

Quite simple actually!

Lets establish some fundamentals:

Your GPU usage depends solely on its utilization through shaders.

Your CPU usage depends on basically everything else.

 

This means: Your CPU will always determine your framerate because it is always the bottleneck unless the GPU is overburdened and starts to become the bottleneck.

The most common GPU usage with shadows on are shadow drawing (not the actual shadow calculations, those are on CPU), the more shadow the shader has to draw, the longer the shader iterates through one pass to draw your final image, the more your GPU is used. This is commonly seen when shadows become longer as the sun sets, more shadows = more iterations = more GPU usage. This also means that more objects and larger objects casting shadows also directly result in more GPU usage as more area is being shadowed.

This means: Your FPS will solely depend on your CPU which always runs at its maximum usage (for SL) that SL is capable of handling. If there are a lot of things around that need to be done on your CPU, such as calculating shadows (a lot of grass for instance really wrecks shadow performance) you'll see your CPU usage stay the same (which is the maximum) but your framerate drop and depending on your environment and the amount of shadows needed to be drawn you'll also see your GPU usage drop (as the GPU needs less to do since it's waiting on the CPU). If you were to set the sun low or have a giant object cover the entire SIM you'd be raising your GPU usage, possibly to the point that your GPU will hit 100% and will start to bottleneck, resulting in your CPU waiting on the GPU and dropping CPU usage as result. This can go from anywhere to "seemingly" low CPU usage and low GPU usage and low FPS to the an even lower CPU usage but high GPU usage but still high FPS (due to the GPU doing most of the hard work but still not managing to keep up and starting to cut into your otherwise excellent framerate).

What this essentially boils down to is: Your CPU determines the baseline performance you can get based on the environment you are in and the necessary work to be done, any additional work cuts into your framerate, your GPU on the other hand is basically "free" resources which do not cut into your performance unless you start lacking GPU resources. Having a place with high framerate can mean that you can still get a high framerate even though your GPU is bottlenecking (due to not bottlenecking too hard, just enough to start lowering CPU usage) by simply "overdoing" GPU usage most commonly achieved with a lot of shadows.

Note that i tried to simplify this as much as possible, i'm not a graphics programmer so it might not be 100% accurate, this is solely based on some basic fundamentals that i know (such as code running on CPU under normal circumstances and shaders running on GPU) and a lot of observation over the past 10+ years.

thanks for taking the time to break that all down...being that my CPU is a 10700k now I doubt it will bottleneck in SL and the GPU being older has more chance of bottlenecking

But I stumbled across a situation this am I hadn't thought of trying before. I was filming a basically empty sim save for some light landscaping (few mesh trees, bushes, like that) on a noonish setting and the whole time I was there the GPU was running at between about 93-100% and CPU about 10%, FPS was excellent (about 40-60fps). Not exactly sure what this means, but basically there were hardly any complex items or many textures to process - but seems to me the less texture/less complex a sim means more GPU usage, and better frame rate. I guess Im overthinking all of this - SL is what it is LOL

[Chic Aeon]

31 minutes ago, Jackson Redstar said:

thanks for taking the time to break that all down...being that my CPU is a 10700k now I doubt it will bottleneck in SL and the GPU being older has more chance of bottlenecking

But I stumbled across a situation this am I hadn't thought of trying before. I was filming a basically empty sim save for some light landscaping (few mesh trees, bushes, like that) on a noonish setting and the whole time I was there the GPU was running at between about 93-100% and CPU about 10%, FPS was excellent (about 40-60fps). Not exactly sure what this means, but basically there were hardly any complex items or many textures to process - but seems to me the less texture/less complex a sim means more GPU usage, and better frame rate. I guess Im overthinking all of this - SL is what it is LOL

I thought the shadows explanation quite interesting. I never turn shadows off :D.   But I do have a couple of thoughts on shadows.  There are settings in FS for shadow quality and such. Mine are at the default and seemingly cause no issues.   So you might check and make sure yours are.  There used to be a different shadows complexity pane but that seems to be gone now.

AND as an aside but possibly backing up what NiranV said --- I have been playing Skylines this last year with no issues (even when running FS alongside). But after a few cities I wanted to do something different which meant getting a mod that would let you make a much bigger city (so 23 tiles rather than 25).   As you might imagine my computer wasn't too happy with that and started crashing.  I looked at my options and tested some. Size of window made no difference. Having the simulation running made no difference. BUT there was a setting for shadows. I turned mind from HIGH down to MEDIUM (which gives you 2 o'clock shadows rather than five o'clock shadows) and I have been doing fine.    

 

So there certainly seems to be something with that theory overall.    I remember in Sansar Ebbe saying that the viewer there would take up ALL you would let it have (so running at 100 percent all the time).  Things worked fairly well for me there. Crashes were the beta thing more than "me" as far as I could tell. 

 

 

[arton Rotaru]

2 hours ago, Jackson Redstar said:

But I stumbled across a situation this am I hadn't thought of trying before. I was filming a basically empty sim save for some light landscaping (few mesh trees, bushes, like that) on a noonish setting and the whole time I was there the GPU was running at between about 93-100% and CPU about 10%, FPS was excellent (about 40-60fps). Not exactly sure what this means, but basically there were hardly any complex items or many textures to process - but seems to me the less texture/less complex a sim means more GPU usage, and better frame rate. I guess Im overthinking all of this - SL is what it is LOL

Few objects, few textures = few drawcalls/state changes.
many objects, many textures = many drawcalls/state changes.

Drawcalls are issued by the CPU. Once the drawcalls become so many that the CPU can't cope with them in time, the GPU starts to wait on the next batch to render. Hence your GPU CPU bottleneck is shifting.

Considering that a SL region can hold a gazillion of objects/materials, it's no wonder that even the fastes CPUs are reaching their limits. E.g. a plain prim cube comes along with 6 materials (faces), which will be broken of into 6 objects in the render pipeline.

[NiranV Dean]

3 hours ago, Jackson Redstar said:

thanks for taking the time to break that all down...being that my CPU is a 10700k now I doubt it will bottleneck in SL and the GPU being older has more chance of bottlenecking

But I stumbled across a situation this am I hadn't thought of trying before. I was filming a basically empty sim save for some light landscaping (few mesh trees, bushes, like that) on a noonish setting and the whole time I was there the GPU was running at between about 93-100% and CPU about 10%, FPS was excellent (about 40-60fps). Not exactly sure what this means, but basically there were hardly any complex items or many textures to process - but seems to me the less texture/less complex a sim means more GPU usage, and better frame rate. I guess Im overthinking all of this - SL is what it is LOL

Your CPU always bottlenecks unless your GPU bottlenecks, like mentioned above. No exceptions. It doesn't bottleneck because its not fast enough, it bottlenecks because the resources available to SL are limited (to roughly effectively 2 cores worth). Whatever is available inside this range of resources is always used as much as possible.

Just now, arton Rotaru said:

Few objects, few textures = few drawcalls/state changes.
many objects, many textures = many drawcalls/state changes.

Drawcalls are issued by the CPU. Once the drawcalls become so many that the CPU can't cope with them in time, the GPU starts to wait on the next batch to render. Hence your GPU CPU bottleneck is shifting.

Considering that a SL region can hold a gazillion of objects/materials, it's no wonder that even the fastes CPUs are reaching their limits. E.g. a plain prim cube comes along with 6 materials (faces), which will be broken of into 6 objects in the render pipeline.

I said it many times and i'll say it again. For what SL is doing its quite fast. It could be much faster and more efficient (theres still a lot of really wonky, broken and unoptimized stuff SL does) but still... my point still stands, for a massive clusterf... of code... sort of like a museum of code of the past 2 decades, its running pretty well compared to many other much more optimized engines and games i've seen in the last decade...

Edited February 9, 2021 by NiranV Dean

[Mollymews]

5 hours ago, NiranV Dean said:

I said it many times and i'll say it again. For what SL is doing its quite fast. It could be much faster and more efficient (theres still a lot of really wonky, broken and unoptimized stuff SL does) but still... my point still stands, for a massive clusterf... of code... sort of like a museum of code of the past 2 decades, its running pretty well compared to many other much more optimized engines and games i've seen in the last decade...

this ^^

 

[NiranV Dean]

Also interesting tidbit i learned today:

Object faces do not incur separate draw calls unless they are materials enabled and going through deferred rendering.

Object faces are batched unless the above is true.

Also further testing revealed that: Fullbrights are Forward rendered = batched even with materials UNTIL you give them a specular texture and set their "Environment" value to at least 1, in that case they become Deferred and will also fully receive post processing again (such as SSAO), this also means they won't be batched anymore because Materials currently are not batched (this could be added but is apparently a big rewrite according to Sovereign Engineer).

@Penny Patton I'm sure this would be extremely interesting for you to optimize your builds even more, knowing that you can remove "Environment" from Fullbrights that shouldn't be visible (to force them to be forward rendered and thus batched), the same also goes for material usage, any face that does not need materials should not have one in order to be batched. This in theory however would only be worthwhile if the object in question has at least 2 faces that can be batched, the more the better. Ultimately the best solution is obviously simply not having said extra faces at all.

[Profaitchikenz Haiku]

I'm interested enough to want to ask does "batched" mean something like "done at your own pace rather than done immediately to the detriment of all the other things waiting to be done" ?

[Jackson Redstar]

i switched my task manger graph view of the CPU Logical Processors so I could see per core performance a and while the Overall Utilization showed about 10% the single core SL was using was most often running 80-100%

this is on a mostly empty sim except for some landscaping at about 100fps (everything on)

[NiranV Dean]

On 2/10/2021 at 11:05 AM, Profaitchikenz Haiku said:

I'm interested enough to want to ask does "batched" mean something like "done at your own pace rather than done immediately to the detriment of all the other things waiting to be done" ?

Batched, comes from batching, clustering/grouping together things to work on them as batch. Generally done to improve efficiency. Say you have to change a single line in several documents, rather than starting your computer doing that change, shutting down and starting it again to repeat the process over and over, you start your PC once, open them all, change that one line in all of them in one go and then shut down the PC again.

On 2/10/2021 at 7:11 PM, Jackson Redstar said:

i switched my task manger graph view of the CPU Logical Processors so I could see per core performance a and while the Overall Utilization showed about 10% the single core SL was using was most often running 80-100%

this is on a mostly empty sim except for some landscaping at about 100fps (everything on)

Right, and the rest is being automatically distributed to other cores as windows sees fit.

Edited February 12, 2021 by NiranV Dean