Zero spare time, 68% scripts Run, Frequent texture blurring

[Vulpinus]

OK, now I have your attention...

I've been watching the statistics panel because I have a very annoying issue going on on my Mainland parcel. I used to live a couple of sims away, and had none of this until I moved here a month or so back.

Textures can take an age to load, even thought they are most certainly already cached. If I TP away to a shop, things load there perfectly, then I TP back home and wait another age for everything to display properly again. Worse, textures randomly blur and (if I'm lucky) reload, frequently. I'm not even moving, just sitting there watching the walls or windows blur out. If I turn around, half the things behind me are blurred and I need to go right up to them to get them back. Normal maps do this too, making my walls look like cobblestone at times (they aren't).

I have read and followed everything in the troubleshooting guide for this stuff - nothing made any difference whatsoever. It happens in both Firestorm and the official viewer (latest of both as of a couple of days ago).

To repeat - this does not seem to happen anywhere else, only my home, and I've noticed the surrounding parcels on the same sim taking a while to display for me too.

The only bad-looking things I can see on the stats panel (these are rough averages from watching for an hour) are:

• Time Dilation lowest spotted was 0.76, but mostly around 0.98 to 0.99
• Agent Updates per second 13.
• Scripts Run, varying from 68% (lowest I noticed) upwards, around 80% average.
• Spare Time 0.0 lowest, averaging around 0.2ms, occasionally over a whole ms, often 0.1ms or less.

Everything else looks OK, but I'm no expert. Display FPS 40+, Sim and Physics FPS at around 45, zero packet loss. My WAN and LAN connections are generally very good, wired, 20+Mbps, dedicated public IP address on the PC I use for SL (so no NAT) and my PC is pretty fast, clean installed recently, runs off an SSD with a separate SSD just for my SL cache!

A couple of weeks ago, I TP'd home after some shopping, and my textures were going so mad that I was about to log off in disgust, then I saw the "Sim is restarting" message so I came back after a while. For several days, almost no symptoms at all, but then they slowly started up again. Subsequent sim restarts (like today's) seem to have made little if any difference.

I would appreciate any advice please.

 

ETA: Another thing I've noticed is that shadows and 'daylight' lighting often seem erratic, as if the viewer doesn't know whether something should be in shade or not. I run at maximum graphics settings, everything turned on except DOF.

EATA: If anyone who knows more about what to look for than me (which isn't hard) wants to come and take a look, You're most welcome to. I'll IM you the location though; don't want an unintentional party - the sim's bad enough as it is.

[ChinRey]

I'm afraid you've been attacked by a combination of gpu overload and the texture bias bug.

 

GPU overload - or client side lag does not show up on the performance stats since it's not a problem for the servers, only for our computers. What happens is simply that the graphics processor unit gets so busy it can't manage all the work it's asked to do. Everything you see in SL will add to the gpu's load but some of the most common causes are:

• Too many and/or too high resolution textures
• Too long a draw distance
• RenderVolLODFactor set too high
• Graphics settings in general set higher than your computer is able to handle

 

Texture blurring is caused by a bug introduced with Project Interesting and never fixed. To put it simply, the gpu cahce (which is not the same as the regular texture cache) gets filled up with textures and the viewer is unable to determine which to keep and which to ditch. The result is that your computer removes textures from the gpu cache to make room for new ones only to discover it still needed them so it has to load them from the texture cache on the harddisk over and over and over again.

Unfortunately there is no cure for this - except to relog every now and then. That is, you can enable full res textures (Develop menu -> Rendering -> Full Res Textures) but that will put a tremendous amount of extra load on your gpu and if it's already as overloaded as it seems, you're likely to just crash after a few minutes.

[Vulpinus]

Hmm... thank you for that answer. It unfortunately tallies with my experience and what I had been thinking from googling the issue for a while.

GPU cache thrashing sounds just like it. A relog does indeed reduce the issue for a short while.

If it's overloading the PC too, I'm not impressed, although it might be. Specs are Gigabyte GA-Z68X-UD5-B3 mainboard, MSI GTX970 GPU, Intel i7 at 3.7GHz, 16GB RAM, Crucial 512GB MX100 SSD for system and Samsung 850EVO 120GB for the cache.

Turning down the graphics settings (at least as far as I'm willing to use) does not seem to make much if any difference, nor does individually dropping my RenderVolumeLOD or draw distance (normally at 120m but I dropped down to 64m).

The textures around my might be an issue. I've built my own house (mansion) and landscaped; I have 12,000sq.m. I have tried to minimise textures wherever I can, dropping them down from 1024x1024 to lower and reuploading, reusing textures where I don't desperately need a new one, etc. Even been as far as finding all the (hidden) default texture use and replacing it with blank.

I do however use a lot of normal and specular maps, so tripling the requirements.

Doing that did seem to very slighly mitigate the issue for a while, but as I furnished the house, the problems got worse again, I think.

I did try the full res texture setting when I came across that, having read the "never use this" comment too. My PC ran, but was obviously straining to pull everything in around me and the GPU fans, which mostly run at idle, powered up like a Harrier jet taking off.

Looks like a relog every 15 minutes when I'm at home is the only way. This isn't good.

Still, I rarely see any issues elsewhere. My home is by far the worst effected with this, and I've tried to provoke the issues at places like Frank's Jazz and The Men's Department, places that normally cause me a little lag. I spent an hour loitering around Frank's Galleria, and never once had a problem.

[gunmaker]

iiits not a bug. itsa bad configuration, if you follow my changes undera post titled

 

HEY I THINK I FOUND OUT WHY SL LAGS SO MUCH

 

youll find your solution

 

another solution is to download singularity viewer and run a macro through the debug menu> debug settings,, and set all option to reset|default.

simpe as a recording a macro with autohotkey with {down} and {click} with click set to the area of the reset button, takes about 2 minutes anda reboot

p.s. after doing that even in intense areas i was getting 256 frames per second.. perhas its time to declutter huh secondlife?

[Qie Niangao]

I think you actually have two different things going on here, and maybe they're interacting in some unfortunate way. Drop me a SLURL or something and I'll investigate.

There's definitely some widespread problem with how cached textures are managed when not in view, and I don't really think developers yet know how to fix that completely. But you're also experiencing something else much less than optimal, sim-side, with zero spare time and a substantial share of scripts overrunning the current simulation frame. That's why I'd like to snoop around and see if you might have something specific to your location that's pushing updates, which would slow the sim as well and crowd network downloads. Maybe.

[ETA: I meant to suggest trying out the Black Dragon viewer to explore some graphics cache settings that may help with the texture-thrasing. See Inara's recent blog post for an overview.]

[Vulpinus]

Thanks for your offer - I've IM'd (and PM'd) you the SLURL. I'll take a look at that blog too.

[Vulpinus]

First thoughts on Black Dragon: Texture thrashing certainly seems to have stopped. Once, that is , I ramped up the graphics memory settings to 1024MB each.

That seems to back up the GPU memory issues then. I've not seen any texture thrashing for the 30 minutes since I upped the memory and relogged. On Firestorm, things would be flashing at me by now.

I just wish I could use the viewer! There seems no support for my Spacemouse, or even the Mouse-Walk that Firestorm had, so I'm stuck purely with the keyboard. I hate that, lol! Using the spacemouse to look around has transormed my SL experience so much that I can't go back to not having it.

[Ayesha Askham]

Qie has it right.  The main culprit here is how the server handles the project interesting code actions.  It is becoming clear that many (though not all) Mainland servers, which are the oldest in SL, simply cannot handle to workload they are being asked to do.

LL are uncomfortably aware of this and are replacing old servers as fast as finance permits but it is a problem for many on Mainland.  A simple sim restart might solve the issue but it is quite probably that it will be just as bad.

The "Spare Time" reading is quite diagnostic here.  If the server has little of no spare time it is not processing information efficiently and texture-related issues WILL ensue.  Lots of  hi-rez textures make the matter worse.

[Vulpinus]

Hmm. Thanks for your input. That's bad news for me, unless they get to my server soon. I've recently paid L$35,000 for my land here at auction, plus getting caught out by paying extra tier because I had to buy as an individual and then deed to my group. It wasn't an expense I took on lightly; I don't have that much money to waste.

The texture thrashing really is spoiling the whole thing for me.

The explanation does seem to fit exactly with what I am experiencing. I was actually hoping to find a problem at my end (like last time I had an issue - a failing NIC) but I feared this time it wasn't.

[Theresa Tennyson]

---

Ayesha Askham wrote:

Qie has it right.  The main culprit here is how the server handles the project interesting code actions.  It is becoming clear that many (though not all) Mainland servers, which are the oldest in SL, simply cannot handle to workload they are being asked to do.

LL are uncomfortably aware of this and are replacing old servers as fast as finance permits but it is a problem for many on Mainland.  A simple sim restart might solve the issue but it is quite probably that it will be just as bad.

The "Spare Time" reading is quite diagnostic here.  If the server has little of no spare time it is not processing information efficiently and texture-related issues WILL ensue.  Lots of  hi-rez textures make the matter worse.

---

Theresa Tennyson sighs...

Okay, I'd love to hear what you're basing this statement on, because it's the complete opposite of my understanding, which is...

Since switching textures and meshes to the CDN very little texture information goes through the servers themselves - they arrive to your viewer via an end-run through up to 8 separate HTTP connection that bypass the servers.

The lack of "spare time" is directly related to the less than 100% figure for "scripts run." There's a sort of walled time-garden in the server routines that allocates a certain amount of time for scripts to run. If all the scripts can be run in less than the allotted time then the remaining time from this block is considered "spare time." If all the scripts can't then only a certain number of them are run in any given "frame" (1/45 second), with the remaining scripts deferred to the next frame. If less than 100% of scripts are running per frame in this region then there can't be any "spare time."

[Qie Niangao]

Yeah, I'm not entirely sure what's going on, despite having made an on-site visit. I do think we found a significant contributor to the script lag effect, but it remains to be seen whether the neighbour will turn off their sakura tree's temp-rezzing of scripted physical petals and blossom clusters long enough to determine how much of the problem they're creating.

It's certainly my understanding that the CDN should be serving-up the texture bitmaps, along with mesh geometry, etc., and that this stuff is no longer delivered by the sim itself. The sim nonetheless must identify which textures and meshes and what-not need to be fetched. I honestly don't know whether the widespread texture-thrashing problems we've all seen are due to a strictly viewer problem of not optimally managing object and texture cache, or if there are sim changes (perhaps Project Interesting related? dunno) that somehow trigger the viewer to re-fetch stuff it should already have. This might be understood by somebody, but if so, one might think the problems would be fixed by now.

The thing is, the effect is much worse some times than others, and at some locations compared to others. That it's (usually) very bad at this particular sim, and that this particular sim is also seeing some modest script lag, may be purely coincidental. On the other hand, if the script load is also flooding the network with object updates to the viewer, I'd guessed that might interact with the texture fetching traffic.

Oh also, yeah, I'm not sure Mainland sims run any slower than Estate ones. I'm not sure there's any particular pattern in which sims run on which model servers, nor am I confident that newer servers deliver better performance, per sim, than their elders.

[Ayesha Askham]

OK Theresa and Qie

My information is essentially gleaned from a "reliable source" within The Lab, other than that I'll let your curiosity run wild.

Now...despite all the fanfare and hoohaa about the CDN reducing server load, the sim-server still has to communicate with the client, whatever that might be.

There is no doubt that increasing client texture memory will help, especially with the "texture thrashing", but that itself is a function of incorrect communication between sim-server and client.

If a sim has zero or very little "free time", script run will indeed be hampered, but so also will all the other functions that a sim-server is required to perform every frame.

Now whether you like it or not Mainland servers are older and less capable than most newer private region servers, both for full-region and Homestead sims.

These problems are, at least in my experience, distinctly more frequent on Mainland than anywhere else and that is circumstantial, but widely observed.  It is not that the Mainland servers run badly, if properly loaded, it is simply that when they are stressed they run badly.

And Qie, the newer servers DO deliver superior performance, it is just that more regions are crammed on per core than ever so the benefit is largely dissipated.

[Vulpinus]

More food for thought.

Qie, thank you for your visit today. I can't believe that for literally the first time in weeks, the texture thrashing wasn't happening much. Oh well, that's (Second) life. Like I mentioned, I actually noticed that it had stopped about ten minutes before you arrived, but I wasn't paying full attention because I was multitasking at the time. Thinking about it, it was just about the time that I filed a support case for the issue with LL. Quick work LL, thanks  :smileylol:

My neighbour arrived just after you left, and agreed to reduce the quantity of blossom coming from the tree but not turn it off. She said it's her favourite tree and she uses it everywhere.. Don't know if that will make much differece but we'll see.

I'll be inworld again later tonight probably and it will be intersting to see if the issue returns with a vengance.

[Ohjiro Watanabe]

You mention that Spare Time is around 0 but don't mention (or I may have missed it) what the other stats in that section are. Something is hogging that precious 22ms the sim has to create its frames which would be a clue to your server side lag.

[Qie Niangao]

I should have mentioned: while I was there, it was pure script lag. That is, no frame dilation at all, merely scripts delaying execution of other scripts. And not disastrously, at that, but clearly there was a very busy script engine, so if those scripts were pushing updates to viewers, it could interfere with some viewer performance, maybe (but not while I was there, and apparently not since, so... must be my magic fairy dust, with an emphasis on "fairy". :smileytongue:)

[Vulpinus]

Yeah, I should have said it was script time, sorry.

I'm looking at it right now and spare time is 0.000ms mostly with the occasional jump to 0.0xx and very occasionally higher.

Script time is 18.6+ms, with total frame time of 22.to 23ms.

---

The texture thrashing issue has not returned since Qie's visit. It's completely unnerving the way it just suddenly stopped (as I mentioned, I noticed stop at the time) after being so bad for a month or more.

I'm sort of afraid to do anything, in case I provoke it to start again. I certainly didn't change anything to make it stop.

There is still the occasional texture glitch as I turn around and stuff like that, but I expect that''s pretty normal and it certainly isn't enough for me to worry about.

---

If nothing else, at least this (and my crow scripting issue) has made me have a very hard look at what's around me. There's nothing really that is unusual on my parcel, but I have realised just how hard a load of alpha blending grasses and trees can hit my GPU. I've been converting some to masking (still high GPU use but higher FPS) and have bought some that use masking to begin with. Every little helps.

Hopefully whatever was going on has resolved itself (or maybe even LL has fixed something). I'll certainly report back it it starts up again.

Thanks for all the help folks, especially Qie.

[Ohjiro Watanabe]

That is a very high script time for a residential parcel. That is the sort of level I would expect in one of the London sims or somewhere like that where there are loads of scripted things going on all around you.

It may not be the only thing that is causing lag but it is definitely worth looking into. You can use the About Land/Script Info to help track down scripted objects.

On the parcel I am on now there are 230 scripted objects and the script time is about 7ms spare time is 13ms.

[Vulpinus]

I've looked at that before. I have about 300 scripts listed, but the majority are door and window scripts, lights with a toggle on/off, chairs with basic sitting animations in and the like. Some are bigger but I really don't think there is much that's very active.

Most of them should be idle and I think only contribute a few microseconds of script time in that state, if I've read things correctly.

Is there any way for me to check my own script time, as opposed to the whole sims? Or another parcel's scripts? (I don't get anything in the about-land/script-info for other parcels than my own).

[Ohjiro Watanabe]

You can get object scanners but I don't think that they will tell you which script is causing the problem.

If I was you I would start up a chat session with support, tell them about the free time and script time and that you have looked but cant find the script that is causing the problem and ask them if they can look into it from their side.

You pay top dollar for the land and are entitled to support.

[Vulpinus]

I'll do that next week then, I've already started a support ticket but not had a response yet so I'll follow it up with a chat sesion.

[Nalates Urriah]

There are a coupe of different fixes in the works for texture thrashing (render, blurry, render, blurry, etc)

The various viewer developers have found that your chat icons are the problem. As people chat their profile picture is loaded and shrunk to a tiny icon for chat. The problem is the video memory allocated is for the full size image, apparently marked 'keep', and it is never released. Once enough of these consume video memory the video card starts swaping releasable textures in and out of video memory,

Third party developers are working on a fix that will come into the SL Viewr via open source contributions. Don't hold your bearth, but a fix is in the works.

[Ohjiro Watanabe]

That's interesting Natates. Do the images still get loaded if you have the beastly critters (chat icons) turned off? As you can in the better viewers.

 

 

 

[Vulpinus]

Hmm, that is interesting. It makes some sense as I noticed loads of people's photos in my cache when I was trying to figure out what was wrong, and guessed they must be coming from chat sessions from groups. I'm in a lot. It fits with why the problem gradually gets worse and is temporarily fixed by a relog, and why it varies depending on time of day (more/fewer people active). Still doesn't answer why only my sim though, but perhaps in combination with the Script Time issue...

---

With regards my issues, I've seen the occasional instance of thrashing since Qie's visit, but not much. Must have been that fairy dust...

I had a chat with support yesterday, and didn't really get anywhere other than they restarted the sim (again). Oddly though, just prior to that for an hour I did notice that the Spare Time was creeping up a little. Not much, but hitting 2ms to 4ms which is higher than I have seen it for weeks. Still bottomed out enough for me to start the chat though.

Now, it's varying mostly between 3ms to 7ms Spare Time, with Script Time 12 to 16 or thereabouts.

Really, it's like someone from LL has been watchnig my posts and fixing thiings on the sly. Keep it up!

I hope it stays like this...

[ChinRey]

---

Nalates Urriah wrote:

The various viewer developers have found that your chat icons are the problem. As people chat their profile picture is loaded and shrunk to a tiny icon for chat. The problem is the video memory allocated is for the full size image, apparently marked 'keep', and it is never released. Once enough of these consume video memory the video card starts swaping releasable textures in and out of video memory

---

I don't know what to say. It's good to know that somebody's managed to figure it out.

But on the other hand, texture thrashing was reported to LL 27th May 2014 (due to modesty I can't tell who filed the JIRA) and given priority as a major bug. After almost a year they still haven't figured out something as obvious as that on their own!?!

[SaraCarena]

---

Ohjiro Watanabe wrote:

.... Do the images still get loaded if you have the beastly critters (chat icons) turned off? .....

 

 

 

---

Yes, according to a comment left on Nals blog by one of the Alchemy viewer team, they do. Group chat and friend list icons do. As far as I know it was made public by the Alchemy team at a TPV meeting early 2015.

https://alchemy.atlassian.net/browse/ALCH-184?filter=10300