Script Lag, why so spotty?

[Qie Niangao]

At the Server User Group this week, the folks from the London City regions reported that their sims were running much better in the past week or so, compared to a lot of lag in the past, and wondered whether there was some change to which the improvement could be attributed. (There wasn't.)

Coincidentally, my "home sim" is having a spot of the ol' script lag at the moment. It's tolerable, bouncing around 70% scripts run -- I only really notice it when changing alpha cuts on my mesh avatar -- but the region only has about 3000 scripts and low event rates, so it usually runs a solid 99-100% of scripts each frame with a few percent spare. I've seen this particular sim have this problem before and the next time it was restarted it immediately returned to the 100% level.

We know the Lab is investigating issues with design of the script scheduler, but that's about general, steady-state overhead of managing lots of mostly idle scripts. I don't know of any attention on the weird fluctuations in script performance of sims from one restart to the next.

I don't know how full region sims are virtualized on the servers, other than "one per core". Are they pinned to a specific core or do they just fight it out with processes of other sims sharing the same server? And how about (other) possibly shared resources -- cache, memory bus, network interface, etc. -- do we think Operations has a handle on which of these peg capacity during script lag events? is it known to developers?

Is there any kind of systematic data collection and reporting we residents could do to help figure out what's going on?

Historically, this sort of unexplained variability in performance has led SL users to sacrifice goats at the altar of "server class" and other superstitions that may have been amusing to Lindens, but I'm thinking they better get a handle on these capacity issues before they move much further to the cloud. It could be expensive to figure it out later.

[KT Kingsley]

I may be starting to get the entirely subjective idea that it's regions that have more comings and goings of agents that seem to be most affected by this.

[Qie Niangao]

1 hour ago, KT Kingsley said:

I may be starting to get the entirely subjective idea that it's regions that have more comings and goings of agents that seem to be most affected by this.

That's probably true in general, and surely matches the situation at London City, but this particular case is a sleepy Mainland sim where it's a big day when a half dozen folks pass through on the SLRR over the course of 24 hours.

It may be the exception that proves the rule -- well, some rule, anyway -- because it's only very rarely script-lagged and usually even has plenty of Spare Time. That's why, when it get in this modestly lagged state, it must be an example of something strange.

[KT Kingsley]

12 minutes ago, Qie Niangao said:

That's probably true in general, and surely matches the situation at London City, but this particular case is a sleepy Mainland sim where it's a big day when a half dozen folks pass through on the SLRR over the course of 24 hours.

It may be the exception that proves the rule -- well, some rule, anyway -- because it's only very rarely script-lagged and usually even has plenty of Spare Time. That's why, when it get in this modestly lagged state, it must be an example of something strange.

Yeah, it's only a very vague and unquantified idea that's started lurking in my head. Maybe the effect spills over from busier regions on the same server. More likely, though, it's something else altogether.

[Israel Schnute]

The main region I am on, where I have been having trouble with for well over a year now just got restarted.  Still at 5500 scripts (not sure how many of that is me versus the farm next door), and it is running the best in at least 3 years.  

I can finally drive my buses and turn.  The trains are running at the proper timings - and is tripping all of my sensors at the wrong time.  Which shows how much I've compensated for the lag before.

I thought it must have something to do with the script improvements on the beta that made it to the non-beta sims, where things are running fine as well.  But I'm not so sure.   I'm going to try to stick around as much as I can to see if it degrades quickly or not.  I haven't been able to have fun on half of my property in almost TWO stinkin years now! And this is the first.

Edited August 20, 2019 by Israel Schnute

[Qie Niangao]

Oh yeah: I should have followed-up here that the sim I was fussing about in this thread also miraculously recovered last week as I noted in the deployment thread, despite not getting the "script improvement" version. I just checked and after about six days it's still humming along with Spare Time to, uh, "spare."

This isn't as dramatic as "best in at least 3 years" because Tussock usually has Spare Time, only falling into the script lag curse very occasionally -- although two restarts in a row, recently.

[Israel Schnute]

Still running around nearly (or probably is) PERFECT.  Still at 5550 scripts or so.  18ms script time which finally just went to 16; spare time never happens until I see a jump to 19ms script time.

I should probably return all the sensors and timers back to where they should be so that I can see just how much the lag messes things up if it does return for a benchmark.  But, gosh, it sure is nice to be able to drive any vehicle, and make a simple turn again without having to back up 8 or 9 times!

[Nalates Urriah]

Others are pointing to building in an area forcing the Pathfinding physics/terrain/map updates which leak memory. Consuming memory would affect scripts as some point.

If that PF thing is a real problem the disparate problems may make it difficult to put a finger on the script % problem. 

[Qie Niangao]

The Pathfinding thing may be real, but I'm pretty sure it's distinct from script performance varying widely from one restart to another, more or less immediately upon restart, seemingly independent of what version of software is loaded. I'm pretty convinced that the amount of script processing represented by a millisecond of "Script Time" varies erratically depending on what's happening on other sims running on the same host.