Is LL putting more sims on fewer servers?

[Aishagain]

This is what I had assumed. It does not, however, explain what changed to drop our script-run from >99% to under 50% as it is now.  Something MUST have changed to effect that...but what?

[Sharie Criss]

I've been running a full region venue for over 10 years now. The region has certainly gone through changes over the years, but I've been monitoring script / prim usage closely, using estate tools to find poorly behaving objects and replace them as needed.  From the peak, script count is down 30% yet performance is down even further, at best running at 50%. I occasionally run Mono benchmarks (see the LL wiki for them) and what is PAINFULLY clear to me is that my region is running at half the speed it used to. I've filed quite a few tickets regarding performance issues, but Larry Mo and Curly show up and claim there is no issue. Frankly, I've completely given up any hope whatsoever that we will ever get good region performance ever again. Wandering around SL vising some of the same places I've gone to for years, everyone is suffering with these same performance issues.

What everyone knows (but a few here have trouble grasping) is that computers now are far far more capable than systems were 12+ years ago, and all that extra performance is available at a fraction of the price.  Data center costs have also dropped due to massively increased competition and other economic factors, and bandwidth is cheaper too. Taking the same money we were paying for regions 12 years ago and applying it to today's systems, we should have zero server side lag. Zero. Also, keep in mind that LL has been doing a lot of work removing work from sims by moving things like asset downloads / inventory / etc. to the cloud (content servers) and away from the sim itself. Why aren't we actually seeing a benefit to the region performance from these efforts?

No, I don't have any proof, LL won't talk about it, and everyone is just guessing, but based on all the data it's clear something has changed that has sucked the life out of region performance. I strongly suspect that LL is saving money by cutting the infrastructure available for regions during the last technology refresh.

What's incredibly disappointing is that we finally have some really neat technology such as Animesh, pathfinding, Experience Keys etc. that could allow creators to build fantastic new venues, games, etc. BUT, now region performance is so freaking horrible that we can do none of that, and in fact, some venues that did take some advantage of these new features are unplayable. Vehicle usage is a joke with lag spikes that send you into a lag freeze shooting you 10 sims away before snapping back every time someone TPs in or out. Region crossing is a crapshoot.

Clearly LL has lost any pride they had in SL. People show up to collect a paycheck. They go through the motions, throw us a bone now and then with some new feature that's pretty much useless due to server performance issues. Bleh.

[Qie Niangao]

38 minutes ago, Sharie Criss said:

Also, keep in mind that LL has been doing a lot of work removing work from sims by moving things like asset downloads / inventory / etc. to the cloud (content servers) and away from the sim itself. Why aren't we actually seeing a benefit to the region performance from these efforts?

Interesting

[Sharie Criss]

After the restarts, my region is now performing at 1/3rd the speed it used to. Restarted a second time, no better. Seriously LL, you owe us some answers. I will NOT pay for another month's service if the performance stays this bad. I'm done. Whatever you are doing, you are doing it wrong. TP's still broken too, which sucks the joy out of even logging in. It's painfully clear based on recent events that the executive team no longer cares about SL and it's paying user base. The Second Life product is in a terrible state - the worst it's been in many years. Not only will I no longer pay for a shoddy product, I'm due a refund. I've been paying for service but am not receiving it.

 -

[animats]

I'm concerned about declining performance, too. Does anyone run a benchmark in LSL occasionally?

One big problem is that scripts doing nothing use script time. Not much, about 0.003 ms/frame. But if you have 4000 scripts in a sim, there goes most of your script time.

Edited April 24, 2019 by animats

[Sayrah Parx]

I agree about all the scripts doing nothing.  I believe there would be an overwhelming performance gain if they made it possible to edit the hover text properties in the object editor.  People can only set the text, color and alpha with a script.  They don't know it's a property of the object and doesn't depend on the script at all for existence.  There are people who have been scripting in SL for 10 years who don't know that it doesn't depend on a script, other than it being updated by a script.  It's possible that the majority of scripts currently loaded across all of SL were only there for the initial llSetText in state_entry, and never got run again once.  All those useless scripts being loaded and using script time really add up.

[MBeatrix]

49 minutes ago, Sayrah Parx said:

I agree about all the scripts doing nothing.  I believe there would be an overwhelming performance gain if they made it possible to edit the hover text properties in the object editor.  People can only set the text, color and alpha with a script.  They don't know it's a property of the object and doesn't depend on the script at all for existence.  There are people who have been scripting in SL for 10 years who don't know that it doesn't depend on a script, other than it being updated by a script.  It's possible that the majority of scripts currently loaded across all of SL were only there for the initial llSetText in state_entry, and never got run again once.  All those useless scripts being loaded and using script time really add up.

The same applies to textures scrolling and objects rotation, unless the script sets them on/off.

Edited April 24, 2019 by MBeatrix

[MBeatrix]

@Sayrah Parx

The same applies to textures scrolling and objects rotation, unless the script works as a switch to set them on/off.

[EDIT] Sorry about the double posting. 😞

Edited April 24, 2019 by MBeatrix
correction

[Qie Niangao]

1 hour ago, Sayrah Parx said:

I agree about all the scripts doing nothing.  I believe there would be an overwhelming performance gain if they made it possible to edit the hover text properties in the object editor.  [...]

Permanent particle systems (e.g., continuous waterfall spray) is another example of script-set prim properties.

I once went on a campaign to remove all those scripts from everything I could, just as a matter of computing hygiene. It's a big help to turn on the beacon for scripts.

The thing is, though, these things have been around for ages. In fact, hovertext is much less common than it was when I was a newbie and everybody labelled everything (including poseballs, which were everywhere). Conceivably it could still help to reduce the number of these scripts now, even though they weren't a problem before, because we have so many more other scripts now. I wish there were a way to quantify trends in what's contributing script count (and script time, too).

One thing that's been nagging at me is the fact that AVsitter is so ubiquitous and uses two scripts per sitter, plus a bunch of modular accessory scripts. I know the engine is actually way more efficient than XPOSE (probably about the same as nPose, maybe a bit better than MLP, no idea about others), but still it adds a lot of scripts to a region. It's open source, so I've mashed some of those scripts together occasionally, but that makes a whole project of version updates, and it's nothing we can expect end-users to do, or even most furniture creators. I wonder if it's too late to devise a more or less plug-and-play replacement that maybe supports fewer poses and somehow shrinks the script-per-function modularity.

Also it's hard to get motivated to cut furniture scripts in half when one HUD-equipped mesh-headed mesh avatar can more than offset the whole exercise. Are those the scripts causing the script lag? I remember when shoes and some hair had scripts in each prim, so are script counts really higher than they used to be? Or have little features and throttles and data collection been added to the server code, making script processing less efficient in imperceptibly gradual increments?

[ChinRey]

3 hours ago, Qie Niangao said:

Also it's hard to get motivated to cut furniture scripts in half when one HUD-equipped mesh-headed mesh avatar can more than offset the whole exercise. Are those the scripts causing the script lag?

No, it's not. I've seen the simulator performance drop drastically even in a region where I was the only avatar and the only scripted item I was wearing, was a standard AO.

It doesn't seem to be scripts only either. I've occasionally managed to catch the frame time stats just as the region froze and it isn't always script time that is peaking, it can be anything really. Nor am I convinced it's about server capacity as such. 100 ms agent time for a region with only one fairly low lag and not very active avatar - that seems more like faulty software than sheer lack of capacity.

The only thing that is certain, is that the servers are struggling hard these days to handle workloads that didn't cause them any problems whatsoever a year or three ago.

[Qie Niangao]

2 hours ago, ChinRey said:

It doesn't seem to be scripts only either.

Other stuff is likely happening too, but one thing that some of us have been focused on is the inordinate amount of script time being consumed. Script-on-script lag, specifically, is getting serious with sims running a smaller percentage of scripts per frame, with seemingly as much frame time devoted to scripts as always. It's not clear why, and it's a thing Linden developers claim is next-in-line for study -- presumably delayed by the current teleport disconnect fiasco.

As you say, sims aren't keeping up with the same loads they easily handled before. Maybe I'm gullible, but I really doubt this is because they're stacking them more sims per core. Something is using up time. My current totally unfounded theory is that it's due to stuff added to the sim ("little features and throttles and data collection") that has more than consumed all the savings they realized by off-loading services to non-sim hosts.

[Torric Rodas]

This issue has returned after the emergency roll of April 18th, so I have made a Jira entry, feel free to "me too" it   https://jira.secondlife.com/browse/BUG-226851

[Billy Daxter]

I'm not seeing this on my regions to be honest.  Nearly full of objects with 15ms of free time.  

The estate control panel offers the top scripts window that you could use to debug and find out what's consuming all that time.  

[Sharie Criss]

40 minutes ago, Billy Daxter said:

The estate control panel offers the top scripts window that you could use to debug and find out what's consuming all that time.  

FYI, the estate tools are USELESS at finding out what is burning time. Why? Because they show you an instantaneous point of time view, not how much resources (RAM / CPU) a script REALLY uses over time. Just hit refresh a few times and you can see what I mean. I've been asking for better tools but it's like a zero level priority for LL, which is sad, because without good performance analysis tools it's impossible to know which scripts are really causing an issue.

[Sharie Criss]

2 hours ago, Qie Niangao said:

As you say, sims aren't keeping up with the same loads they easily handled before. Maybe I'm gullible, but I really doubt this is because they're stacking them more sims per core. Something is using up time. My current totally unfounded theory is that it's due to stuff added to the sim ("little features and throttles and data collection") that has more than consumed all the savings they realized by off-loading services to non-sim hosts.

But this doesn't make sense. As I mentioned in a previous message, servers today are MUCH faster than the old systems used to be. SL is NOT running off 12 year old hardware. LL has moved subsystems off the servers (pretty much any of the old UDP delivery of data such as inventory, assets, textures is gone and it's now cloud delivered, or comes from other servers) so theoretically, with faster hardware and less to do the regions should perform better, not twice as bad as they used to. When you move stuff off, use faster hardware, and end up running slower, you are doing something very VERY wrong. The fact that LL REFUSES to acknowledge server performance issues AT ALL is Highly suspect. If they aren't sticking more regions on a server, then they screwed up royally. I can't believe LL is THAT incompetent.

[Qie Niangao]

3 hours ago, Sharie Criss said:

The fact that LL REFUSES to acknowledge server performance issues AT ALL is Highly suspect.

That's not the message I've gotten at Server user groups. Maybe I misinterpreted the Lindens, but I thought they were pretty committed to working on script performance, high on the to-do list. (I think that was before the teleport disconnects were recognized as needing everybody's full attention.)

I understood the focus to be on script performance, I guess because it's the most evident and widespread performance problem affecting the sims. I have seen some sims in non-script trouble, but I don't know if there really are more of those in more trouble than in the past.

Has anybody asked Ebbe or another Linden at a Town Hall whether they're still running full sims one-per-core? Or if they have some other theory of diminished performance?

[Qie Niangao]

4 hours ago, Billy Daxter said:

I'm not seeing this on my regions to be honest.  Nearly full of objects with 15ms of free time.  

The estate control panel offers the top scripts window that you could use to debug and find out what's consuming all that time.  

There are certainly Mainland sims with plenty of spare time, too, but they seemed relatively low script-count and low script-event-count (probably low object count, though) compared to some other sims where script performance has fallen off a cliff. 

Come to think of it, "falling off a cliff" does seem pretty non-linear. I wonder... could it be memory after all? Or it could be only some LSL functions that cause the script-lag to arise in some regions and not others. (For example, Pathfinding was a hypothesis at one point but the problem often arises with no Pathfinding on the sim.)

[Billy Daxter]

4 hours ago, Sharie Criss said:

But this doesn't make sense. As I mentioned in a previous message, servers today are MUCH faster than the old systems used to be. SL is NOT running off 12 year old hardware. LL has moved subsystems off the servers (pretty much any of the old UDP delivery of data such as inventory, assets, textures is gone and it's now cloud delivered, or comes from other servers) so theoretically, with faster hardware and less to do the regions should perform better, not twice as bad as they used to. When you move stuff off, use faster hardware, and end up running slower, you are doing something very VERY wrong. The fact that LL REFUSES to acknowledge server performance issues AT ALL is Highly suspect. If they aren't sticking more regions on a server, then they screwed up royally. I can't believe LL is THAT incompetent.

I use them all the time to keep my script time in check.  They work ok and you can sort by CPU time and by memory use.  You can also look at individual objects and use firestorm to show script use.  You just have to invest the time and dig a bit. The information is there if you know what you are looking for and how to interpret it. If you don't understand what the numbers mean there's a lot of resources out there that can help.  There's room for improvement of course but it's pretty useful to me as is.  

[Billy Daxter]

59 minutes ago, Qie Niangao said:

There are certainly Mainland sims with plenty of spare time, too, but they seemed relatively low script-count and low script-event-count (probably low object count, though) compared to some other sims where script performance has fallen off a cliff. 

Come to think of it, "falling off a cliff" does seem pretty non-linear. I wonder... could it be memory after all? Or it could be only some LSL functions that cause the script-lag to arise in some regions and not others. (For example, Pathfinding was a hypothesis at one point but the problem often arises with no Pathfinding on the sim.)

Here's the fun part.  It "could" be any number of things.  Most sims aren't carbon copies of one another and use different objects all scripted by different people with different abilities in LSL coding.  Comparing any one sim to another really is an apples to oranges comparison considering that they are completely different.  The only way you could really compare reasonably without just it being a wild guess would be to have two completely identical sims.  If they aren't identical there's no telling what's actually causing issues without looking and quantifying it.  

Think of it this way.  You want to benchmark some desktop computers.  You have the same OS on both machines but one machine is loaded with spyware and viruses and the other one is a blank install of the OS.  Which do you think is going to perform better? While spyware and viruses is a pretty extreme case it's not really too far from the truth with some of the poorly written LSL I have helped friends remove from their sims.  

Remember.  It's a user created world.  Some of us are idiots.  I certainly am at times slinging out thousands of poorly scripted items that's tanked my performance.  I learned a lot from being an idiot though and have nice smooth running lag free sim's now just by taking the time to understand "what" I am rezzing and how to diagnose issues when I create them for myself.  

 

[rasterscan]

On 4/24/2019 at 12:19 AM, Sharie Criss said:

After the restarts, my region is now performing at 1/3rd the speed it used to. Restarted a second time, no better. Seriously LL, you owe us some answers. I will NOT pay for another month's service if the performance stays this bad. I'm done. Whatever you are doing, you are doing it wrong. TP's still broken too, which sucks the joy out of even logging in. It's painfully clear based on recent events that the executive team no longer cares about SL and it's paying user base. The Second Life product is in a terrible state - the worst it's been in many years. Not only will I no longer pay for a shoddy product, I'm due a refund. I've been paying for service but am not receiving it.

 -

Ouch. I was on a frustration induced hate fest of my own recently, and then things seemed to suddenly improve dramatically. Breathing a sigh of relief, I felt bad for my outbursts and could enjoy the game as it should be once more. Until last night. I logged in, and found myself staring at my desktop slightly puzzled, it took several moments to realise firestorm had crashed me straight back to the desktop before I had even finished logging in. Bwahaha!

My current peeve is wtf has happened to my mesh body alpha hud settings.  I set the alpha slices to suit whatever I am wearing, TP off to DJ or whatever, and bosh, post tp the alpha settings are lost, chunks of skin showing through my ill fitting expensive jacket. There is a secondary alpha hud issue for me, if I click a saved alpha preset, it can take upwards of 60 seconds or more for the preset to take effect, I sit here watching the alpha slices change maybe one every 5 seconds or so, its really weird. Still, mustn't grumble, it doesn't happen all the time.

Lastly, I fear for the future of the game. The current .... ongoing issues with sim crossings, tp fails, lag and crashouts is frankly putting off a lot of people like yourself. I don't want to see the game fold, I'm as addicted as most players, but I do fear people are being seriously put off playing and drifting away.

Let the flame fest begin, but I had to say my piece apoligies.

[Theresa Tennyson]

On 4/20/2019 at 8:24 PM, Sharie Criss said:

I've been running a full region venue for over 10 years now. The region has certainly gone through changes over the years, but I've been monitoring script / prim usage closely, using estate tools to find poorly behaving objects and replace them as needed.  From the peak, script count is down 30% yet performance is down even further, at best running at 50%. I occasionally run Mono benchmarks (see the LL wiki for them) and what is PAINFULLY clear to me is that my region is running at half the speed it used to. I've filed quite a few tickets regarding performance issues, but Larry Mo and Curly show up and claim there is no issue. Frankly, I've completely given up any hope whatsoever that we will ever get good region performance ever again. Wandering around SL vising some of the same places I've gone to for years, everyone is suffering with these same performance issues.

 

And it's current lean-and-mean script count is over 9500 active scripts.

[Qie Niangao]

7 hours ago, Billy Daxter said:

Here's the fun part.  It "could" be any number of things.  Most sims aren't carbon copies of one another and use different objects all scripted by different people with different abilities in LSL coding.  Comparing any one sim to another really is an apples to oranges comparison considering that they are completely different.  The only way you could really compare reasonably without just it being a wild guess would be to have two completely identical sims.  If they aren't identical there's no telling what's actually causing issues without looking and quantifying it.  

For my sins, in a former life I studied social science, and I may be more comfortable with trying to extract meaning from crazily messy data. So I'm okay with informally correlating possible contributing factors across a bunch of wildly diverse sims, at least as a way of generating theories to test experimentally, to collect more specific data or, if it's quicker, to find sections of simulator code to investigate for flaws.

Studying the brain and debugging software have stuff in common. Maybe not the probe insertion part, but stuff in common nonetheless.

Also, the way we got here is really the most informal longitudinal study ever: Performance on the same sims, degrading over time. Sure, there are still multiple possible contributing factors, sometimes including changes in the user-generated content on the sim, but sometimes apparently not.

[Qie Niangao]

4 hours ago, rasterscan said:

My current peeve is wtf has happened to my mesh body alpha hud settings.  I set the alpha slices to suit whatever I am wearing, TP off to DJ or whatever, and bosh, post tp the alpha settings are lost, chunks of skin showing through my ill fitting expensive jacket. There is a secondary alpha hud issue for me, if I click a saved alpha preset, it can take upwards of 60 seconds or more for the preset to take effect, I sit here watching the alpha slices change maybe one every 5 seconds or so, its really weird. Still, mustn't grumble, it doesn't happen all the time.

I've seen my alphas change with TP, too (at least with my Slink male avatar, maybe others) but not often enough to try to figure out what's going on. (For the past week or so I have auto-alpha turned off and haven't seen it, but I often go weeks without seeing it again.)

The second thing, though, with the horridly delayed response to mesh avatar HUD settings: that's a classic first indication of degraded scripts run percentage. Check the sim statistics the next time this happens and I bet you'll find that number below 50% -- even if the sim is otherwise perfectly normal with no time dilation or anything. (Of course that's the way script performance is supposed to degrade, without affecting physics or anything else, but we think it's somehow degrading too soon.)

[Aishagain]

I'll just repeat what I've said umpteen times before:

My homestead has run the same number of scripts for over two years now, give or take a few plus whatever I am wearing at a given time.  In the past, until a few weeks ago, the region ran 99+% of them per frame.  Then, at a restart suddenly (and to my shame I didn't make a note of the date) the proportion run per frame dropped to <60%. Since then despite various rolls and restarts, that figure has been in the 40-65% range.

It wasn't gradual, it wasn't that I rezzed something new.  So just what caused the performance drop if it wasn't LL putting more regions on each server?

Edited April 25, 2019 by Aishagain
Additional text

[Theresa Tennyson]

1 hour ago, Aishagain said:

I'll just repeat what I've said umpteen times before:

My homestead has run the same number of scripts for over two years now, give or take a few plus whatever I am wearing at a given time.  In the past, until a few weeks ago, the region ran 99+% of them per frame.  Then, at a restart suddenly (and to my shame I didn't make a note of the date) the proportion run per frame dropped to <60%. Since then despite various rolls and restarts, that figure has been in the 40-65% range.

It wasn't gradual, it wasn't that I rezzed something new.  So just what caused the performance drop if it wasn't LL putting more regions on each server?

With moving various services off of the main region simulation to other systems, more outside communication needs to be done and even the most efficient possible communication takes time. The "more regions on the same server" theory reminds me of the movie Chicken Run where Babs just can't wrap her mind around the idea that life could exist without a farmer.