Is LL putting more sims on fewer servers?

[Coffee Pancake]

3 hours ago, MBeatrix said:

I am curious too, but not enough to spend my time in-world doing what Lindens are supposed to do. Also, I'm sure that they gather more and better data than any people from forums.

They do depend to a fair extent on residents, content creators, third party developers and JIRA witches & wizards (aka @Whirly Fizzle) to report bugs, service issues, etc.

If something is wrong, we are very good at making noise about it, we're less good at really drilling into a problem and coming up with specifics. Finding, fixing, and testing if a bug has been fixed almost always depends on finding a step by step repeatable reproduction.

SL is a stack of complex systems interacting in various ways, so sometimes finding a way to trigger a bug or even just decide what information to log can pose a real conundrum. If you have two tiny bugs that for the most part go unnoticed, when the planets align and their powers combine we end up with an invisible monster that would be at home in any 60's sci fi. Anecdotal information like 'it happens when I TP sometimes' is akin to telling the police that the suspect was humanoid, got away in a car and wasn't wearing a hat.

If and when it's possible, it's always better to try and work out what conditions and steps are involved to trigger an issue repeatably. We get this part right and it can literally save a developer going down some rabbit hole for days, which in turn gets the fix completed and in our hands much faster.

[Profaitchikenz Haiku]

I just went back to look at the parcel I was at yesterday. Time dilation and FPS was still 1.0 and 45, number of objects, scripts and active objects was unchanged, there were five avatars in total in the sim, but the scripts run time was now 70%. One of the avatars left, and five minutes later scripts run time had dropped to 60% and remained there for the ten minutes I stayed.

This suggests to me that the low values of scripts run time is not due to the activities of any avatars or scripted objects in the sim.

[MBeatrix]

1 hour ago, Profaitchikenz Haiku said:

I just went back to look at the parcel I was at yesterday. Time dilation and FPS was still 1.0 and 45, number of objects, scripts and active objects was unchanged, there were five avatars in total in the sim, but the scripts run time was now 70%. One of the avatars left, and five minutes later scripts run time had dropped to 60% and remained there for the ten minutes I stayed.

This suggests to me that the low values of scripts run time is not due to the activities of any avatars or scripted objects in the sim.

Today may not be the best time to check that, as there are rolling restarts and it's possible that they partially affect the region.

[Profaitchikenz Haiku]

 But they haven't happened there yet, so I'm going to be interested to see if they cure the problem. Today is actually an ideal day

[MBeatrix]

1 hour ago, Profaitchikenz Haiku said:

 But they haven't happened there yet, so I'm going to be interested to see if they cure the problem. Today is actually an ideal day

Right. But what I mean is that when you were there you didn't know what was happening with the other regions running on the same hardware.

Anyway, your conclusion may be right. One of the things I've been wondering is that the issue (if it really is an issue) may be caused by the simulator code itself.

Edited May 7, 2019 by MBeatrix

[NeoBokrug Elytis]

I run The Wastelands, 10 regions big, a mix of Full Regions and Homesteads.  I like to keep my estate running nice, so I am a bit of a stats hawk.  Sometime last year things started to sometimes get bad enough to notice, and since then I have been doing casual research.  Here's what I know so far: @Oz Linden, @Whirly Fizzle

"Top Scripts" in the region debug menu always reports HALF or less of what the Statistics menu says.  Even with "Spare Time" available, "Scripts Run" doesn't seem to use it.

"Scripts Run" seems to be affected by a regions networking burden in addition to the obvious events per second (more scripts).  You will find that regions that have a exceptionally bad "Scripts Run" stat directly correlates to how much networking it's doing.  Could be object updates, or packets in/out, and especially noticeable when people teleport in/out. 

Now, when regions come online they have been increasingly slow for all the "services" to fully start.  Specifically the dataserver() event is most noticeable right now, sometimes a few objects that I use to monitor the estate are slow to come online and take a few retries.

I also run an Experience and during todays rolling restart someone tried to interact with an object that uses llRequestExperiencePermissions() and it returned an XP_ERROR_INVALID_EXPERIENCE despite that script being compiled for an experience for years.  It took probably 5 minutes for it to finally work properly without my intervention.

As strange as it seems, I think networking for the regions is borked or throttled and is probably a bigger meta issue that just happens to affect "Scripts Run" as well.  I would probably look into the HTTP/UDP changes that have been made in the past year.

[Qie Niangao]

1 hour ago, NeoBokrug Elytis said:

I also run an Experience and during todays rolling restart someone tried to interact with an object that uses llRequestExperiencePermissions() and it returned an XP_ERROR_INVALID_EXPERIENCE despite that script being compiled for an experience for years.  It took probably 5 minutes for it to finally work properly without my intervention.

You may have been lucky not seeing this before. I've been running an Experience across twenty or so sims for years, but it (mostly) does nothing with Experience permissions -- rather, it uses the KVP persistent store to communicate telemetry among those locations several times a minute at each site. During Experience beta and ever since, pretty much every rolling restart one or more of those locations hits some Experience snafu (usually, yes, XP_ERROR_INVALID_EXPERIENCE) that takes a minute or several to recover.

That's not to dispute the larger point about Network being a likely source of worsening problems. There is definitely a "blocking" problem somewhere -- we can really see it when sims get into the weird state where "Sleep Time" isn't going to Spare Time but is instead booked to Script Time. I had mostly seen that on @animats Vallone region, but last week I was seeing it on Tussock* just before the restart, and after that restart all the Spare Time returned. I don't know if sims are blocking on network or something else, but that "Sleep Time" anomaly sure seems as if it must mean something.

___________
*ETA, forgot to mention: I also saw it on Peocila last Friday (a couple days after its rolling restart) during Fifty Linden Fridays at a local merchant. That region was restarted again on Saturday, but I see it's currently in deep script lag again (about 15% run) with 0 Spare Time and close to 14 msec Sleep Time.

Edited May 21, 2019 by Qie Niangao

[NeoBokrug Elytis]

Additionally, I forgot to mention that regions sometimes degrade in performance for no describable reason.  The region had only been online a few days (Tuesday to Saturday), but by that time it got bad  Case in point:  My main region "The Wastelands" had only 33% scripts executed one day.  It took TWO restarts to clear it back up to 100% -- but the restarts fixed it nonetheless.  This happens more often than I'd like to admit on all of my regions.

Just last week I had one region drop down to 50%, and it was because someone had rezzed a collection of objects that were pounding the viewer with minor object updates; a steady stream of 0.5mb for each av in view.  In total these objects also used about 1.5ms of script time (I know, a lot), but relatively speaking a small fraction of the total script time in the region.  As soon as these objects were removed, UDP data to the viewer more than halved, and scripts executed jumped back up to 75-80%.  That's my basis for script execution being related to a network problem.

Over a month ago I used the SALT HUD to request some mega prims for something I'm working on.  At the time it seemed like it was broken, and I assumed it was because it's old.  However, just two days ago I got two inventory offers for the prims I requested way back when.  Now maybe all the delivery nodes were all offline, or maybe the tubes at LLs end need to be flushed out with lottery balls.

@Qie Niangao  I've seen it occasionally in the dataserver() event after a restart, but that's why I usually wait a little bit to do anything.  I've been pushing that timer back further and further as time goes on.  However this instance of XP_ERROR_INVALID_EXPERIENCE was specifically with llRequestExperiencePermissions and it lasted for about 5 minutes before things suddenly worked in the region.  What's baffling is, that the player got a game HUD from a neighboring region and walked into the problem region (where upon more checks were made) and the HUD spat out the same error.  That shows that communications with experiences (and in general) is lagged until the region catches up.

I really think this is a big networking problem that's been quietly growing for years.

[animats]

13 minutes ago, NeoBokrug Elytis said:

Additionally, I forgot to mention that regions sometimes degrade in performance for no describable reason. 

That's what drives me nuts with Vallone. It's in script overload, I complain to support, they move the region to a different server, and then there's some spare time. A few days later, there's no spare time again, and restarts don't fix it.

It's as if there's a resource leak somewhere in the sim code, and it's persistent over restarts, but not moves.

Edited May 21, 2019 by animats

[Torric Rodas]

Some good news.  The lab has reopened my original JIRA on this and accepted it. On Monday Oz and Simon said that they are actively working on this issue.  You can follow the JIRA  here https://jira.secondlife.com/browse/BUG-226851

[Aishagain]

Additional to what Toric mentioned, Rider Linden was on my region on Monday collecting data on the region, which has had <50% script run for months, irrespective of how many restarts I give it.  Prior to this phenomenon beginning script run had never dropped below 99%.

[Resi Pfeffer]

On 5/23/2019 at 9:16 AM, Torric Rodas said:

Some good news.  The lab has reopened my original JIRA on this and accepted it. On Monday Oz and Simon said that they are actively working on this issue.  You can follow the JIRA  here https://jira.secondlife.com/browse/BUG-226851

Is there a way to contribute to a Jira? Im renting a homestead parcel what has about ~30% script run since weeks (Higginston), and the landlord GLH just doesnt know what to do at all.

[Aishagain]

4 hours ago, Resi Pfeffer said:

Is there a way to contribute to a Jira? Im renting a homestead parcel what has about ~30% script run since weeks (Higginston), and the landlord GLH just doesnt know what to do at all.

No, Resi, you cannot comment but you can "watch" the subject.  Once a JIRA has been "accepted" by LL, no further comments can be added but adding yourself to the watching list will both keep you up to date with changes made to the JIRA and show interest in the matter.

Edited May 24, 2019 by Aishagain
Correcting predictive text

[Qie Niangao]

4 hours ago, Resi Pfeffer said:

Is there a way to contribute to a Jira? Im renting a homestead parcel what has about ~30% script run since weeks (Higginston), and the landlord GLH just doesnt know what to do at all.

If you're not already, try logging in to the jira (it'll show your profile icon in the far right corner of the black menubar when you're logged in), then scroll all the way to the bottom of the jira you want to comment on and there should be a "Comment" button.

I popped over to Higginston and it's behaving kinda normally for a Homestead: of the 22.222 msec frame duration it's reserving about 16 for other Homesteads running on the same core. There are kind of a lot of scripts on this homestead, over 2000, which is equivalent to 8000 on a full sim. Sometimes you can get away with that but it depends how busy those scripts are... and, as we've been noting, it seems that sims are just generally less tolerant of heavy script loads these days.

[Qie Niangao]

3 minutes ago, Aishagain said:

Once a JIRA has been "accepted" by LL, no further comments can be added

Oh, sorry, that's probably correct. It seems willing to let me comment (though I haven't pressed the "Add" button in the comment editor), but the jira lets me do stuff I probably shouldn't be trusted to do, clueless as I am about how the Lab is using the jira system.

[Resi Pfeffer]

9 minutes ago, Qie Niangao said:

If you're not already, try logging in to the jira (it'll show your profile icon in the far right corner of the black menubar when you're logged in), then scroll all the way to the bottom of the jira you want to comment on and there should be a "Comment" button.

I popped over to Higginston and it's behaving kinda normally for a Homestead: of the 22.222 msec frame duration it's reserving about 16 for other Homesteads running on the same core. There are kind of a lot of scripts on this homestead, over 2000, which is equivalent to 8000 on a full sim. Sometimes you can get away with that but it depends how busy those scripts are... and, as we've been noting, it seems that sims are just generally less tolerant of heavy script loads these days.

Thank you for having a look at.
I think i got it... Its not necessarily my or my neighbours fault alone, if the script run goes down. Its also caused by the remaining 15 homesteads running on the same core.
Sounds not like we can do a lot against... but i will have a look at my setup at least.

Edited May 24, 2019 by Resi Pfeffer
typo

[Qie Niangao]

15 minutes ago, Resi Pfeffer said:

 the remaining 15 homesteads running on the same core

actually only 3 other homesteads on the same core. Might be 15 others on the same server -- or potentially more, I just don't know how many cores are on their current hardware platform(s).

But anyway yeah, other than registering any evidence that performance has indeed worsened, there's not much to be done except try to make the most of what we think is diminished script capacity -- which may mean encouragement to find less script-intensive content, but that would be up to the Estate manager to consider. (I can imagine they might worry their tenants just don't want to be hassled by the landlord, and some tenants might even react by increasing script count.)

[Billy Daxter]

On 4/26/2019 at 4:32 AM, Aishagain said:

OK Billy , you make the valid point that there is no logical reason for LL to stack more regions per core.  So then what IS causing this very real performance hit to script run?

Please understand that I am clear about the use of the in-viewer admin tools and the performance drop was very sudden with a set of quite well integrated and understood items.  Nothing else had changed.

My question is and always has been: since I had changed nothing, why did script performance drop so drastically?

That's really the question.  

Think for a moment though through your assertion.  "since I changed nothing".  Are you sure?  Not a single new object rezzed that contains scripts?  Perhaps you actually do have a completely static unchanged environment but that's in most cases the exception rather than the rule.  The problem is that most sims tend to change over time.  That change tends to be the addition of more scripted objects. 

As well.  Objects themselves are not a static thing that remains constant.  Content creators are making more capable products that do more things and thus as a result use more script time.  This is why I only buy copy and mod products so I can turn many of these "features" off.  

We can all get out our pitchforks and complain.  It certainly feels good but accomplishes very little.  How many people have actually gone through their sims and have done something to turn off scripts that aren't being utilized?  I have done this many times before and have managed to gain a lot of idle time as a result.  

Some good candidates for script killing

• color change scripts
• landscape season scripts
• door and window scripts in builds that don't need said doors or windows opened or closed regularly
• Sit scripts in objects that aren't used for sitting
• light switch scripts when you leave the lights on all the time
• Any script used "occasionally" that can be turned back on when you want to make a change.  

Avatars also have many of the same scripts inside them and they come and go into our sims.  They also have the same level of script bloat that grows over time.  How many scripts does your avatar have in it that aren't being used at all?  Do you wear your alpha hud 24/7 or do you make copies of alphaed bodies and save them as outfits with the clothing?  

Firestorm has some really good script diagnostic stuff built into it and also the ability to turn off scripts for objects.  Explore that scripts pie menu sometime when right clicking on objects. 

What I'm getting at is that we are also part of the problem here.  There's stuff that we can actually do that will make things better in the short term on our own. The sim I am on right now has 17ms of idle time.   

 

[Aishagain]

Billy, I said nothing had changed.  NOTHING! One night I logged out and script run was >99%. Next day it was <50%.

[Billy Daxter]

56 minutes ago, Aishagain said:

Billy, I said nothing had changed.  NOTHING! One night I logged out and script run was >99%. Next day it was <50%.

I never noticed this myself on my land...  What would be interesting is to compare the time frame against the release notes of what ended up going out that changed this.  You might have some objects that worked well on on one version and then have a serious performance penalty on another.  You could probably identify the offending objects by looking at their script use and then either shutting their scripts off or eliminating them all together.  

In a "user created world" some scripters are better than others and of course everyone's mileage varies depending on what they have rezzed out.  

I once had a similar situation happen to me.  One day I had tons of idle time and the next day it was gone.  Took me a while but I eventually found an offending door script on a boat that I had rezzed that was eating up a ton of physics time.  The tools provided aren't the most user friendly out there but they do work if you take the time to try to understand what they are telling you.  

The important takeaway here is all of our sims are different and it's impossible for content providers or the Lindens to know about performance issues that we are seeing unless we tell them and are able to back our "it's laggy" claims with hard data.  This is sometimes difficult even for people like myself who know exactly what's going on with their sim performance but it's a much better way at pointing the right people at the right problem.  

Edited May 28, 2019 by Billy Daxter

[Aishagain]

All well and good Billy, were any of the other stats changed...they weren't.  The idle time, physics time, spare time or anything else showed no variance.  Just script-run percentage.  I have no explanation, nor have any other the Lindens that have analysed my homestead's performance.  Most recently Rider Linden did an in depth analysis using his Linden tools and found nothing unusual, save the script-run.  It is good that LL are investigating this issue, but so far we have absolutely nothing to go on.

As an aside I rather resent your supposition that I might NOT have looked at this issue in depth.

[Billy Daxter]

2 hours ago, Aishagain said:

All well and good Billy, were any of the other stats changed...they weren't.  The idle time, physics time, spare time or anything else showed no variance.  Just script-run percentage.  I have no explanation, nor have any other the Lindens that have analysed my homestead's performance.  Most recently Rider Linden did an in depth analysis using his Linden tools and found nothing unusual, save the script-run.  It is good that LL are investigating this issue, but so far we have absolutely nothing to go on.

As an aside I rather resent your supposition that I might NOT have looked at this issue in depth.

No offense was intended. I'm really sorry that you viewed it that way.

My assumption was based on the general experience that I have had in my many years on Second Life.  Most people don't dig into things or ask for help.  I applaud you for doing that and hope that you are eventually able to figure out what's causing the trouble.

It's also important to keep in mind that one experience does not describe the experiences of thousands of other people.  I never noticed the event that you described and all of my regions have plenty of idle time to spare.  Does my individual experience represent everyone?  No.  Does yours?  Unlikely.  

Please also don't draw the conclusion that my assumption is that you are "doing something wrong".  From what you have described you are doing the right thing getting the right people involved.  All I'm saying is I don't have the same problem.  Is this because I'm lucky?  Is it because I dig in and know what scripts are running?  Who know's for sure.  There's no data to back any real conclusion.

At the end of the day I replied to this thread to try and be helpful and share the experience I have had working through the same performance issues many describe here.  There's no magic button to be pressed just standard resource management like with any other resource.  If you have X resources available stay under that threshold to avoid issues.  Prims are easier to manage of course as there's a limit to them. Perhaps if we scream louder and make more accusations the Lindens might adjust the Land impact score to help us manage our resources better?  I have a feeling that's not what we really want in the long run.  

In any event.  It's been a lot of fun but I don't see a lot of value in this thread anymore. Since I have managed to offend someone it's probably best I bugger off somewhere else.  

[Aishagain]

It's a shame, Billy.  You must be one of the many lucky ones.  I know quite a few of what must be the Unlucky ones.  No, no-one knows, and as yet certainly not the Lindens.   I did consider removing as many scripts for the region as I could, but I now know someone who did try it with virtually no improvement (OK 5% improvement).  We just keep our heads down and do the best we can.

Sorry I bridled, having been in SL since 2008, I get a bit precious now and then.

[Old Bailey]

I stumbled across this thread as I was looking for answers as to why my sim has suddenly degraded performance.

A simple question. Are their any tools I can use to find the offending objects that are causing both high script load and high physics load on my sim?

 

[Qie Niangao]

1 hour ago, Old Bailey said:

A simple question. Are their any tools I can use to find the offending objects that are causing both high script load and high physics load on my sim?

Physics? That would preempt scripts, so you want to focus there first, and just turning on World / Show Beacons / Physical could give clues -- there shouldn't be many on a sim, ever (which is why the old timey breedables were such a disaster).

Estate managers can look at Top Scripts to get some hints about where those heavy-hitters are. (Actually, Top Colliders, too, which can be a clue about where the physics time is going.)

On Mainland, though... it's a lot more labor intensive and messy.