Massive script time reporting Changes now

[dd Temin]

All residents are now seeing massive changes in  script time reporting in both statics script time and  Region Debug Top script times since the  17 Jan 22 update!

Could the Lindens please give us an  update  on  how all script times have  suddenly been reduced by 1/3 to  1/4 of their historical values?

Actually ,I am pretty sure this is a screw up  by  the Linden programmers.  

 

[Odaks]

I was looking at a Bellisseria region an hour ago, Sirinnial. 100% scripts run but script time only 4 to 5ms.

I'm used to seeing figures more like 30% scripts run and 16ms script time!

This was, I think, the latest discussions about all of this:-

That topic is still active; pehaps a good idea to continue any further quieries there.

Edited January 20, 2022 by Odaks
Correction

[Jenna Huntsman]

20 minutes ago, Odaks said:

I was looking at a Bellisseria region an hour ago, Sirinnial. 100% scripts run but script time only 4 to 5ms.

I'm used to seeing figures more like 30% scripts run and 16ms script time!

That's reporting correctly - If 100% of scripts are being ran per tick, the script time will only report the max time spent processing scripts, any remaining time will be dumped into the 'spare time' pool - What you're seeing there is something on the backend has changed (*ahem* configuration changes) which has allowed the scripts on that region to execute much faster, thus allowing for more scripts to run per tick.

To give an example, the region i'm currently in:

Scripts run: 100%
Script time: 3.717ms
Spare time: 18.420ms

Regions which have a low amount of scripts run would expect to see little-to-no 'spare time' as (to my knowledge) scripts are the last items to process every tick - so if scripts in my region was to exceed about 21ms of run time, the scripts run count would drop as there is no more time to process scripts in that tick.

Edited January 20, 2022 by Jenna Huntsman
English lol

[Odaks]

Yes, thanks @Jenna Huntsman. All of what you say is true. I was really only sort of sympathising with  @dd Temin with regard to the masive difference in the way the stats are now calculated and presented, inevitably causing great surprise to anyone who hasn't been able to keep up with the latest changes. It's a very intricate issue though  and that's why I simply pointed to the thread that is currently running and busting with interesting stuff.         

[dd Temin]

@ Jenna Huntsman   "What you're seeing there is something on the backend has changed (*ahem* configuration changes) which has allowed the scripts on that region to execute much faster, thus allowing for more scripts to run per tick."

I seriously doubt  that there has been some magical change/fix   found and implemented by the Lindens  to suddenly  make scripts run 3 to 4 times more efficiently after 16 years!

It is far more likely that  the Lindens have simply  made  an error in how script times are calculated/reported  for this last server update and are keeping quiet about it while they try to figure out what went wrong this time.

   Any comments by Henri Beauchamp  would be greatly appreciated.    Thanks

[Jenna Huntsman]

@Monty Linden would be the best person to CC into this topic as he has all the knowledge about how the Scripts Run metric is calculated; but to quote from something he said in the thread that @Odaks linked to:

Quote

But from memory and ignoring many exceptions:

• It's a one-second, unweighted mean of
• percentages calculated each frame
• based on the number of scripts scheduled to run relative to
• the number of scripts found to be runnable during the frame.

This leads to questions you can ponder.  Say you have a 5-frame second.  Four frames of one runnable, one actually run script.  One frame of 96 runnable, zero run.  Is the correct metric 80% ((4*100%)/5) or 4% (4/100)?

We also know that some pretty major backend work has / is being undertaken recently, including (but not limited to) an OS upgrade serverside, along with some updates to the Mono engine (?), which could be having an affect on the scripts run metric

[Profaitchikenz Haiku]

4 hours ago, dd Temin said:

Actually ,I am pretty sure this is a screw up  by  the Linden programmers.  

 

I hope not. What i think I see in the regions I follow closely is that there is now more spare time available, and that avatars entering a region no longer cause the 1-second freeze. I second your request for the Lindens to explain a bit more about what they're done, but as I am seeing some realistic performance improvements to scripted objects I don't think this is a screw-up, quite the opposite.

I have been running some scripts for several weeks now in four regions to try and see if users can assess region performance. One of these tests measures how accurate a 1-second timer is compared to the wall-clock. Up until Tuesday this test showed that when there were more than a few avatars in a region, or when some avatars entered or left a region, the timer would miss one or more seconds. Since Tuesday's restart with the new configuration, theese instances of lost seconds have dropped dramatically. The RC channel in the regions I look at had already been switched to the new configuration and had been showing the greatly improved figures for some time now, this is the basis for my saying I think this is an intentional change, not a mistake.

I could of course be wrong, that is the curse of being male...

Edited January 20, 2022 by Profaitchikenz Haiku

[Grumpity Linden]

Loving this thread! 

We're going to publish a more detailed explanation of the work we've done and the improvements you're seeing soon.  Spoiler alert: it's not a mistake in the metrics! 

[Profaitchikenz Haiku]

2 minutes ago, Grumpity Linden said:

Spoiler alert: it's not a mistake in the metrics! 

There goes my reason for living...

[dd Temin]

I honestly Hope you are right.......But if the Lindens had made  this type of massive  improvement in script times (3x to 4 x  reduction), don't you think they would be shouting it from the rooftops of what a great  improvement they had just made?    As opposed to not even mentioning it.

[Fenix Eldritch]

Like this?

 

Edited January 20, 2022 by Fenix Eldritch

[dd Temin]

Thanks for the heads up  Grumpity!      Lindens are alive again

I will be looking forward to  your explaination.  Thanks

[tomm55]

Maybe related to this? https://community.secondlife.com/blogs/entry/9883-second-life-simulator-update-with-script-performance-improvements/

 

Edit: Haha too late 😁

Edited January 20, 2022 by tomm55

[Monty Linden]

Not an accident and not a change in metrics.  The latter are computed the same as they were before.  I can't tell you everything that happened but the blog post gives you an idea of the breadth and depth of it.

[Winged Heron]

Could the changes on 17 Jan 2022  have sunk a SIM terrain by 1.5 mtrs? We are having issues. Some terra took place in one corner but how could that drop the whole SIM by 1.5 meters and this I think have a couple of days before hand. 

The drastic drop change was notified to my as land owner but the person I share the SIM with on 18 Jan 2022. It was not niticeds until then. We are on a Grandfather Sim.

[Monty Linden]

@Winged Heron I wouldn't expect that to be related.  But it is worth a support ticket.

[Grumpity Linden]

49 minutes ago, Fenix Eldritch said:

Like this?

 

yeah, exactly like that! 
that timing worked out funny. the left hand types faster than the right hand or somesuch.

[Winged Heron]

@Monty Linden the drop or the fact it could have been terraforming that drop the whole sim. Monty? Have to remember how to put in a support ticket. Maybe may massive cave system with is still to reach the ground tilted it. Seriously though it is concerning. Because I didn't notice it when I was in last and not sure if that was Sunday. can't remember coming in on 17th and then getting message freaked me big time. Thank you will do a ticker.

[Aishagain]

I should've known, the Lindens were too quiet!  I knew they  were up to something and if they can work this sort of magic on the render pathway we'll all be even happier (even me).

Edited January 20, 2022 by Aishagain

[Lucia Nightfire]

I see it took LL 10 years (See SVC-7889) to finally implement a region-wide HTTP throttle.

Sadly it punishes everyone and not those who are consuming a majority of the region's resources.

So when you enter a region with this going on, your HTTP device's single HTTP call will be met with a 429 response error code.

Edited January 21, 2022 by Lucia Nightfire

[Monty Linden]

3 hours ago, Lucia Nightfire said:

I see it took LL 10 years (See SVC-7889) to finally implement a region-wide HTTP throttle.

An SVC Jira?  Now that's some ancient history.  No, nothing new in HTTP throttle land.  Just the potential for hitting local or remote throttles more easily with more script code being executed.

[Qie Niangao]

16 hours ago, Monty Linden said:

Not an accident and not a change in metrics.  The latter are computed the same as they were before.  I can't tell you everything that happened but the blog post gives you an idea of the breadth and depth of it.

It would be helpful to get a little more sense of what's changed, mostly because I don't know how broadly to extrapolate from the quite astonishing improvements I've seen on some regions that were script-lagged for years (congratulations!) and to make sure not to jinx it by accidentally writing scripts that use up all the goodness.

Also, the blog post refers to compiler modernization and "better code generation" and… well… might we expect benefits from recompiling existing scripts? Or did that all happen behind the scenes already?

[Candide LeMay]

@Monty Linden now that we can rent servers in the same datacenter as LL, would it be possible to increase the throttle thresholds for http requests that stay in the same AWS region?

Edited January 21, 2022 by Candide LeMay

[Monty Linden]

1 hour ago, Qie Niangao said:

Also, the blog post refers to compiler modernization and "better code generation" and… well… might we expect benefits from recompiling existing scripts? Or did that all happen behind the scenes already?

That is one that hasn't happened yet.  The Mono system itself is mostly unchanged from previous releases.  So recompilation isn't expected to deliver a benefit at this time.

[Monty Linden]

1 hour ago, Candide LeMay said:

now that we can rent servers in the same datacenter as LL, would it be possible to increase the throttle thresholds for http requests that stay in the same AWS region?

We probably should look at our throttling levels whenever the environment changes dramatically, as it has now.  Not certain when or if reconsideration will happen but some are thinking about it.  Different thresholds for different targets is less likely to happen.  Throttles are an attempt at fairness as well as civility and special classes are not usually compatible with fairness.  Practically, answering the question "is this targeted for us-west-2?" is more difficult than one would like.  And as Jeff would tell anyone, hosting on AWS is its own reward.