Rumors about improved script performance.

[Monty Linden]

2 hours ago, Aishagain said:

Rubbish metrics or not, why is it, then, that while most other folk are seeing improvements in script performance and "Spare Time" values, I see nothing?

  No change lifts all boats and so many factors are in play that it is impossible to draw conclusions from anecdotes.  This is why good metrics that eliminate independent variables are essential as systems become more complicated.

Abstract scenario...  four scripts, A, B, C, and D, work as a pipeline to perform some useful interaction (A->B->C->D).  They get scheduled in that order and so D produces a response in the same frame as A receives input.  Four scripts become executable in frame.  All four run.  Result:  100% of runnable scripts run, large amount of spare time.  Now say those scripts are scheduled in reverse order for some reason (D, C, B, A).  Only A runs on the first frame, only B runs on the second frame, etc.  A->[frame]->B->[frame]->C->[frame]->D.  Now it takes three full frame times (66ms+) to produce a response.  Metrics:  100% of runnable scripts run, large amount of spare time yet interactivity falls.  Can it get worse?  Yes, it can.  Now have A's second firing become dependent on D's previous output.  In the first case, an output event can be generated every frame, in the last case, only every fourth frame.  Metrics continue to report the same 100% of runnable scripts running and *tons* of spare time but throughput is now 75% lower than optimal.  And nothing has changed other than scheduling order.

 

2 hours ago, Aishagain said:

At this point words fail me.

They'll be back.  They're outside having a smoke.

[Aishagain]

Thankyou @Monty Linden  I am sure I will understand that once my brain can be coaxed back from behind the virtual sofa.

[Ardy Lay]

42 minutes ago, Madelaine McMasters said:

Once I moved into engineering, I encouraged my team to speak to the "idiots" who always outsmarted us by exposing flaws in our hardware and software. The best idiot we knew was our engineering manager, George, who had a remarkable knack for toppling anything he touched. He was possibly the most valuable member of the department, simply because he couldn't be bothered to understand how we thought our stuff should be used. That made him the quintessential proxy for our customers.

I don't mind. I eventually got tired of fixing other people's problems and resolved to create my own.

Haha!  Awesome!  When I was working full time as a product tester, some product developers would just drop stuff to me and give no documentation or instructions other than how they wanted defects reported.  Others had very strict test structures they wanted me to follow.  Most alternated back and forth between exploratory testing and regression testing.  I was never asked "How did you do that?" when I broke something, because I told them in the defect report.  What I was asked a lot was "Why did you want to do that?" and variations on the question.  "What made you think to try that?"  Quite reasonable questions.  "What did you expect to happen when you tried that?" was already explained in the report.  It was quite common for the product owner to direct the developer to prevent that breakage by eliminating the possibility someone would again do what I did to break the product.

Hey, if there is a menu item, I will try it.  If there is a command, I will try it.  You asked me to TEST THIS THING.  Hehe.

Red Hat got so tired of me.  I was, like, "Bobby, baby, you want this on desktops?  I will crawl all over this stuff like a desktop computer user does.  There will be paw prints EVERYWHERE."  Wasn't long before they decided to let the desktop market look elsewhere and turned their focus on Enterprise Server products.  As you can see, I now wear a white top-hat instead of a red fedora.

[Andred Darwin]

The performance improvement is quite noticeable, comparing to what it was a few weeks ago (and welcomed! Thank you for that!). 
I think its also true, the script performance is way lower than what it used to be before the cloud move, and by a lot... the cloud move from a user standpoint, brings a huge improvement due to the network speed provided by AWS... but, from a script performance standpoint, the chosen virtual machines just cant handle the same as it used to be.

I looked at a few regions, where a few friends use parcels and the ones I where I use parcels... and the biggest issue for me in this topic, is that the ones that shows low performance metrics, its mostly due to one or two users, that probably without even knowing, are "sucking" most of script time available to the region... 
I'm going to use the Brown as an example... one user has more scripts in 2 small parcels than all other users together in the region (in fact almost triple in size)... and in another region, after the "upgrade/update", a user simply doubled the number of objects (scripted pets) bringing the region performance to be worst than what it was before...

So... my point is, by increasing performance of the region without some limits available to users, you are just pushing the problem a few weeks/months further... (sincerely, seems like its the management tactic)... It's more than overdue to add such limits or at the very least invest in educate the players on simple things they can do to get the most of the experience SL can provide... like watch script counts... don't overload a small area with too many high poly mesh items, you don't need texture changing trees with listeners open when you dont change that texture that often... , you don't need 1000 furniture pieces with 100 animations each, when the only one using it, is yourself, etc... 
Also, people that enjoy breedables, deserve a set of regions that can handle such objects demand, where they can be together with other users that enjoy it, etc... (Brown as mentioned above... guess what the user likes to do...)

Current metrics, even not being the best way to technically measure performance as you mentioned, they are comparable when looking at the same region, in a short time frame (a week as an example) as long as there were no major change in scripted objects, etc... at least in my case and most of friends, regions below 90% in script run with close to zero spare time suffers the most, less than 80% is downhill... and that is reflected in what matters the most, which is user experience... 
Monty, its great to see that you come and comment at the forums, thank you for that, but what you are hearing is user experience, and most don't care about the technical explanation or right measure of performance... LL gets paid for it... It doesn't matter the performance improvement if in most regions, players still put the simulators under stress (and usually one or two per region, and in most cases breedables impacting scripts run or boats, furniture impacting script events per second... ) 

[Henri Beauchamp]

2 hours ago, Kathrine Jansma said:

TextureTest2 looks useful for tuning the pipeline. But guess i need a bigger network pipe (as it can saturate my 100MBit/s link but not the CPU when running with 16 texture decode threads in CoolVL Viewer and forcing full size texture loads).

You mean something like this ?... 😜

The textures in this test sim are not large enough to saturate the bandwidth... Note how fast the Cool VL Viewer rezzes it too (about 15 seconds, starting the timer at the ”Loading world” login step), here on a 9700K @ 5GHz with 1Gbps FTTH Internet link.

Edited December 6, 2021 by Henri Beauchamp

[Lucia Nightfire]

7 minutes ago, Andred Darwin said:

Also, people that enjoy breedables, deserve a set of regions that can handle such objects demand, where they can be together with other users that enjoy it

No. No they don't.

Far too many breedables models are horribly scripted and run high script timing.

Couple this with rezzing dozens of them and you easily exhaust script timing in the region.

Since we do not have an enforced fair resource usage policy, you often see breedables owners owning 1% of the land in a region while consuming 30% - 50% of the region's script timing.

Users need to be aware of and understand the resource usage their content is using else be penalized by a fallback protection.

[Profaitchikenz Haiku]

2 hours ago, Monty Linden said:

our scripts, A, B, C, and D, work as a pipeline to perform some useful interaction (A->B->C->D).  They get scheduled in that order and so D produces a response in the same frame as A receives input.  Four scripts become executable in frame.  All four run.  Result:  100% of runnable scripts run, large amount of spare time.  Now say those scripts are scheduled in reverse order for some reason (D, C, B, A).  Only A runs on the first frame, only B runs on the second frame, etc.  A->[frame]->B->[frame]->C->[frame]->D.  Now it takes three full frame times (66ms+) to produce a response.  Metrics:  100% of runnable scripts run, large amount of spare time yet interactivity falls. 

An interesting scenario, but you are only talking of scripts sending data to scripts: I tested llRegionSay, llSout, llSay, llChat and link_msssages in two parcels over a 2 week period and not one message failed to be received, let along score a delay of any measurable amount. These activities are relatively light but equally vital for things like Huds, and so it's good to know they don't seem to show degradation, despite what the stats might show.

But in that same period I was recording instances where moving_start or moving_end failed to be raised with a 30-second timeout period beyond the point at which they ought to have occurred. In all the cases where moving_start had not been raised the object had indeed failed to move, but more puzzling was that in the moving_end failures, the object had got to within half a metre of the destination, just decided to do a Zeno's paradox demonstration and not quite make it.

My thoughts are that scripts on their own are not subject to problems, it's when they are trying to cause movement in world that degradation of performance is becoming noticeable. 

[Profaitchikenz Haiku]

13 minutes ago, Lucia Nightfire said:

Couple this with rezzing dozens of them and you easily exhaust script timing in the region.

Funny you should say that: I was asked yesterday to examine a looprezz-derived script that was apparently failing to work for the owner. I rezzed the object on my parcel and proved it rezzed 16 items in an ellipse, but then got an immediate burst of moving_end errors from the funicular, which has been running error-free for over two months now.

I repeated the test several times but couldn't get any further errors to occur, it seems that the initial shock to the region was a one-off, but it's left me puzzled. There are plenty of prims free so that's not the cause, and the rezzed items were just flexible path prims, unscripted, so it wasn't a hit on the region script resources either.

Regarding the suggestion of a suite of benchmark tests to examine region performance issues, I think we need to include things like keyframed motion, texture changes on display boards, object-rezzing, rather than just a looped counter seeing if 11000 iterations of a routine takes more or less time than anticipated,

Edited December 6, 2021 by Profaitchikenz Haiku

[Andred Darwin]

8 minutes ago, Lucia Nightfire said:

No. No they don't.

Far too many breedables models are horribly scripted and run high script timing.

Couple this with rezzing dozens of them and you easily exhaust script timing in the region.

Since we do not have an enforced fair resource usage policy, you often see breedables owners owning 1% of the land in a region while consuming 30% - 50% of the region's script timing.

Users need to be aware of and understand the resource usage their content is using else be penalized by a fallback protection.

Part of me, agrees 100% with you Lucia, but I interacted with a few of those players, and they "freaking" love those (I mean, they really enjoy breedables), and there was no way for me to just say (neither my place to it),  "hey, your freaking breedables are a drag in this region... " ,  I just think its an opportunity for SL, people love it.... why not have an option suited for them... 

LL will not intervene on how an item is scripted... we can't help, bad or good scripts are subjective... its a fact, that whenever they are present, performance is most likely affected.... so why not own it and try to make it work... (regions tweaked for it, maybe with some sort of covenant... hopefully same price, everyone happy.... )

Anyway, that was just a thought.... again, I think what you mentioned makes total sense.... but I also think could be an opportunity for SL... and would not require a huge effort... (a few AWS clicks using a suitable machine specs and a few blog posts... ). I'm sure the couple users I had contact with would be very happy to see their pets functioning better  ... and I would love to see the region running without them.... 

 

 

 

 

[Lucia Nightfire]

8 minutes ago, Andred Darwin said:

Part of me, agrees 100% with you Lucia, but I interacted with a few of those players, and they "freaking" love those (I mean, they really enjoy breedables), and there was no way for me to just say (neither my place to it),  "hey, your freaking breedables are a drag in this region... " ,  I just think its an opportunity for SL, people love it.... why not have an option suited for them... 

LL will not intervene on how an item is scripted... we can't help, bad or good scripts are subjective... its a fact, that whenever they are present, performance is most likely affected.... so why not own it and try to make it work... (regions tweaked for it, maybe with some sort of covenant... hopefully same price, everyone happy.... )

Anyway, that was just a thought.... again, I think what you mentioned makes total sense.... but I also think could be an opportunity for SL... and would not require a huge effort... (a few AWS clicks using a suitable machine specs and a few blog posts... ). I'm sure the couple users I had contact with would be very happy to see their pets functioning better  ... and I would love to see the region running without them.... 

 

 

 

 

Sorry, but I don't agree that nothing can and/or should be done to reign in excessive script usage, especially in scenarios where one land owner's/renter's content affects the entire region and everyone else's content's performance.

Aside from implementing a fair usage policy enforcement protocol, we also need to do something about how script impact on server cost is completely lost within today's mesh environment because it's incorrectly calculated to begin with.

See https://jira.secondlife.com/browse/BUG-227584

[Andred Darwin]

2 minutes ago, Lucia Nightfire said:

Sorry, but I don't agree that nothing can and/or should be done to reign in excessive script usage, especially in scenarios where one land owner's/renter's content affects the entire region and everyone else's content's performance.

Aside from implementing a fair usage policy enforcement protocol, we also need to do something about how script impact on server cost is completely lost within today's mesh environment because it's incorrectly calculated to begin with.

See https://jira.secondlife.com/browse/BUG-227584

Again, part of me agrees 110% with you... but looking at the past 18 years ... how many times SL actually intervene? (Gee, there are avatars with more scripts and polygons than entire parcels! )

Also, some people would just leave when approached that way... you are dealing with something they like SL for, they spend money in land and pets , which they would recommend the system to others...  although sound.... it will be a nightmare to implement...  all for just a "set" of all items available in SL that people actually like with a risk to just lose that revenue... ( its not they're fault... ), it's much easier to own it... with limits and allow people continue to enjoy SL ... 

[Lucia Nightfire]

20 minutes ago, Andred Darwin said:

Again, part of me agrees 110% with you... but looking at the past 18 years ... how many times SL actually intervene? (Gee, there are avatars with more scripts and polygons than entire parcels! )

Also, some people would just leave when approached that way... you are dealing with something they like SL for, they spend money in land and pets , which they would recommend the system to others...  although sound.... it will be a nightmare to implement...  all for just a "set" of all items available in SL that people actually like with a risk to just lose that revenue... ( its not they're fault... ), it's much easier to own it... with limits and allow people continue to enjoy SL ... 

When one's enjoyment of SL begins to affect other's enjoyment whether intentionally or unknowingly, then there is a problem and there should be awareness, education and lastly, protection if/when those two fail.

[Andred Darwin]

11 minutes ago, Lucia Nightfire said:

When one's enjoyment of SL begins to affect other's enjoyment whether intentionally or unknowingly, then there is a problem and there should be awareness, education and lastly, protection if/when those two fail.

I agree with you... I just think there is opportunity in this specific case... its not easy to deal when there are no rules.... ( which I believe its the reason why SL is still alive, and at the same time the reason why its stuck for so many years. )

Edited December 6, 2021 by Andred Darwin

[Profaitchikenz Haiku]

On 12/6/2021 at 7:43 PM, Monty Linden said:

This is why good metrics that eliminate independent variables are essential as systems become more complicated.

I've started tidying up some scripts that can be posted as examples of how to measure what might be happening in a region, but in writing up a description I've realised something. I don't know precisely what "% Scripts Run" is actually purporting to show. @Monty LindenIs it

a) x% of all scripts are able to complete what they should have been able to do in the frame/timeslice in question

b) A typical script is able to complete x% of what it should do within the frame/timeslice in question

c) neither and a reasoned explanation can be provided

d) none of the above, it's analogous to the food pellets that the rats in the Behavioural Psychologists cages gambled for

So far I am thinking that you favour c) and Coffee favours d).  I, and many others, have always believed it's a) but reviewing some of my earlier results I'm now pondering it might be b)

 

 

Edited December 9, 2021 by Profaitchikenz Haiku

[Coffee Pancake]

That's why I'm thinking a more generic set of benchmarks that test a load of things scripts can do.

That way we get a whole set of numbers and can than run the tests to track changes over time and from region to region and not really need to care what the stats floater says.

[Profaitchikenz Haiku]

2 minutes ago, Coffee Pancake said:

That way we get a whole set of numbers and can than run the tests to track changes over time and from region to region and not really need to care what the stats floater says.

Is there not a risk in going so that we're all of us going to be slowing SL down? I'm reminded of the time a VAX cluster I was working on began to run slow, and everybody then suddenly put on monitor to see for themselves, and.... we swam through treacle

[Monty Linden]

6 minutes ago, Profaitchikenz Haiku said:

Is there not a risk in going so that we're all of us going to be slowing SL down? I'm reminded of the time a VAX cluster I was working on began to run slow, and everybody then suddenly put on monitor to see for themselves, and.... we swam through treacle

Hahaha.  I'm an ex-Decie who worked in VMS engineering during VAXcluster development (not my project).  I've returned to make your life miserable.

[Profaitchikenz Haiku]

2 minutes ago, Monty Linden said:

I've returned to make your life miserable.

 Somebody once asked me "How come your're always happily singing and humming? What' your secret?" and without thinking I answered "I'm too stupid to be miserable"

Years later, I've realised how true those words were.

[Monty Linden]

To the question...  not a) or b).  Those do not have practical implementations and probably not useful.  Right now, it's a mix of c) and d) and I want to get these documented properly at some point.  But from memory and ignoring many exceptions:

• It's a one-second, unweighted mean of
• percentages calculated each frame
• based on the number of scripts scheduled to run relative to
• the number of scripts found to be runnable during the frame.

This leads to questions you can ponder.  Say you have a 5-frame second.  Four frames of one runnable, one actually run script.  One frame of 96 runnable, zero run.  Is the correct metric 80% ((4*100%)/5) or 4% (4/100)?

[Profaitchikenz Haiku]

So can I assume that

some scripts, such as very simple lightweight ones, will always run every frame when they need to

more complex scripts, (possibly waiting on events), might not run at all in any one frame and so might perhaps only get run in 40% of the frames within the measuring period?

I see what you mean about the metric.

I think the crucial thing is how many scripts didn't get their full run time. If a large number of simple scripts all complete, their effect is to skew the metric too far towards the "I'm alright Jack" side.

In your example to ponder, the single script that managed it isn't the issue, it's the held-back others that indicate something isn't optimal.

The implications for me is that the test scripts I am putting up may need to be fleshed out with a lot more stuff so that they start to challenge the scheduler.

[Profaitchikenz Haiku]

I have completed some testing in five regions using a script to find as many prime numbers as the script memory will allow. I chose this method after Monty's explanation because I realised some scripts might be parked in the "not able to run" category simply because they might be waiting for the timer, as well as for something far more sinister such as a data server request or they issued one of the calls that sleep the script for anything from .1 to several seconds.

Here are the results:

Region, details, %script run, time to find first 403 primes

Premium sandbox - almost empty -                          100% - 427 secs 

Mainland - light residential all in use -                       100% - 640 secs

Full private region - busy residential 2/3rd in use - 100% - 759 secs

Mainland - residential -                                                   65% - 1388 secs

Mainland - residential -                                                    32% - 2047 secs

 

I see a correlation between an increase in the time required to complete the task and a decreasing value of the % scripts run figure. I see it as a reasonable metric, perhaps closer to seaweed than a barometer, but it nevertheless does seem to be of some use as-is.

Do bear in mind that the people who began clamouring abut this a couple of years ago weren't constantly watching their stats bars and started seeing a low figure and went "Ooh That's not right!" and became obsessed with the figure.

They saw things around them not quite behaving as expected, and in looking through the things they had available, came upon this scripts run figure. They then discovered that if they requested a region restart and repeatedly did so until they got a better % scripts run figure (and it could take half a dozen goes), the poor performance that had been annoying them went away.

 

ETA, the script that produced these figures can be found in the sub-forum LSL Library, together with a few others. I would be interested to see how homesteads perform.

Edited December 9, 2021 by Profaitchikenz Haiku

[Monty Linden]

14 minutes ago, Profaitchikenz Haiku said:

I see a correlation between an increase in the time required to complete the task and a decreasing value of the % scripts run figure. I see it as a reasonable metric, perhaps closer to seaweed than a barometer, but it nevertheless does seem to be of some use as-is.

Used in moderation, it is a tolerable vice.  A single-valued measure of goodness for something as complicated as a region is a hard thing to devise.

[Profaitchikenz Haiku]

Finally, a result, even though it's a by-product. Not the full in-region testing but at least some confirmation that running scripts in-world to examine region performance issues may indeed have some merit. (I was undecided at first, the concept goes against my ingrained ideas about reference points, calibration, independence of measurement from measuring instrument, etc).

As a by-product of running three tests that look at time interval drift and KFM and moveToTarget scripted objects, I got an error message from one of the non-test inworld objects that lined up with one of the test events to give an explanation of what might have been the cause of the problem.

At rare intervals, one or both of a pair of Funicular cars refuse to reach their destination, sometimes not even beginning to move. Here is a recent example (the first in quite a few days

[14:26]  funicular car 1: Timeout moving_end at <3.12700, 135.85810, 23.19921> 
                                  target <3.12700, 141.73800, 30.00500> error 8.993976 
                                  in region 4 LastFps 11.242760 thisFps 40.748150 started 14:26:04

A KFM funicular car has timed out when it should have reached the end of the journey, which began at 14:26:04, and in fact had barely moved at all since the error 8.993976 is the missing distance it hasn't achieved. 

In the region monitoring test for timings this shows up

13:29:15,Agents,1,2
13:30:33,Agents,2,1
14:14:43,Agents,1,2
14:14:49,Agents,2,1
14:26:04,Agents,1,4
14:26:04,Drift,0,3
14:26:33,Agents,4,3
14:26:34,Agents,3,2
14:26:35,Agents,2,1
14:27:49,Agents,1,3
14:27:49,Drift,3,4
14:27:49,Drift,4,3
14:28:24,Agents,3,2
14:28:25,Agents,2,1
14:29:01,Agents,1,2
14:31:32,Agents,2,1

Drift 0,3 means that three consecutive one-second interval timer events must have been missed completely, Drrift 3,4 later shows an additional second dropped, followed by drift 4,3 where it appears the time-dilation mechanism has responded to try and cope with an in-region problem? 

At 14:26:04, the time the Funicular started the journey, 3 agents enter the region (possibly in a vehicle as there is a highway adjacent to my parcel). 3 seconds get missed.  (Notice that the other agent occupancy changes before and after do not cause timing drift, these particular agents had something heavy with them).

Three lost seconds of events isn't all that occurred, a KFM motion was effectively frozen.

There's nothing that can be done about it, obviously, the agent-arrival mono hit is well known, but it means I can put aside previous thoughts about region-idling or resource-shifting to implement load-balancing as the reason for odd and intermittent poor performance.

The tests still have to run for another two days to get a full week's running, and I then have to graph the results, but it does look as though resident-monitoring isn't totally without merit.

Technically of course this post is off-topic since I didn't say anything about the Scripts Run % figure, I'll have to hope those scrutinising the posts are tolerant tonight.

 

 

Edited December 21, 2021 by Profaitchikenz Haiku

[Profaitchikenz Haiku]

@Aishagain Sorry, I guess it just seemed like a mess and a let down. I'll try to explain.

For close to two years now I've struggled to try and work out just what has been going wrong with some  things in the world I make, and for a long time it seemed that wherever the Scripts Run % figure was measuring it was related to all of my troubles.

One item in particular was constantly under-performing both visually and in terms of times, and another item was prone to apparently random misbehaviour.

I took Animat's techniques of defensive programming he described having to apply to his people in their wandering and applied them to the randomly misbehaving objects. I could find no way of improving the other item. Despite all my efforts, I was unable to pinpoint what was going wrong in the randomly misbehaving item until today. I can now see it is being caused by things that will not be in the scope of "steady state" performance that I have assumed Scripts run % refers to, but instead it is due to the goings and goings of certain avatars and I can code for that, to some extent, now I know what to look for.

Going back to Script Run %, whatever has happened over the last few week has picked up the figures, and the visual and timing behaviour that caused me to start looking at the in the first place is now longer an issue. I could just say "Oh well, I don't need to bother anymore, the world's fine and I don't need to worry". But there does need to be a measurement that we can take as guidance to know if we need to seek external help, much as getting the error "Unable to create object that has caused problems in this region" when coupled with a high  physics memory figure in the stats bar will lead to you having to contact live chat to get the region restarted to fix the problem.

The issue we face is that there is no "unable to create object ...." error message that goes along with the lowish scripts run figure, all we have are the visuals in the world telling us that things aren't right. I am trying to come up with a set of benchmark tests that could be run whose results would say either "yes, this looks like a problem" or "No, nothing found amiss here".

I just need to complete two more days of tests so I have a complete week to week results  ,including region restarts ( about 464 seconds downtime according to the timing test), and then I will be able to present the results and show what degree of correlation there is between the stats figures and measurable repeatable behaviour. I can tell you now that the figure isn't a continuum, or if it is, it's closest to a logarithmic type of scale, 100% down to 60% there's almost nothing changes, 60% to 40% there are odd little things, since the recent configuration changes I don't have any regions I can test in that are under 40%.

I would have liked to include a homestead in the range of regions I have been testing. I did try to visit your region but it's closed to outsiders and so I left it at that. When I have completed the testing I'll publish the finalised scripts and you can try running the tests for yourself.

Timescale for producing the results is, realistically, after the holidays. I won't be allowed to get away with spending all of them at the machines