Rumors about improved script performance.

[Quistess Alpha]

So, I recently heard a rumor that several weeks ago the lindens did some magic that drastically improved script performance/time-usage, so I took a very small and unscientific look at the reported stats for 4 different regions:

1. My microparcel on Zindra:
   • Server: 2021-11-11.565737
   • Scripts: 9202
   • Spare time: 7ms
2. A hangout place I frequent:
   • Server: 2021-11-11.565737
   • Scripts: 8610
   • Spare time: 7.5 ms
3. My house/private sandbox on mainland:
   • Server: 2021-10-25-565008
   • Scripts: 3878
   • Spare time: 2.5 ms
4. My friend's club:
   • Server: 2021-10-25-565008
   • Scripts: 9289
   • Spare time: 0
   • Script run: 50%

Are these performance gains (decreased time-usage per script) real, an artifact of some change in the way stats are reported, or just sampling bias?

Edited December 6, 2021 by Quistess Alpha
(fixed typo in title)

[Profaitchikenz Haiku]

I have tried comparing the viewer stats for several parcels and am as puzzled as you. The %scripts run seems to be the simplest comparison method of all but by itself doesn't really tell me much. The only real comparison I can think of is to have a scripted object that performs a regular travel between several points, at a set speed, such that the journey time should be a specific figure. I can confirm that in one parcel where %scripts run improved from 30% to 75% the journey time is now close to optimum, but I am not able to put that same item on different parcels.

Elsewhere Month Linden has said that % scripts run is of questionable use for comparisons but I don't know if he meant comparing parcel to parcel or on parcel before and after a change in the figure.

My own study suggests it is a weather vane.

The lack of any spare time at your friend's club is slightly ominous.

[Aishagain]

I really don't know what to think about this topic.

Despite being informed by Monty Linden that my region is definitely running this new "configuration", I have seen no improvement  in either spare time or %script-run.

I therefore have seen no benefit whatsoever on a homestead region running about 1500 scripts showing an apparent 15 plus seconds of spare time.  I have to ask why not, since others are clearly seeing improvements.  Quite the opposite in fact.  Script-run has in fact reduced from relatively relatively stable 50+% to an unstable 35-45%.

It makes me wonder just how much control in fact does Linden Lab have over its creation?

Edited December 6, 2021 by Aishagain

[Profaitchikenz Haiku]

I think what Monty was trying to say is that although you might have a lowish scripts run % figure, you might not actually be suffering any performance as a result of it.

As I said above, using a scripted object that should perform to a known schedule is probably the only way to really evaluate the performance of scripts in your region. If the object behaves acceptably, doesn't jitter when moving, and keeps reasonable time, then the value you're seeing for your % scripts run indicates an adequate level. If the object takes much longer to complete the movement and is noticeably jitter then your scripts are not performing adequately.

I am coming to the conclusion that many of the metrics we have in the Ctrl-Shift-1 floater  are quite old diagnostics dating from when SL was in it's infancy and haven't been updated for some time, but are left there because trying to remove them from the code could 

a) break something because parts of the code involved may have other purposes

b) cause all the TPVs a real headache in having to revise all their code in line with the Lab's sweeping changes

So they're there, and because they don't actually have  relevance today no attempt has been made to document them.

[Coffee Pancake]

Horizons parcel has been in the doldrums for weeks, 45% run was typical .. after the update, the eps, frame time breakdown and total scripts are all about the same but we're up to 99% run.

Am I wrong suspecting that the % run number is just being calculated differently now ..

43 minutes ago, Profaitchikenz Haiku said:

I am coming to the conclusion that many of the metrics we have in the Ctrl-Shift-1 floater  are quite old diagnostics dating from when SL was in it's infancy and haven't been updated for some time

Those numbers are junk and always have been, I did suggest way back that we should have an actual region benchmark package that we can run and gauge actual performance, but everyone was too busy staring the rubbish numbers to put any serious thought into what that might look like.

[Aishagain]

Rubbish metrics or not, why is it, then, that while most other folk are seeing improvements in script performance and "Spare Time" values, I see nothing?

Still further, I am expected to be grateful that I am not seeing a performance hit? At this point words fail me.

[Monty Linden]

1 hour ago, Profaitchikenz Haiku said:

I am coming to the conclusion that many of the metrics we have in the Ctrl-Shift-1 floater  are quite old diagnostics dating from when SL was in it's infancy and haven't been updated for some time, but are left there because trying to remove them from the code could 

a) break something because parts of the code involved may have other purposes

b) cause all the TPVs a real headache in having to revise all their code in line with the Lab's sweeping changes

So they're there, and because they don't actually have  relevance today no attempt has been made to document them.

or

c) Lindens are lazy scum and will be first against the wall when the revolution comes.

They are poorly documented and incomplete in scope.  Some are also just wrong.  The Agent Update item was recently fixed after having been wrong since... 2007?  Useful metrics supporting a real development environment would be a nice goal.

That said, I have been through them once recently and they're in better shape.  But documentation updates are needed.  I also have a diagram breaking down the main simulation loop.  A version of that for public consumption may be interesting...

[Monty Linden]

10 minutes ago, Coffee Pancake said:

Those numbers are junk and always have been, I did suggest way back that we should have an actual region benchmark package that we can run and gauge actual performance, but everyone was too busy staring the rubbish numbers to put any serious thought into what that might look like.

Benchmarking regions are absolutely useful and we don't do enough of that.  We do have *some* and these should be available on the Beta grid.  For example, TextureTest2 and MeshTest2 regions are carefully constructed for asset download testing.  (They also serve as a demonstration of how the interest list and asset priority schemes function.)  I invite comments on those....

[Coffee Pancake]

2 minutes ago, Monty Linden said:

Benchmarking regions are absolutely useful and we don't do enough of that.  We do have *some* and these should be available on the Beta grid.  For example, TextureTest2 and MeshTest2 regions are carefully constructed for asset download testing.  (They also serve as a demonstration of how the interest list and asset priority schemes function.)  I invite comments on those....

I was specifically thinking we should have a standard benchmark package, preferably something a user can rez at home and get some numbers

[Monty Linden]

3 minutes ago, Coffee Pancake said:

I was specifically thinking we should have a standard benchmark package, preferably something a user can rez at home and get some numbers

Ah, a bucket of scripts, providing a kind of performance dashboard.  Also good.  And you don't need us for that. 

[Coffee Pancake]

Just now, Monty Linden said:

Ah, a bucket of scripts, providing a kind of performance dashboard.  Also good.  And you don't need us for that. 

Apparently we do .. because everyone here is happy to just look at the rubbish numbers rather than actually do literally anything that might be more illuminating.

We're also not privy to the secret inner workings of the SL service, what are the pinch points we should be testing and what, specifically, would a box of scripts be testing.

Besides, as any such box of scripts would be used to bemoan the SL service, might be wise to be involved in the development of those. At least then any generated numbers would be meaningful when bright up in discussions.

[Profaitchikenz Haiku]

16 minutes ago, Monty Linden said:

They are poorly documented and incomplete in scope.  Some are also just wrong.  The Agent Update item was recently fixed after having been wrong since... 2007?  Useful metrics supporting a real development environment would be a nice goal.

That said, I have been through them once recently and they're in better shape.  But documentation updates are needed.  I also have a diagram breaking down the main simulation loop.  A version of that for public consumption may be interesting...

As n old-school programmer* I have to say give absolute priority to getting the code to work properly, documentation can be given to less able people during quiet spells.

* Those who can do, do,

Those who can't do but know, teach.

Those who can't do and don't know, go into QA

[Profaitchikenz Haiku]

5 minutes ago, Coffee Pancake said:

Apparently we do .. because everyone here is happy to just look at the rubbish numbers rather than actually do literally anything that might be more illuminating.

I ought to bring to everybody's attention that Monty said "some are just wrong".

Some != all.

[Madelaine McMasters]

6 minutes ago, Profaitchikenz Haiku said:

As n old-school programmer* I have to say give absolute priority to getting the code to work properly, documentation can be given to less able people during quiet spells.

* Those who can do, do,

Those who can't do but know, teach.

Those who can't do and don't know, go into QA

I started my career in QA, where I learned that people who think they "can do" often... can't.

Edited December 6, 2021 by Madelaine McMasters

[Monty Linden]

4 minutes ago, Coffee Pancake said:

We're also not privy to the secret inner workings of the SL service, what are the pinch points we should be testing and what, specifically, would a box of scripts be testing.

Besides, as any such box of scripts would be used to bemoan the SL service, might be wise to be involved in the development of those. At least then any generated numbers would be meaningful when bright up in discussions.

Engineering blindness comes into this.  If we do it, we'll measure technical structures that we know are significant, it is true.  But these don't necessarily align with the quality of user experiences.  The script developers who live in that space may produce more relevant tests than we would.  Lack of internals knowledge isn't a barrier and Lindens are here to consult as needed.  Open a new script today!

[Coffee Pancake]

2 minutes ago, Profaitchikenz Haiku said:

I ought to bring to everybody's attention that Monty said "some are just wrong".

Some != all.

I think at this point a standard, open source, easy to use box of tricks to run some tests, generate some numbers would be more broadly useful. It would also give people an immediate follow up action to the numbers in the stats floater looking "off" and would be somewhat immune to the stat floater gradually going off base as the server product evolves.

[Profaitchikenz Haiku]

3 minutes ago, Monty Linden said:

Open a new script today!

Those of us who noticed something strange a few months back already had scripts that used keyframed motion or prim puppeteering to move transport objects to and fro, and observed these movements were becoming jerky and erratic, and the times for the journeys were longer than had originally been programmed for, These changes coincided with a dropping in the contentious figure. The only problem, as I have discussed with Coffee previously, is that helivators, railways, busses, wandering pets are too complicated to form the basis for a jira. I on't think we need new scripts, just simpler test cases derived from them.

[Profaitchikenz Haiku]

1 minute ago, Coffee Pancake said:

a standard, open source, easy to use box of tricks to run some tests, generate some numbers would be more broadly useful.

Yes, I agree, but from my own experiences I found that running scripts on their own to just count messages sent from other inworld objects showed no signs of any issues. It is when events requiring server and client communication are involved that the problems do start to become apparent. Things like Hud comunications don't seem to be affected, scripted attachments all seem to function normally, it is objects moving inworld that seem most affected by whatever is at the heart of this issue.

[Ardy Lay]

I found a scripted object that was clobbering performance of other scripted objects on one of the regions I own land in.  It was a "sim performance meter".  It's gone now.  The owner of the meter left SL claiming the performance sucked too much to bother with it.  I bought his parcel within minutes of his setting it for sale then ensured everything on it was returned.  The region is working great now that he and his box of whine is gone.  A friend, Moe, told me he knows of the "sim performance meter" I encountered.  He told me it gets "wall clock time", runs a fixed number of iterations in a tight loop, then gets "wall clock time" again, does some math then lies like a used pencil salesman.  Moe is a bit of a performance meter collector, so I suspect he knows from experience which ones eat that which they claim to measure.

[Kathrine Jansma]

1 hour ago, Monty Linden said:

Benchmarking regions are absolutely useful and we don't do enough of that.  We do have *some* and these should be available on the Beta grid.  For example, TextureTest2 and MeshTest2 regions are carefully constructed for asset download testing.  (They also serve as a demonstration of how the interest list and asset priority schemes function.)  I invite comments on those....

TextureTest2 seems unaccessible. TextureTest works, MeshTest2 also, but could not TP to the TextureTest2 region on the Beta grid.

[Profaitchikenz Haiku]

53 minutes ago, Madelaine McMasters said:

I started my career in QA,

Sorry if I stubbed your toes, but I'm under doctor's orders to ruffle somebody's feathers once a day to keep my blood-pressure down

[Monty Linden]

24 minutes ago, Kathrine Jansma said:

TextureTest2 seems unaccessible. TextureTest works, MeshTest2 also, but could not TP to the TextureTest2 region on the Beta grid.

Permissions fixed.  Should be available now.

[Kathrine Jansma]

The hard part with benchmarks is usually to come up with useful metrics AND make the measurment halfway reliable and repeatable.

With around 500 functions, which one do you care about?

Sometimes performance is a little suprising.

E.g. i tried to script some system to detect objects based on proximity and wind etc. (basically a primitive smell simulation with a bit of monte carlo sampling thrown in).

I assumed it would be much faster to detect statically named objects in range via llSensor() as thats basically a trivial geometry query with a static condition, which should be super fast if backed by any kind of database with spatial index (e.g. https://en.wikipedia.org/wiki/SpatiaLite ). Turns out llSensor(fixed_name, ...) is so high on latency that it is around 10x more efficient (for latency) to drop a script with a listener in every single target instead and send out static pings and listen for the echos. Which is sad, script spamming for no good reason.

Other parts of the code create random numbers for a Monte Carlo simulation and get starved on the llFrand() performance (and would love a CSPRNG like /dev/urandom) so would see improvements there.

Another part does vector manipulation and distance calculations for many objects, which would really benefit from SIMD optimizations. So if the server JITed those loops to use AVX it would be awesome. But totally different performance metric again.

I'm surprised the script works at all, but it is super lag/script performance sensitive.

[Kathrine Jansma]

2 hours ago, Monty Linden said:

Benchmarking regions are absolutely useful and we don't do enough of that.  We do have *some* and these should be available on the Beta grid.  For example, TextureTest2 and MeshTest2 regions are carefully constructed for asset download testing.  (They also serve as a demonstration of how the interest list and asset priority schemes function.)  I invite comments on those....

TextureTest2 looks useful for tuning the pipeline. But guess i need a bigger network pipe (as it can saturate my 100MBit/s link but not the CPU when running with 16 texture decode threads in CoolVL Viewer and forcing full size texture loads).

[Madelaine McMasters]

12 minutes ago, Monty Linden said:

Engineering blindness comes into this.

Once I moved into engineering, I encouraged my team to speak to the "idiots" who always outsmarted us by exposing flaws in our hardware and software. The best idiot we knew was our engineering manager, George, who had a remarkable knack for toppling anything he touched. He was possibly the most valuable member of the department, simply because he couldn't be bothered to understand how we thought our stuff should be used. That made him the quintessential proxy for our customers.

22 minutes ago, Profaitchikenz Haiku said:

Sorry if I stubbed your toes, but I'm under doctor's orders to ruffle somebody's feathers once a day to keep my blood-pressure down

I don't mind. I eventually got tired of fixing other people's problems and resolved to create my own.