Open Source Script Benchmark

[Coffee Pancake]

It would be useful to have an open source benchmark for scripts in SL. Something that could be rezzed / worn, run through a few tests and then give  results that can be meaningfully compared over time and to other regions.

It must be able to give a consistent result over sequential runs in the same location assuming nothing else changes. Is this even possible?

What should be tested and how?

What results should be generated?

What should be done with the data?

Thoughts, ideas, code to test ?

[Qie Niangao]

I bet people who design benchmarks for a living have developed a whole theory of benchmarking. (I remember thinking the same about standardization, the first time I was on a standards committee.) In lieu of knowledge, some random thoughts:

How the benchmark is exercised determines what can be measured. For some measurement objectives there's virtue in collecting a lot of samples over many locations -- something we might want lots of people to wear many places much of the time, to get a kind of broad-gauge measure of script performance over time across the grid. That kind of measurement, though, must be very low-impact, compared to a thoroughly invasive benchmark that effectively takes over a sim and works the script engine as hard as it can. These different approaches address different measurement objectives - they measure different things for different purposes.

Benchmark use cases:

1. LL server developers want to make sure cumulative changes haven't degraded script performance, or that some change has indeed improved performance as intended. (They obviously have access to better internal test points than we can access. Nonetheless... I wonder what benchmarks they use, and what they used for example when they tested Mono pre-release.)
2. Scripters want to know of changes in the execution environment affecting performance. Maybe link_message dispatch is delayed, or the scheduler is generally slower at context-switching, or... (maybe this level of detail isn't productive to consider now?) These changes may come from LL server development or from user practices (e.g., more mesh avatar and Bento head HUDs roaming the grid, more use of KVP store with grid-scope Experiences, etc.)
3. Scripters want to test the effect their own scripts have on the general execution environment: Are they accidentally stretching resource limits? (probably griefers have the best knowledge of which resources can be exhausted, and how... so a good benchmark might be a tool for developing more effective griefing??)
4. An individual region user wants to determine if scripts are running faster or slower than before on that specific region.
5. etc?

Here's a thing I haven't tried, but might measure something useful: Two scripts in the same object ping each other with link messages and measure the interval between sending and receipt. (Is this any different at all from a single script pinging itself with the same messages? I'm actually not sure about that.)

The measurement would mean something different under different sim loading conditions. Worst case, if the sim is in time dilation, script times get wildly unpredictable: sometimes it seems as if multiple frames-worth of script time get allocated to the extended frame... or something. Anyway, I'm not sure what it means to measure script performance under time dilation. But even just within a normal frame, it's one thing for scripts to be slower because they don't get to run each frame (no Spare Time, less than 100% Scripts Run) and something else if they're slower when they're the only scripts on the sim.

Both those are of interest, separately, but we'd need to tease them apart. Either we need to run the benchmarks only under controlled, contrived conditions -- dedicated sim, artificially loaded or unloaded -- or we need better ways for a script to know the load conditions on its sim in the wild. We have llGetRegionFPS and llGetRegionTimeDilation, but what about Scripts Run percentage?