Scripts Delay in most SL Lands!

[Wulfie Reanimator]

12 hours ago, animats said:

Oh, no, never use an infinite loop. What I'm saying is to make your llSetVehicleVectorParam calls to move a vehicle directly in the control event. Don't just make a llMessageLinked call to another script for ordinary steering and movement actions. I have a helicopter which does that, and it works unreliably under load. The basic move and steer stuff has to be on a fast path.

Yeah... Case in point.

When you have a script that only does some initialization in state_entry before calling llSetTimerEvent and never does anything outside of that timer, you get magnitudes greater performance with a single loop+sleep instead of generating an entirely new event each interval. There are costs you have to pay for events. It's the reason your helicopter can't cope.

Same with guns. Often, the gun would send linked messages to rezzer-scripts while firing -- one message per rez, all rezzers are usually in the same prim so it creates n events -- which greatly limits the effective fire rate and you end up with event queues so long you won't stop firing even if you let go of the trigger.

A rezzer-script which has an infinite loop, rezzing per iteration, can be controlled exactly by simply turning it on/off with llSetScriptState. No messages, no events, no resets, and it stops firing exactly when it needs to.

When I say "infinite loop," I don't mean a loop that just goes as fast as it can forever. You treat it like a timer with a controlled delay. It's a tool no different from your method of reducing events.

10 hours ago, Nova Convair said:

100 Bullets will not use 100K but still 1K in mono (plus variables space) (for equal scripts the code is in the SIM's memory ONCE) and 1600K in lsl.

Only the bytecode is shared for Mono scripts. (Also, non-Mono is LSO, not LSL...)

Bytecode is all the constant stuff, variable names, operators, literal values, function names, etc... What data the variables actually hold is entirely independent. One instance could use 1K, while another is doing something that increases it memory usage.

8 hours ago, KT Kingsley said:

I've also seen that the avatar script memory reported in the viewer is frequently just plain wrong: when I've been attaching and detaching scripted objects the value shown has got lower and lower, and it's taken a trip to a different region to bump it back up to what it should be. I don't know if that bug affects a region's reported script memory.

Are you sure you're not thinking about CPU time? It's an average of the past up to 30 mins, or since the script reset/rezzed(/teleported), whichever is smaller.

6 hours ago, Qie Niangao said:

I never thought about the loading-delay disadvantage of Mono scripts for rezzing -- I've only ever considered it for region crossings -- and it certainly makes sense. One thing I wonder, though: Mono has some mechanism for sharing the program memory for all identical compilation units loaded in the sim. This does nothing to help with initializing state variables (which, in non-Mono, just blit into memory along with the program), but I've no idea how expensive it is for Mono to load the next instance of a bullet script, compared to that initial one. It's not going to be faster than non-Mono but, I mean, how many state variables does a bullet script have?

It's not that Mono scripted objects rez slower, just that the scripts don't do anything for a while.

The biggest benefit of Mono is also its biggest problem -- dynamic memory. When the script is rezzed (or teleports), the sim needs to calculate how much memory of its maximum limit it's actually using. (This is why an avatar entering a sim is basically a performance nuke.) Even the simplest script will take its time, at least with the default limit of 64K. Lowering that limit guarantees the script won't use more than that, or it will crash. I would imagine a lower limit would also make the script initialize faster, but I haven't checked.

Edited June 26, 2020 by Wulfie Reanimator

[KT Kingsley]

14 minutes ago, Wulfie Reanimator said:

Are you sure you're not thinking about CPU time? It's an average of the past up to 30 mins, or since the script reset/rezzed(/teleported), whichever is smaller.

Memory.

[Profaitchikenz Haiku]

1 hour ago, Qie Niangao said:

All the viewer can know is the memory LIMIT that's been set for a script. It has no idea how much memory is actually used by the script. What's more, there's no way for a script to know how much memory is actually used by another script

So only an estate or region manager's console (top scripts) is going to give a definitive answer?

[Qie Niangao]

5 hours ago, Profaitchikenz Haiku said:

So only an estate or region manager's console (top scripts) is going to give a definitive answer?

Kinda doubt data about dynamic memory allocation within Mono gets to Top Scripts either. There'd be significant overhead in collecting those statistics. Not impossible, but seems unlikely (but I haven't been in a position to look at region manager stuff for many years).

[Doctor Haiku]

This issue is widespread, but not every sim is being affected. Whether a sim is affected or not seems to be random, no matter if it's a homestead or a full sim.

I've also seen that some functions seem to have more stable performance than others.

Example: If I were to make a script that uses an llSleep delay before the object deletes itself, the script more than likely will stall before activating. Compare that with one that uses an llSetTimerEvent for the delay, and it seems to be more stable. Likewise this may or may not happen for llTriggerSound vs llPlaySound.

Has anyone else noticed similar events like these?

[CrowleyCorp]

HERE IS THE JIRA REPORT FILED ON THIS TOPIC LAST WEEK

Last week my JIRA report of a major bug affecting many items in Second Life including my Elite Golf System and Battle Karts was 'accepted' by LL which means the problem was reproduced by them and is 'not expected behavior'.  It has also been 'triaged' which means they have put it on their to-do list of things to fix.  As for when well that's anyones guess.  Go to  https://jira.secondlife.com/browse/BUG-228939 and click 'more' then 'watch' for updates.

The issue is not affected by size of the script, region performance or object owner.  It comes and goes affecting different regions on different days and is completely an LL issue with no work-around.

Edited June 26, 2020 by CrowleyCorp

[sandi Mexicola]

Thank u

[Hooten Haller]

I have an object that is rezzed several times at the start of a gig for dancers to sit on. While in the rezzer inventory the seat object is in a “standby” state. That state has an on_rez event handler whose first line is llSetText. There is seconds of lag between the seat appearing and the hover text changing. This effect has been coming and going over the past couple weeks. Currently it’s back. Fortunately rezzing is a rare occurrence for the overall product.

[ItHadToComeToThis]

I have been noticing this for months and it’s getting worse. It used to be a 2 second delay and now in all sims for me it’s around 4 seconds before a script executes

[Love Zhaoying]

5 minutes ago, ItHadToComeToThis said:

I have been noticing this for months and it’s getting worse. It used to be a 2 second delay and now in all sims for me it’s around 4 seconds before a script executes

Maybe something script infrastructure-related has moved to the cloud, and LL did not think we would notice..

[Jasdac Stockholm]

Since I can't post on the JIRA, here's some info I've gathered myself and from my userbase:

• The issue seems to be that on_rez takes forever to trigger. Items spawn in a timely fashion.
• The issue affects all sims, but only starts appearing after the sim has been running for a while. I'm not sure if there's a certain event that triggers it, or if it's some sort of buildup.
• This means if you restart the sim, it'll run fine for a while.

This is a simple script that rezzes an object that triggers some particles when it rezzes, and then dies. The object has a light emitter on it, so you can see it spawns in a timely fashion, the particles however do not.

https://i.gyazo.com/75c6970aed8dc069831111a851747549.mp4

After a sim restart, this is the same script:

https://i.gyazo.com/f4b84a9f396c6e2e3e61909ec281a11f.mp4

It's been present for a few weeks now. Hopefully that helps someone.

[Wulfie Reanimator]

4 minutes ago, Jasdac Stockholm said:

This is a simple script

Does Mono on/off show any noticable difference?

[Jasdac Stockholm]

25 minutes ago, Wulfie Reanimator said:

Does Mono on/off show any noticable difference?

Haven't tried, I'll do it once the region bogs down again

[ItHadToComeToThis]

Reviving instead of creating a whole new thread. This is getting f-ing irritating. It’s a 4 second delay for state entry to kick in after a save or after a script reset. About the same for the on_rez and attach events. Iv noticed message delays on empty sims of up to 4-6 seconds before the receiver reacts towards it testing with a simple send and receive script and over multiple sims. If this is what “SL in the cloud” is going to be like then woo. 

Edited July 20, 2020 by ItHadToComeToThis

[Qie Niangao]

1 hour ago, ItHadToComeToThis said:

If this is what “SL in the cloud” is going to be like then woo. 

I don't think this is related to the cloud migration. Related problems were reported months ago, with various delays and some events not firing at all. There's a thread from March, for example, with a post citing a change from November 2019. They seem to have been testing a fix a week or so later in March but maybe that was only a specific problem with object_rez.

There certainly are SL services that are already in the cloud, but the vast majority of sims (and I think all sims on the main "Agni" grid) are still on datacenter hosts.* It's not obvious what part of the current delays could be happening outside the sim itself; I'd be surprised if it's any at all.

One thing that could be related to cloud migration: the developer attention available to work on fixing it. Pretty sure they're aware the problem is severe, but they need to complete the cloud thing, too, and it's easy to squander resources pulling them back and forth between priorities.

_____________
(On the other hand, Oz keeps insisting that users must remain uninformed as the grid moves to the cloud, lest we superstitiously attribute problems to the migration -- which means nobody outside the Lab is supposed to know for sure how much of Agni is on the cloud until they pop the champagne corks.)

[Love Zhaoying]

54 minutes ago, Qie Niangao said:

_____________
(On the other hand, Oz keeps insisting that users must remain uninformed as the grid moves to the cloud, lest we superstitiously attribute problems to the migration -- which means nobody outside the Lab is supposed to know for sure how much of Agni is on the cloud until they pop the champagne corks.)

I detect the slightest hint of cynicism!

[Qie Niangao]

3 minutes ago, Love Zhaoying said:

I detect the slightest hint of cynicism!

Not about the migration itself, but I can't really hide my skepticism about blowing extra entropy into residents' understanding of what's running on the grid at any given time. I get why they want people not to assume every glitch is Uplift-related, but to me the secrecy seems counterproductive to that goal.

[Love Zhaoying]

2 minutes ago, Qie Niangao said:

Not about the migration itself, but I can't really hide my skepticism about blowing extra entropy into residents' understanding of what's running on the grid at any given time. I get why they want people not to assume every glitch is Uplift-related, but to me the secrecy seems counterproductive to that goal.

In a perfect world, we would not notice the "uplift" because LL would have tested it so thoroughly and completely, that there would be zero unknowns. #FightMe

[Profaitchikenz Haiku]

3 hours ago, Qie Niangao said:

lest we superstitiously attribute problems to the migration -

Which many people seem to be doing anyway

This problem with unpredictability suffers from the fact that there is a workaround, restart the region repeatedly until the problem eases. While there's that method, I can see why the Lab might decide to let it ride and hope that the cloud migration cures it anyway due to better hardware.

[Jasdac Stockholm]

I know it's an old thread, but since the issue still persists and I've found some more info:

The bug doesn't only affect on_rez/attach. It seems to prevent the WHOLE SCRIPT from running any code for a few seconds upon rez. I had the same issue with collision events not raising at all while the sim is rez-bugged. I'm gonna try some other events like timers etc to see if those are affected too, which I suspect would be the case. A forced sim restart still fixes it. I think the automatic Tuesday sim restarts might also be related, as I usually have issues after those. I'll try to verify this on Tuesday.

And no, the AWS migration didn't resolve anything.

Edited December 27, 2020 by Jasdac Stockholm

[Jasdac Stockholm]

I just witnessed it happening. I rezzed a cube with an ownersay on rez before rezzing a large build and it output it instantly. After rezzing the large build it also worked instantly. But a few moments later I rezzed it again and it now took 2-3 sec before outputting the ownersay. I also added a timer to it, and the timer doesn't trigger either until a few sec after rezzing it. Removing the large build had no effect, the test script still took multiple seconds to output anything. It wasn't a gradual slowdown. It was instant the first time, then took multiple seconds.

Script:

default
{
    on_rez( integer total ){
        llOwnerSay("IT BEGINS");
    }
    
    state_entry(){
        llSetTimerEvent(.5);
    }
    
    timer(){
        llOwnerSay("Tick");
    }

}

 

[Fourmilab]

I have been investigating rez to script execution delays ever since they started happening in early 2020.  Here is what I have observed.  When a script rezzes an object out of its parent prim's inventory, there is a delay between the time of the llRezObject() call and when the script in the rezzed object begins to run.  Before early 2020, this delay was almost always very short—if you rezzed a projectile from a launcher, its script would almost always get control almost instantaneously.  In early 2020, delays began to appear, and their occurrence was distinctly odd.

I built a test object which rezzes an object and performs a handshake in which the rezzed object sends the time its script got control to the parent, which prints the delay time.  What I found is that when you run this test in a given region at a given time, the region will always be "fast" or "slow".  By "fast", I mean less than 100 milliseconds (usually around 70 milliseconds prior to the uplift, and more like 30 milliseconds after the migration to AWS).  A "slow" region will always have a rez-to-script-execution time of around two seconds, before or after uplift.  These figures are utterly bimodal—you will hardly see a number in between them or much greater than two seconds.

If you test repeatedly over a number of hours, a "fast" region will remain "fast", and a "slow" region will remain "slow", with the exceptions that a "fast" region may become slow and stay that way, and a slow region may become fast after it is restarted (but then may become slow again a few days later).

Here is a test I ran across 25 regions on Christmas Eve, showing the delay, standard deviation, and uptime of the regions, using my "Gridmark" automated benchmark which will be available soon for free/full perm so anybody can do their own experiments.

 

                         2020-12-24 13:59 UTC

  Region                           Delay    n   Std. dev  Uptime  Type
  ----------------------------    ------   ---  --------  ------  ----
  Fourmilab                       0.0296    5    0.0102    12.7    ER
  Sandbox Pristina                0.0263    5    0.0011    12.6    ER
  Sandbox Exemplar                2.0227    5    0.0006    13.8    ER   Slow
  Sandbox Verenda                 2.0228    5    0.0006    13.8    ER   Slow
  Sandbox Formonsa                0.0269    5    0.0010    12.7    ER
  Sandbox Amoena                  0.0283    5    0.0010    13.8    ER
  Sandbox Artifex                 2.0267    5    0.0104    12.6    ER   Slow
  Sandbox Mirificatio             0.0262    5    0.0003     4.2    ER
  London City Brittany            0.0358    5    0.0098     2.2    EH
  Debug1                          2.0222    5    0.0004    12.6    ER   Slow
  Mauve                           2.0325    5    0.0052    12.6    MR   Slow
  Devolin Mal                     2.0310    5    0.0056    13.8    MR   Slow
  Limia                           0.0299    5    0.0096     9.1    EH
  Arowana                         0.0250    5    0.0009     9.1    EH
  Orville                         0.0262    5    0.0009    12.6    ER
  Woodbine                        0.0440    5    0.0006     4.5    MH
  Lapara                          0.0366    5    0.0071    13.8    MR
  Caledon Oxbridge                0.0609    5    0.0090     2.3    EH
  Babbage Palisade                0.0374    5    0.0180    12.7    ER
  Maryport                        2.0224    5    0.0104    12.7    MR   Slow
  Combat (sandbox) Rausch         0.0266    5    0.0006     2.1    MR
  Langdale                        2.0309    5    0.0095    12.7    MR   Slow
  Sandbox Bricker                 2.0209    5    0.0013    12.7    ER   Slow
  Vallone                         2.0333    5    0.0130    12.6    MR   Slow

  Regions: 25, 15 fast, 10 slow.
  Mean uptime (days): Fast regions 8.9, Slow regions 13.0
  Total test time 66.1 minutes, 27 teleports.

Some of the regions which tested "slow" (for example, Mauve) in this test have since been restarted and the last time I tried, tested "fast".  Because my test requires the ability to rez an object, I can only run it in sandboxes and rez zones, hence the odd selection of regions.

To my ancient programmer's eyes, this looks like a timeout situation.  There's something that's supposed to happen in starting the scripts of a rezzed object which sometimes fails but gets retried successfully by a timeout set to 2 seconds.  Why this situation appears to manifest itself more frequently the longer the simulation has run since its last restart is a mystery which remains opaque to observers outside the simulation code.

 

[Lucia Nightfire]

The developer responsible for the introduction of that behavior back in November 2019 said they'd like to work on a fix someday.

If I see/hear they're working on more features nobody asked for, I'll remind them of this problem and how it should have been given higher priority instead.

[Jasdac Stockholm]

I've noticed a different behavior on this recently. After a sim restart items rezzed through a script (llRezAtRoot etc) start instantly. But when rezzed from your avatar inventory, it takes multiple seconds to begin.

[sandi Mexicola]

1 hour ago, Jasdac Stockholm said:

I've noticed a different behavior on this recently. After a sim restart items rezzed through a script (llRezAtRoot etc) start instantly. But when rezzed from your avatar inventory, it takes multiple seconds to begin.

When you say "rezzed through a script", do you mean a script that is in a rezzed object?

Does it matter if a rezzed object is worn or not?