Scripts Delay in most SL Lands!

[Tanaka Cloud]

Hello everyone, I come to inform linden lab that second life is having serious problems with scripts and this has been weeks
which is not tidy, I am a creator, I create swords, Huds magic for roleplay and etc ...

I have an land of roleplay with friends of mine, and I started to notice a huge delay between our attacks, when we tried
use some power made by us the script just doesn't work as it should, objects stay in the air for much longer than they should before they die, objects don't make a sound when they should, there is always a delay of at least 5 seconds to be executed at function of the script !, This disrupts all the dynamics that existed in the past, making people unable to enjoy their powers.

At first I thought it was a problem with the land, because when it was reset things "went back to normal" for half an hour, so I started going to other lands and testing different powers, weapons of different brands of second life, and in the vast majority of lands (including Bluesteel and Le Tigre which are Linden Lab's own lands) things still didn't work as they should! This in addition to disrupting the roleplay disrupts our business, as our customers and several others from other stores, think that the products are poorly made, that they are not working normally, when 1/2 months ago everything worked normally, please Linden Lab help fixing this script delay on the servers to get everything back to normal.

I made a short Video showing this Delay its just an example of a product where 2 months ago it worked perfectly, with the cuts appearing next to the animation.

Sorry for my rust english and Thank You for your Attention.

[SlayerRune]

yeah Also, it is very difficult to work with this long delay, and to force a collection of some scripts that is not working, I hope they will fix this soon.

[o0Agito0o]

Yeah, we've all been having this problem for 3 weeks now.  I thought it was just the sim i was on, but its almost everywhere. So i thought waiting a few days would fix it, but nope. 

Its almost impossible to duel someone now when you have 3 to 5 seconds delay.   The delay is so bad you can walk into your own spells. 

All the spell / fighting huds i bought are all useless now. 

Please fix this :(  

 

 

[Skye Alpha]

Unfortunately this is a problem and must be fixed. Something similar happened with my friends who are also creators and lag is almost unbearable to work with, they haved show to me the exactly same problem. 😓

[uy7767rtrtrtyty6]

This is so Bad :/. I'm too having problems with delay of scripts. I hope they fix this.

[Wulfie Reanimator]

I'm not experiencing this kind of delay. Any weapons I test rez instantly. It's definitely not a grid-wide issue.

Have you guys checked the sim(s) stats at all?

Edited June 25, 2020 by Wulfie Reanimator

[Profaitchikenz Haiku]

Ctrl-shift-1 and look for "% scripts run". There other things you can look at that people like Qie and Animats are more knowledgeable about than I, but % scripts run is the quickest measure.

It's been variable in regions for months, and has been discussed quite a bit at the Server user Group meetings.

If the region is affected badly by it the estate management team will need to try restarting the region several times until things get back to normal.

[CeltyStar]

yeah i notice the lag too

[AtenasVampirinos]

The lag it extremely annoying and it affects all kinds of scripts, no matter where you are. Please, we need this issue fixed because it is making everything slow.

[o0Agito0o]

Not all regions are affected by this, but alot are. 

[sandi Mexicola]

I was seeing about 20-25% "script per frame run success rate" (or whatever the heck u call it) before i logged out... yesterday i recall seeing 5% somewhere or another.  

[Trevorios Latzo]

2 hours ago, Wulfie Reanimator said:

I'm not experiencing this kind of delay. Any weapons I test rez instantly. It's definitely not a grid-wide issue.

Have you guys checked the sim(s) stats at all?

It is a problem that is occurring in some regions but not in all of them. You may not have noticed the problem because you are in a region where the problem is not occurring, or because you are using weapons with a physical bullet, so you have not noticed the problem happening since the problem does not affect the time of Rez, Rez it's instant. Because the current problem affects the initialization of the scripts contained in the objects, that is, there is a delay of up to 5 seconds for the scripts of the objects to start, however this only affects recent objects in the region, that is, an object that is recent Rez. This problem does not affect objects that have been loaded in the region for the longest time, as they have already had their scripts loaded. An example is that you fire a physical bullet that in the script was defined to perform an action after the collision, but what you actually do is the bullet bouncing several times before the script actually records the collision, because the script was not initialized .

[Lucia Nightfire]

https://jira.secondlife.com/browse/BUG-228939

[Wulfie Reanimator]

3 hours ago, Trevorios Latzo said:

It is a problem that is occurring in some regions but not in all of them. You may not have noticed the problem because you are in a region where the problem is not occurring, or because you are using weapons with a physical bullet, so you have not noticed the problem happening since the problem does not affect the time of Rez, Rez it's instant. Because the current problem affects the initialization of the scripts contained in the objects, that is, there is a delay of up to 5 seconds for the scripts of the objects to start, however this only affects recent objects in the region, that is, an object that is recent Rez. This problem does not affect objects that have been loaded in the region for the longest time, as they have already had their scripts loaded. An example is that you fire a physical bullet that in the script was defined to perform an action after the collision, but what you actually do is the bullet bouncing several times before the script actually records the collision, because the script was not initialized .

That makes more sense. In that case it's not really a new issue, just the same ongoing one (that's why I asked about sim stats). Scripts Run % has been in the gutter for the better part of a year by now. Who knows when/how LL is going to fix it.

[animats]

6 hours ago, Wulfie Reanimator said:

Who knows when/how LL is going to fix it.

They don't know how to fix it.

Scripts that do nothing but sit there are a huge drag on sim resources. Each one uses about 3us/frame, so 6000 scripts doing nothing uses up about 100% of script time. We brought this up over a year ago at Server User Group. I've demonstrated this in an empty sandbox with thousands of do-nothing scripts. Fixing this means going deep into the Mono scheduler and doing a sizable redesign. This is beyond LL's capabilities.

Alternatively, LL would have to support multiple CPUs working on scripts for a single sim. That, too, would be a sizable redesign. Worse, it would increase billed charges after the uplift to Amazon Web Services if sims needed more than one CPU.

We thus have to accept script overload as the new normal. I've designed my NPCs to function well at 50% scripts run and survive down to 25% scripts run. My bikes can run adequately down to about 40% scripts running.

Often, the overload is not on the sim showing problems. Another sim on the same server can reduce performance for the sims which share that server. Asking support to switch the sim to a different server often helps. Of course, that means some other sim probably starts to suffer.

A few hints for scripters:

• When scripting vehicles, accept the keyboard and mouse events and actuate the vehicle motors in the same script and event handler. Don't put llLinkMessage passing in the real-time path. I have a helicopter which uses message passing in its main control path, and it loses control in overloaded sims. But my bikes, which don't, behave well under overload.
• Expect some error in keyframe animation under overload. You may not end up exactly where you expected. The error increases with script overload and can, rarely, exceed 2 meters. My NPCs detect this and try to correct. (Currently, I have a bug where, close to a parcel edge, they go off-parcel and get stuck. Comes up about once a month. Another special case to handle.)
• Looped keyframe animation works well under overload. The viewer is doing most of the work. That's how my escalators work.
• Pathfinding is hopeless under script overload. It works OK in an empty sim, but has even lower priority than scripts, which results in pathfinding characters not getting enough steering corrections and going way out of position. Sorry, Virtual Kennel Club.
• llGetStaticPath works fine even under extreme overload. But check the status for PU_FAILURE_OTHER, which can appear under overload, rarely, and retry.
• llCastRay can fail under overload. Check the status and retry after a short delay.
• Plan ahead. The trick that makes my NPCs work under overload is that their path planning happens seconds before they move, and actual movement is executed using keyframe animation. As they're moving, the planning for the next part of the move, which involves lots of llCastRay calls and maze solving, is underway. This concurrency allows more time for the planning, without, usually, stopping motion to wait for the planner. It's a lot like how a superscalar CPU works. Doing this in LSL is excessively complex, though.
• Put in stall timers. Sometimes it's going to take so long that something will break.

Rezzing delays are a big problem for weapons. I don't do weapons, so I don't have a good answer there. Perhaps use a system where, rather than rezzing a bullet when the weapon is fired, you rez it invisibly when the weapon chambers a round, then tell it where to go with a message when the weapon is fired. Or a bow where the arrow rezzes when you start to draw the bow (with mouse-down), and fires on mouse-up.

With enough defensive programming, it's quite possible to make things work even as SL's performance deteriorates. Learn to embrace the pain.

Edited June 26, 2020 by animats
Clarify

[sandi Mexicola]

Awesome post!  Thank you!

[Wulfie Reanimator]

17 minutes ago, animats said:

They don't know how to fix it.

I tend to agree, and although I'm very aware of everything you mentioned, it's good to have it all listed here where some new people can be aware of it.

23 minutes ago, animats said:

With enough defensive programming, it's quite possible to make things work even as SL's performance deteriorates. Learn to embrace the pain.

But some of these "defensive techniques" would normally be considered "bad practice" and often invites a lot of "don't do that / do it another way" when you try asking related advice in public. (eg. Infinite loops to avoid event overhead.)

28 minutes ago, animats said:

Rezzing delays are a big problem for weapons. I don't do weapons, so I don't have a good answer there. Perhaps use a system where, rather than rezzing a bullet when the weapon is fired, you rez it invisibly when the weapon chambers a round, then tell it where to go with a message when the weapon is fired. Or a bow where the arrow rezzes when you start to draw the bow (with mouse-down), and fires on mouse-up.

The biggest recommendation I would make specifically for weapons and anything that needs to rez fast and/or has a short lifetime: Don't use Mono. Mono takes a long time to initialize even when there's no lag. Never use Mono for "bullets" (any projectile or effect), what ever you do.

The "rez early" method is pretty niche and only really works for slow, single-attack weapons. It would not work, for example, for a 600-1000+ RPM gun.

In other engines, one optimization you could do is "re-using" bullets, instead of having them be destroyed (llDie) on contact. There would exist some set number of bullets, and the oldest one would be placed at the front of the gun each time it fires. That's not really possible in SL, at least not with the LL Combat System (llSetDamage) and without lots of listen events per avatar.

[animats]

12 minutes ago, Wulfie Reanimator said:

But some of these "defensive techniques" would normally be considered "bad practice" and often invites a lot of "don't do that / do it another way" when you try asking related advice in public. (eg. Infinite loops to avoid event overhead.)

Oh, no, never use an infinite loop. What I'm saying is to make your llSetVehicleVectorParam calls to move a vehicle directly in the control event. Don't just make a llMessageLinked call to another script for ordinary steering and movement actions. I have a helicopter which does that, and it works unreliably under load. The basic move and steer stuff has to be on a fast path.

[Profaitchikenz Haiku]

47 minutes ago, Wulfie Reanimator said:

Don't use Mono. Mono takes a long time to initialize even when there's no lag

I've been going along this path for a few weeks now, and there's another benefit to dropping back to LSL for many scripts: they use far less memory (which may be why Mono takes ages to initialise anyway). A lot of the scripts I have in objects still work perfectly well with on 16K memory, so why grab more when there's no large amounts of data to be stored?

A door script to respond to touches fits into 16K with loads to spare and isn't going to grow, so why grab 64K for it? A building with 4 doors need not have four separate scripts (64K or 16K each), you can have one script in the root which has a list of child prims that are doors, together with their local rotations for open and closed. Just thinking here when you have a region with 50 buildings each having four scripted doors, what the demands on memory are and could be reduced to...

I still remember the good old days when an incoming avatar would cause the whole region to freeze for several seconds as all their mono scripts got loaded up. I can't help feeling we're sliding back towards that pit.

 

Edited June 26, 2020 by Profaitchikenz Haiku

[Profaitchikenz Haiku]

1 hour ago, animats said:

They don't know how to fix it.

I got that strong impression at the last couple of server user group meetings, further, I got the impression it was lower down the list than glitter such as Environment.

If we're lucky, the move to cloud machines will improve the situation due to better hardware. 

"I know what you're thinking, did he use five servers or six servers to host those regions? Well to tell you the truth, in all the excitement, I clean forgot. So the question you should be asking yourself about performance is, do you feel lucky?"

[Nova Convair]

1 hour ago, Profaitchikenz Haiku said:

I've been going along this path for a few weeks now, and there's another benefit to dropping back to LSL for many scripts: they use far less memory (which may be why Mono takes ages to initialise anyway). A lot of the scripts I have in objects still work perfectly well with on 16K memory, so why grab more when there's no large amounts of data to be stored?

Completely wrong.

Mono scripts do NOT use 64K while LSL scripts always use 16K. If you have many small scripts like bullets LSL will use up alot more memory that mono. A few 100 times more for a stream of bullets is a possibility.
Simple example: a bullet with a short 1K script - will use 1K in mono and 16K in lsl.
100 Bullets will not use 100K but still 1K in mono (plus variables space) (for equal scripts the code is in the SIM's memory ONCE) and 1600K in lsl.

But the question was not about memory but performance. Things changed for rezzed objects lately and I noticed that my rezzers take some time now to rez 50 objects. Not a problem here though.

Numbers needed. Need to make tests and see what comes out.

[Profaitchikenz Haiku]

1 hour ago, Nova Convair said:

Completely wrong.

I am male, I'm used to it @)

Thanks for the distinction about what is actually used, but I have to say it doesn't line up with what I see on the parcel memory tab,  where the scripts I know are using mono show memory used as multiples of 64K, and the scripts I know are LSL as multiples of 16K. Would it be possible that the viewer metrics are misleading in this area?

[KT Kingsley]

A mono script using the llSetMemoryLimit function will show up as using anything between 4K and 64K. I am given to understand that that function's setting and what a script actually uses have only a tenuous relationship. Nevertheless, it does make you look good on the memory displays.

I've also seen that the avatar script memory reported in the viewer is frequently just plain wrong: when I've been attaching and detaching scripted objects the value shown has got lower and lower, and it's taken a trip to a different region to bump it back up to what it should be. I don't know if that bug affects a region's reported script memory.

Edited June 26, 2020 by KT Kingsley

[Qie Niangao]

All the viewer can know is the memory LIMIT that's been set for a script. It has no idea how much memory is actually used by the script. What's more, there's no way for a script to know how much memory is actually used by another script -- again, it can only know the memory limit set for that [other] script [though of course it can know more about its own instantaneous memory usage].

I never thought about the loading-delay disadvantage of Mono scripts for rezzing -- I've only ever considered it for region crossings -- and it certainly makes sense. One thing I wonder, though: Mono has some mechanism for sharing the program memory for all identical compilation units loaded in the sim. This does nothing to help with initializing state variables (which, in non-Mono, just blit into memory along with the program), but I've no idea how expensive it is for Mono to load the next instance of a bullet script, compared to that initial one. It's not going to be faster than non-Mono but, I mean, how many state variables does a bullet script have?

Edited June 26, 2020 by Qie Niangao

[Love Zhaoying]

3 hours ago, Nova Convair said:

Things changed for rezzed objects lately and I noticed that my rezzers take some time now to rez 50 objects. Not a problem here though.

Perhaps they throttled rezzing again (vs. previous throttle)?