Recent heavy script lag?

[JoachimLichttrager]

Hey, I'm not precisely sure where to put this as it's my first time posting here, but I help run a combat sim in the Second Life Military Community and lately we've been having an issue that's causing me to tear my hair out in frustration.

So, to give a little background, most sims rely on passive interaction when it comes to scripts and whatnot, with RP sims, adult activity sims and such usually being functional even under comparatively heavy script lag (anywhere between 50-60% script run). Not so with combat sims, where that amount of lag causes a significantly reduced experience.

Recently we've run into a problem where script run in statistics is consistently very high (around 94% on average over one minute), yet we are experiencing significant delays or simple script failures (without anything throwing an error) that we haven't had issues with previously. It's causing a LOT of scripts and devices that we rely on to not function properly. Vehicles aren't rezzing in the proper manner, contact-fuse grenades or instant trigger explosives are bouncing and hanging upon striking a solid surface or are significantly delayed in the case of manually triggered explosives rather than detonating immediately as they should, bullets are going a significant distance through walls or bouncing off solid surfaces instead of dying nearly immediately.

In a competitive scenario, you can probably imagine my frustration, especially since our content creators try and stay as script-efficient as possible. We'd like to continue to provide a quality combat experience in a sim that runs well, but we simply can't do it like this.

We have tried narrowing down the problem to no avail. Restarting, cleaning the sim up, checking for griefing objects through either debug panels or process of elimination has not helped. We have even contacted the chungs to request a datacenter move, which they have stated was completed as of earlier today. Nothing changed, and script run in statistics STILL reports around a 94% average over a minute with nothing much going on, which is pretty good with all the group infrastructure devices we use.

Now, all this being said, something that I HAVE noticed is that other sims in the same community have not been suffering like ours has, everything still works fine in a couple others, and something I noticed was this:

[22:18] Second Life: Teleport completed from NOT OUR SIM (212,149,121)

[22:18] Second Life: The region you have entered is running a different simulator version. Current simulator: Second Life Server 2019-09-13T20:04:44.530946 Previous simulator: Second Life Server 2019-09-06T22:03:53.530715

[22:18] Second Life: Teleport completed from OUR SIM (245,224,41)

[22:18] Second Life: The region you have entered is running a different simulator version. Current simulator: Second Life Server 2019-09-06T22:03:53.530715 Previous simulator: Second Life Server 2019-09-13T20:04:44.530946

The above is me teleporting FROM a sim that works fine TO our home sim (the defective one), then FROM our home sim TO another sim that also functions well. As you can see, the sims that are working share the same server version. To this end, I'm THINKING that it's perhaps a bad server update?

So, in conclusion, my questions are:

Does anyone have any idea what else we can try?

Has anyone else noticed this issue?

If anyone can get back to me on this, I appreciate it in advance.

Edited September 24, 2019 by JoachimLichttrager
Reworded for better description of problem.

[Qie Niangao]

If you look at the (barely) current server deploy thread you'll find that the version loaded on your sim this past week corresponds to the Magnum RC whereas the other sim is on the much more common standard release. This week, the standard server release gets updated (today, in fact) to the version that was tested on Magnum last week, so if it was the cause of the performance issue, there shall be widespread wailing and gnashing of teeth by nightfall, and it won't even be role play.

Of course, it could be something else, and it could just magically improve on its own. I have to say, it's weird that you're seeing server-side lag when scripts aren't affected -- they're usually the first to go. Well, except Pathfinding... you're not using any Pathfinding characters in your region, right? (also, just to be paranoid: no Animesh NPCs or critters around either, right?) You may need to post the fully-expanded Simulator section of the Statistics Bar to see if anything stands out to anybody else (or invite us all over to sniff around for ourselves, I guess).

[JoachimLichttrager]

Hey, thanks for the response.

Nope, we don't have anything with pathfinding out as far as I know, and perhaps only 2 animesh NPCs? I'd be happy to invite folks for a nip 'round, but also don't want anyone to jump down my throat for "advertising" in an improper place (which is why I redacted the sim names from the logs), so on that front I'll stick to posting the screenshot of the statistics bar as you suggested unless I get permission to do otherwise. I'll throw in top scripts and top colliders as well, for good measure.

I also looked at what you said with the server rollouts and I guess unless I'm absolutely crazy I'll get some more feedback soon.

Edited September 24, 2019 by JoachimLichttrager
Redacting a parcel name from one of the screenshots.

[Lindal Kidd]

I have no idea how to solve your problem, but anyone who has someone named "Vlad the Inhaler" on their sim has a fine, fine life indeed!

[Nalates Urriah]

@JoachimLichttrager You are not alone. There is a thread in here, somewhere, about this problem over the last 3 (?) months. Run Scripts %

I suggest you attend the Server-Scripting UG meeting Tuesdays noon. People have been talking about the issue. Rider Linden has been working on it.

The Lindens can see the problem. They just haven't pinned down the cause or exactly when it started. There is also the problem that it is intermittent. Sometimes a region restart helps or eliminates the problem for a time.

It really messes up those trying to use Pathfinding. It also messes up sailing, flying, and driving as it adds huge lag to steering control inputs.

[JoachimLichttrager]

Hey thanks for the feedback, I'll definitely check that out, especially if things don't improve by the time it rolls around again.

Also, apologies if it sounded like I meant "This isn't affecting anyone else." when I mentioned things like RP sims and such. I'm sure others are suffering too, but I can only speak for my experience when I say that back when I did RP stuff, I could still ultimately do what I was in the sim to do (posting, simple dice HUDS, and such), even with a lot of lag.

[JoachimLichttrager]

Alright so with the recent server rollouts I'm even more confused because our sim performance is stable again and back to what we're used to.

I don't exactly have a technical background but it seems weird as hell to me that the rest of the grid now being up-to-date with the latest release candidate has now fixed the issues plaguing us when we haven't restarted the sim today, and we were on Magnum to begin with. Maybe less power is dedicated to server testing versions? But that's a different question entirely, so this particular issue is closed, and thanks @Qie Niangao and @Nalates Urriah for your counsel.

[Qie Niangao]

14 hours ago, Nalates Urriah said:

@JoachimLichttrager You are not alone. There is a thread in here, somewhere, about this problem over the last 3 (?) months. Run Scripts %

The problems we've been chasing in other threads I know about are associated with a reduced Scripts Run percentage. This case was different because that metric was ideal on this sim despite the reports of severe lag. Indeed, the whole Statistics Bar looks perfect. Still, though, it could be related to the other cases because (we now know) the problem similarly sometimes comes and goes on restart (or version update), seemingly at random.

This appears to be the region I got to by perving the OP's profile. While my alt was visiting (before this last round of restarts) things seemed quite normal -- but that's not informative because I didn't try any of the things reported to be problematic. (Now I wish I'd tried some but I ran out of time.)

So I think this was in fact a very rare case -- and if it recurs, it could be a very informative case for Lindens trying to track down what causes these other strange performance problems (such as the depressed Scripts Run percentages of other threads) that seem to come and go at random upon restart.