Modern Sim Lag Times

[Canis Canning]

Many of my guests argued about sim lag on the 4000sqm parcel on a fullsim when they are walking around. You walk and suddenly you are only gliding und then jump bag from where you were before starting to glide. So I wrote a note to my landlord and also tried to find out myself what is happening. First the landlord told me ther is no lag on my parcel (because they are only short peaks) The result was: Lag is normal nowaday on fullsims. The performance breaks down on evry entry of an avatar to the sim. For me this happens about 40 times in an hour. Many do not stay for long on the sim. There are seldom more than 10 avatars on the sim at the same time. But 40 walk-lags during an hour are nasty.

The landlord thinks this situation is totally normal, so I got angry that he does not do anything to fix this problem. I decided to move to a better sim . And now my odyssee startet. Everywhere I looked at the sim statistics using ctrl-shift-1 and nearly everywhere if the sim wasn't empty I saw the same picture:

https://gyazo.com/429df6dd5be2b6bc14f972a6048b3b5e

1. nearly all half filled sims ar at their limits. "Spare Time" is often nearly 0.
2. "Script Time" is very near to to  "Total Frame Time", so scripts take the most part of the the total time of a sim., even if there are only 1 or 2 agents on the sim.
3. The new "Scripts Executed" (not sure about this tag name) offen is just above 30%, that means as far as I know that 70% of all scripts are waiting to get executed again (Remember that the server stops scipts first if a sim is in danger to lag)

There are a lot of questions I have, does anybody have an opinion on them?

1. Do I see this problem right?
2. Is this also caused by the 30% more prims, that often also contain scripts?
3. Is it possible to get the values of "Spare Time" and "Scripts Executed" from a script to judge the lag-danger for a sim?
4. Is there a way to get away from this lag?
5. Are the sim owners aware of this bad situation and are there any activities on LindenLab side to solve this problem?
6. What will happen when LindenLab also shifts the computation of graphics from the viewer to the already over strained server as is planned?

In old times the problem of lag when an avatar jumps in only existed on homesteads, because more homesteads are on one cpu. Now sim lag seams to be common on a full sim

 

[Qie Niangao]

The statistics you show are from a very overloaded sim. It has way, way too many scripts (and unusually many per scripted object, so not really due to extra prims, but rather suggests a plague of breedables... something else, even left-behind griefer objects), and that inflated script count is one reason even 14msec / frame is only enough for a tiny percentage of the scripts to execute.

But scripts by themselves do not cause the kind of lag this sim is showing: hugely inflated "Simulation Time". It's possible some scripts are doing something to cause that statistic to inflate, but whatever is causing that, it's what's extending the frame -- which means time dilation and rubber-banding type lag.

I'm sorry, I honestly don't know what Simulation Time means in detail, so I'm hoping somebody else has a clue. The Help text says it's " The amount of time that frame spent running other simulations (agent movement, weather simulation, etc.)" but that's a big "etc" there -- it's surely not agent movement with only 3 avatars on the sim.

If it were me, I'd want to expand the Pathfinding statistic to see if there happens to be some "AI Characters" defined for the sim. Or maybe look around with "Show Updates to Objects" (Ctrl-Alt-Shift-U) toggled on, to see if there are a bunch of critters actively using simulation resources.

[animats]

The estate owner tools provide a list of which scripts are using the most time. Unfortunately, you have to be an estate owner to use them. Your stats show a sim in severe overload.

I have a simple tool to report intermittent lag.   This just reports when its timer events are being delayed. It's useful when the average lag is low but there are intermittent overloads and stalls. If you're on permanent overload, it won't help, but if it's an intermittent problem, it might.

[Klytyna]

On 23 June 2018 at 9:40 AM, Canis Canning said:

"Script Time" is very near to to  "Total Frame Time", so scripts take the most part of the the total time of a sim., even if there are only 1 or 2 agents on the sim.

Odds are most of the sims scripts, are, like most non tuned sims, in the sims objects, all the furniture, those fancy trees that shed 'blossom', the babbling brook sounds thing hidden under a stream, etc., and NO)T worn on avatars.

I've seen sims with that number of scripts, running with % run figures of 30-50% when the sims almost empty of people, lagging down to under 4% run is a bad sign.

Heres how it works...

Bottom of the crtl-shift-1 display, "Frame Time, thats supposed to be about 22 ms, that is to say, 1/45th of a second, for that Sim 45/ Phys 45 Dilation 0.997 that you see on the WORTHLESS sort of lag meter.

Frame time is made up of a list of sub times, and they are displayed in a priority order.

Physics time and Simulation Time is higher priority than scripts for example, and if physics time climbs, scripts gets cut back to keep the Frame Time at 22 ms, thats a major cause of script lag, unresponsive menus etc., low % run scores.

But scripts do not in themselves make the sim laggy to walk around on.

The problem seems to be net time and simulation time. I'm guessing here, but it looks like a LOT of the scripts on the sim use http, use it too damn muchg, too many vendor boards, or prim tv's, or something.

Or items connected to some database, breedables of some sort, the fact that you apparently have the sim running better most of the time, with regular pulses of major lag, suggests whatever those mystery things are, a lot of them are all firing up at once and doing heavy web work, or theres a lot of things on the sim using really bad cases of llSensor...

 

[Whirly Fizzle]

On 6/23/2018 at 9:40 AM, Canis Canning said:

The performance breaks down on every entry of an avatar to the sim.

It's very likely to be this problem: BUG-8946 - Issue with sim time dilation and FPS dropping significantly with first-time mesh injection/rezzing/attaching/rendering in a region

[Ayesha Askham]

33 minutes ago, Whirly Fizzle said:

It's very likely to be this problem: BUG-8946 - Issue with sim time dilation and FPS dropping significantly with first-time mesh injection/rezzing/attaching/rendering in a region

Sadly this JIRA has been running for over 3 years, pretty much since the introduction of fitted mesh, I think and despite the JIRA being "resolved and closed to further comment" according to LL, the issue is very much ongoing.  I can only assume that LL cannot fix this since there has been so much clear data submitted after Caleb's request.  We (I) see its effect at almost every heavily loaded ( and on some I would call lightly laden) region whenever an avatar wearing mesh arrives (and that , today, means essentially EVERY avatar.  The server load generated by the need to resolve a mesh item is clearly taxing the simulator and the operation interferes with all other functions on the region. WHY there has been such a deafening silence from LL has me wondering just HOW much they understand their own processes.

Edited June 24, 2018 by Ayesha Askham
factual correction

[Willow Wilder]

18 hours ago, Ayesha Askham said:

despite the JIRA being "resolved and closed to further comment" according to LL, the issue is very much ongoing.  I can only assume that LL cannot fix this since there has been so much clear data submitted after Caleb's request. 

"Resolved" under the "Dates" section of the ticket, refers to the date the action occurred. The Status and Resolution under the "Details" section of the ticket are relevant and https://jira.secondlife.com/browse/BUG-8946 both state Accepted. This means it has been imported to an internal LL project and tracked internally as MAINT-7504. You can find the MAINT number under Activity - History tab. Viewer release notes then reference the MAINT numbers.

It's a bit of a maze to be sure.  

http://wiki.secondlife.com/wiki/Bug_Tracker/Status

 

[Qie Niangao]

This jira history is strange. It sat for two years in a "needs more info" status after that information was provided. If it's as big a problem as it's described, I don't understand how that happened, nor how it's been a year without a fix since it was Accepted.

Maybe I'm not understanding the description, but I don't think I'm seeing this where I should. Pretty much every avatar is covered with multiple layers of fitted mesh now, as Ayesha says, so wouldn't we expect this problem to crop up constantly at every place avatars come and go? And I really don't.

Is there any way somebody can peek inside the MAINT jira to see if there's some hint of an explanation for what developers think might be going on? That might clue us in on measures we could take to protect a region, in lieu of a fix.

(I'd just reiterate that, notwithstanding this bug, that gyazo image is for a wildly overloaded sim with way too many scripts in relatively few objects, so there are objects in that particular region with abnormally high script-count, and that'll be a problem for its scripts-run percentage -- so scripts will be delayed -- regardless of time dilation. I'd be interested to know if BUG-8946 typically causes this inflated Simulation Time -- it doesn't appear so in the image attached to that jira.)

[Willow Wilder]

6 minutes ago, Qie Niangao said:

This jira history is strange. It sat for two years in a "needs more info" status after that information was provided. If it's as big a problem as it's described, I don't understand how that happened, nor how it's been a year without a fix since it was Accepted.

Maybe I'm not understanding the description, but I don't think I'm seeing this where I should. Pretty much every avatar is covered with multiple layers of fitted mesh now, as Ayesha says, so wouldn't we expect this problem to crop up constantly at every place avatars come and go? And I really don't.

Is there any way somebody can peek inside the MAINT jira to see if there's some hint of an explanation for what developers think might be going on? That might clue us in on measures we could take to protect a region, in lieu of a fix.

(I'd just reiterate that, notwithstanding this bug, that gyazo image is for a wildly overloaded sim with way too many scripts in relatively few objects, so there are objects in that particular region with abnormally high script-count, and that'll be a problem for its scripts-run percentage -- so scripts will be delayed -- regardless of time dilation. I'd be interested to know if BUG-8946 typically causes this inflated Simulation Time -- it doesn't appear so in the image attached to that jira.)

No peeking! Oh how I wish we could sometimes. 

I can't comment on that particular issue, but I reported a region resource problem where I have a parcel which became practically unusable months ago thanks to the most recent neighbors who DGAF. I was told the region had about 16,900 individual scripts running at best about 70% average. They typically recommend around 4,000.  

[Ayesha Askham]

Willow, Qie

Thanks for the clarification, Willow, I clearly do not understand Lindenese!  The fact that no further information can be submitted is my chief gripe, after the simple fact that absolutely NOTHING has been done about this issue.

I am surprised, Qie, that you do not see this issue since I experience it daily in the sims that I visit, not all of which are over-stuffed either with scripted items or vendors.  All one needs to see is that way the simulator nearly stalls in some locations when a teleport in occurs.  Teleports out do not have much effect, unsurprisingly, until sims are so overladen that they are on the point of collapse!

[Canis Canning]

Thank you for all your comments, they were very helpful for me. Renting booms on the sim where I live. It is an adult sim with everybody thinking he needs a BDSM Dungeon with lots of "machines" (maybe with capturees running). Normally with nobody on the sim but me executetd script are 50%. Last night with about a dozen avatars the percentage of executed scripts sank below 30%. My Landlord thinbks the situation is normal and just closes  tickets, saying "there is no permanent lag", because nowadays the lag is script lag, by stopping most of the scripts. My wardrobe does not work anymore, my holodeck system (mostly in idle state with only  7 scripted objects and rarely used) does not rez and move objects right. I have to look for a less overloaded sim and maybe a landlord that cares for this problem - any hints who may that be?

THE NEW SL PROBLEM IS SCRIPT LAG

 

By the way: At the moment I rent from Amestate. They are not aware of this kind of problems.

Edited June 26, 2018 by Canis Canning

[Richardus Raymaker]

Anybody thinked about 'temp rezzers' ?

The are good for spikes, hard to spot. but in the past i always picked then out of it by eye. And some tricks. 1 temp rezzer is not si bad. but not sure if there are 10

[Qie Niangao]

I took a field trip to NARCISCO and looked around. I think you might slightly improve your own situation by removing all the fish in the pond. I'm also a bit suspicious of the Jian dove, the TLC Moonlight Dragonfly/Liliy Set", the TLC woodpecker, and I don't know why "HPMD* Bright Particle (texture2)" always sends full object updates -- but none of these are the source of the script lag, per se. (Actually, the swans and swan baby burn more script time than the fish, but the fish are just flooding the network with constant object updates; the other objects tend to be more "bursty" in update load.)

At about 3000m the HPMD weasel is also sending full object updates. Somebody else has an "Animated Ground Skeleton" at about 2700 that is actually using more than usual script time as well as streaming constant object updates (despite having only 2 scripts), but again it's not accounting for much of the script load all by itself, and if there's nobody around to receive the object updates they don't much affect anything. The reason to care a bit more about ground level is that (I assume) there are more avatars around there more of the time to receive all those object updates and experience lag.

Before I could descend further, the sim got its turn at rolling restarts, so I'll try to remember to get back and look more later.

Usually this number of scripts indicates a breedable factory farm somewhere on the sim, but I'm not finding it so far. As you say, there are a lot of different levels of buildings with heavily-scripted furniture. (There's even some old XPOSE stuff, which is like a nightmare version of AVsitter: both engines use way too many "modular" scripts, but XPOSE is several times worse.)

[Qie Niangao]

As far as I can tell, the huge script count per object is boosted by a bunch of XPOSE-scripted furniture -- with 25 to 40 scripts per item -- but a lot of the script time comes from a flock of collectible Schadenfreude Kiwi birds each with only one script but those scripts using upwards of 60 µs each, per frame. They're at about 813m height, scattered around (14, 148), some under the floor... it's quite a menagerie here. On the same platform to the northeast, there's a pond with a lot of  *HEXtraordinary* Gecko wanderers, each with 4 scripts totalling 20µs per gecko, so not that bad although there are quite a few of them.

I didn't find a true breedable farm here, and during my visit there was no time dilation at all, just scripts delaying other scripts. I notice there are about 16,000 objects total in the sim, which wouldn't have been possible before the upgrade, but there aren't such an unusual number of scripted objects -- just a lot of scripts total.

If it's any consolation, I have land on a Mainland sim that really is breedable-infested, where about two thousand scripted objects have a total of over 14,000 scripts, and less than 25% of scripts run each frame. Lemme tell ya, adjusting alpha cuts on a mesh avatar is no fun -- it's easier to teleport out to another sim, make the settings, then teleport back. So I totally agree: script lag is the new SL problem, made both worse and more important because our avatars depend more on scripts all the time.

[Coffee Pancake]

 

6 hours ago, Richardus Raymaker said:

Anybody thinked about 'temp rezzers' ?

The are good for spikes, hard to spot. but in the past i always picked then out of it by eye. And some tricks. 1 temp rezzer is not si bad. but not sure if there are 10

(mainland) Our neighbor is making excessive use of them with several rezzors producing hundreds of objects on a 20 second loop spread over half a dozen builds on his plot .. literally everything he could put in one is in one (if not floor...temp!). "free sex" skyboxes with only 9Li of static content and 80 Li of temp content. Everything in his multiple stores that isn't a vendor. A floating art gallery shop in which the only static content is the floor, and so on.  The rezzors zip around his builds for no reason other than to hide their existence and he is fully aware that they are bad (to the point of getting paranoid when some Lindens showed up to test a region crossing issue I had reported) ... but he's a "srs biznss man" so that must make it ok, somehow.

The region is less than half populated and the script time is maxed out.

Multiple abuse reports from more than one of his neighbors ... and they are all still there. Seems LL believe their built in limits are satisfactory (they aren't) and unless the region is actually tanked and on fire aren't prepared to do anything.

 

[Qie Niangao]

One thing about temp-rezzers has changed, though: Mesh LI accounting. Temporary mesh counts toward the parcel's total Land Impact limit, not drawing on a special pool like the temp-rezzed prims of old. This also applies to prims with "mesh-accounting" LI, which might be triggered by setting a face Alpha-Masked instead of -Blended. At first I objected to this, along with a host of other "special case" LI rules that came along with Mesh and Materials, but now I think at least this part is a step in the right direction.

Maybe you can trick your neighbor into "saving prims" by setting those temp-on-rez prims Convex Hull... and then they'll count against the parcel LI whether they're temp or not. 

[Coffee Pancake]

He intentionally sticks to vintage prim based content to avoid this gotcha ... and I can't convince him of anything, he banned and blocked me the day he moved in.

[Canis Canning]

I once lived on a sim with temp rezzers (they were not allowed and I had to convince my Landlady again and again to tell it to the renter). Normally they produce a peak every 50 seconds, because the objects will live 60 seconds. They are not a such problem in the case of my carps (ok, you are right, I better rez them permanently), because it is only one small  vintage-prim, but if you use temprezzers for large buildings, then they produce a lot of lag.

Thank you Quie for looking around on Narcisso. In the mean time I travelled to dozens of sims looking for another place to rent, but it is the same picture everywhere. If a sim is nearly empty i.e. with only 12000sqm rented and so not that many skripts), then the sim works well. But if nearly all parcels are rented, then only 60% of the scripts are executed with no other avatars being on the sim.

So I do not see any chance it gets better if I move somewehre else. I also looked at homesteads and with "only" 2000 active scripts they show the same problem of "scripts blocking scripts".

[Phil Deakins]

On temp rezzers: Most people realise that, for about 16% of the time, there are 2 instances of each temporary object rezzed simultaneously, because the next one is rezzed significantly before the current one is removed by the SL system. But what many people don't realise is that there are sometimes 3 instances of a temporary object rezzed simultaneously. When it happens, there are always 2 instances rezzed and, for some of the time, 3 rezzed.

It happens when, say, a temporary sofa is sat on (temp rezzers were often used for furniture, and probably still are). The system doesn't remove temporary objects when they are sat on, so the one that's sat on stays rezzed, and the rezzer still rezzes the next one. Then, of course, the one after that is rezzed before that next one is removed, making 3 rezzed simultaneously for a while. The rezzing system can be scripted to stop rezzing if the current instance is sat on, but I doubt if any do. Mine didn't (I created and sold one back in the day).

Note: I have a Linden-made temporary objects detector if anyone wants a copy (created by Frontier Linden). He gave it to me many years ago, and once in a while someone will find mention of it in the archived forum and ask for a copy.

Edited June 27, 2018 by Phil Deakins

[Coffee Pancake]

Would appreciate if you could drop a copy on me in world Phil.

[Phil Deakins]

20 minutes ago, CoffeeDujour said:

Would appreciate if you could drop a copy on me in world Phil.

Duly dropped.

[Whirly Fizzle]

Please can I have a copy too Phil.
Thanks 

[Coffee Pancake]

Damn ... depending on activity on his land (as some rezzors are set to only fire up if there is a nearby avatar). He's rezzing between 320 and 660 temp prims every 20 seconds on a parcel that only supports 702 actual Li. 

How can this be ok? If it's ok, why aren't I doing it !!!

He really should own more land.

[Phil Deakins]

2 hours ago, Whirly Fizzle said:

Please can I have a copy too Phil.
Thanks 

Done it, Whirly. Things aren't loading properly but I dropped it anyway. Hopefully you'll get it.

[Phil Deakins]

2 hours ago, CoffeeDujour said:

Damn ... depending on activity on his land (as some rezzors are set to only fire up if there is a nearby avatar). He's rezzing between 320 and 660 temp prims every 20 seconds on a parcel that only supports 702 actual Li. 

How can this be ok? If it's ok, why aren't I doing it !!!

He really should own more land.

You need to move.