Found out why my sim is in script overload.

[animats]

Finally found out what's using all the script time. And it's something I've never seen mentioned before.

 Vallone is running incredibly slow. I used Firestorm's area search to get a list of all objects, including child objects, which turned out to matter. Selected them all, then right clicked and selected "Script Info". That produces over a thousand lines or so of data in local chat, so I copied everything in chat and pasted it into a text file. Then I wrote a Python program to reduce that data. Here's the result:

Processing 'simscriptinfo2.txt':
1343 items.
Top items:
 10.9959ms  186 items: "BuiltUsingSkidzPrimz"
  1.7218ms   29 items: "a Skidz Prim"
  1.1475ms   15 items: "cyberpunk kitchen"
  0.6147ms    8 items: "bedroom shutters"
  0.6127ms    8 items: "bedroom windows"
  0.3492ms    9 items: "Door"
  0.3061ms    4 items: "cyberpunk lamp"
  0.3058ms    4 items: "cyberpunk awning"
  0.2749ms   76 items: "Curved top road"
  0.2298ms    3 items: "cyberpunk room separation"

Look at the top item. "BuiltUsingSkidzPrimz" child prims are using most of the available script time. That's a clever prim building tool from the early years which makes prims snap together like Lego blocks. It's still for sale on Marketplace. Apparently it still uses script time, long, long after the building job is complete. Anyone know how to use Skids Prims? Maybe someone just needs to push the "Finished" button or something so the scripts shut down.

A house in the sim was built with Skidz Primz. Nice looking build. The 186 child prims of the 232 LI house are using most of the script time for the sim just sitting there. There's only 16ms available, and this house is using 13ms of that doing nothing. I've never seen the house used, although it's near me. Sent the owner an IM and filed a ticket.

A few other items on that list could use some attention. But this is the big one.

Edited April 13, 2019 by animats

[animats]

Here's what the raw script info data looks like:

[01:06] Script info: 'BuiltUsingSkidzPrimz': [35/35] running scripts, 560 KB allowed memory size limit, 0.062450 ms of CPU time consumed.
[01:06] Script info: 'BuiltUsingSkidzPrimz': [35/35] running scripts, 560 KB allowed memory size limit, 0.062926 ms of CPU time consumed.
[01:06] Script info: 'BuiltUsingSkidzPrimz': [35/35] running scripts, 560 KB allowed memory size limit, 0.062922 ms of CPU time consumed.
[01:06] Script info: 'BuiltUsingSkidzPrimz': [35/35] running scripts, 560 KB allowed memory size limit, 0.062924 ms of CPU time consumed.

136 lines like that. Does that mean each prim has 35 scripts? This has me puzzled. I can examine the house's prims, and I see a few scripts, but I don't see any in the child prims named "BuiltUsingSkidzPrimz". Not sure what's going on here. It's never been this bad before, and that house has been there for the last year. Something strange is going on.

Sim load stats:

I've never seen Pathfinding Characters Updated that low before. 20-30%, yes, but 2.6%, never. My one animesh NPC can't function at all.

[Qie Niangao]

2 hours ago, animats said:

Something strange is going on.

That's for sure. There aren't even all that many scripts active on the sim, and the Scripts Run percentage isn't all that bad. (It's not great, but not that bad.)

Do we remember what it means for Sleep Time to be a huge chunk of the Script Time? This seems to ring a bell for me, but only very faintly.

(Just in passing: the Physics time looks just a little higher than I might expect. I wonder if that extra might be due to Pathfinding being active. The region doesn't need navmesh rebaking, does it?)

[Theresa Tennyson]

Oh, Jeezum Crow....

The house is a commercial build from about 2007, looks like. The description says it's been "resized."

From what I've seen, I get the horrible suspicion that every single prim in that build might have a resize script and any resizing was never "finalized." Possibly there's no "time-out" in the scripts either.

It may be a case that every single prim in that house is sitting there going, "Bueller?... Bueller?... Bueller?" listening for the next command and will continue doing so until Second Life closes down.

Ahhh, nostalgia for the "good old days" of building...

Edited April 13, 2019 by Theresa Tennyson

[animats]

6 hours ago, Theresa Tennyson said:

From what I've seen, I get the horrible suspicion that every single prim in that build might have a resize script and any resizing was never "finalized." Possibly there's no "time-out" in the scripts either.

It may be a case that every single prim in that house is sitting there going, "Bueller?... Bueller?... Bueller?" listening for the next command and will continue doing so until Second Life closes down.

Something like that. But when I look at the objects involved with Edit, I don't see such scripts. I only see useful stuff like window shade and light scripts, and not many of those.

I've found out more about "Skidz Prims". They still have a big inworld store. Their marketplace page says there's a class there at 11 AM each Saturday, but nobody showed up to teach it. Someone from Builders Brewery told me that prims have a name of "a Skidz Prim" during building. When you're done, you click "Finished". That broadcasts a message which causes the prims to change their name to the name of the build ("BuiltUsingSkidzPrimz" if you don't name it) and delete all the scripts. For most of the prims, that was done. The builder did do the "Finish" operation.

The number of scripts seen in Area Search / Script Info for these prims is 35, but Edit doesn't show any of them. This might be a permissions problem, but I do see a few useful scripts, like drapes and lighting control, in Content. It's as if some resource from scripts in the past was never released. This is a build from 2007, so this may be the result of some long-ago bug.

I've tried to reproduce the problem by creating a prim with scripts and deleting them. Area search script info and edit script info agree as I add and delete scripts, or change them to non-Mono. So, not able to reproduce the problem.

[Alyona Su]

Skidz Primz tools was a thing back in 2006 (because people didn't know how to do a "duplicate off the edge" of a prim when building for a perfect fit. Could it be that those crusty old scripts are the same crusty old scripts from back then? Just curious.

Edited April 13, 2019 by Alyona Su

[animats]

9 hours ago, Qie Niangao said:

Do we remember what it means for Sleep Time to be a huge chunk of the Script Time? This seems to ring a bell for me, but only very faintly.

Me too. I've seen it before but not understood it. High Sleep Time with low Spare Time is very unusual. Not the usual situation. Deleting my pathfinding character does not help.

 

Charlesville sim. Zero sleep time, even with a pathfinding object active.

 

Quote

(Just in passing: the Physics time looks just a little higher than I might expect. I wonder if that extra might be due to Pathfinding being active. The region doesn't need navmesh rebaking, does it?)

No, the navmesh is current. Removing my pathfinding characters does help, and takes about 3-4ms off the sim load. (Pathfinding load is not per-character; additional characters after the first don't increase it much.) "Sleep time" remains high.

[animats]

I'm very frustrated about this, since it breaks my pathfinding characters. In a way, though, it's encouraging to be able to point to some specific prims and say "these prims are being mishandled by the sim code". For a while, there's been sort of a vague feeling that sim script loads are increasing, but no one knew why. Now there's something to point to that's clearly not working right.

This image sums it up.

Area search with child prims has brought up the list of relevant prims. I've blanked out the ownership info; it's not their fault at all.

Script info says: 35 running scripts for that child prim, and using script time at the typical rate for an idle script. Edit shows nothing in Contents. No scripts. That's not right. And that's what to look for. Can anyone find a similar situation in another sim? Thanks.

Submitted JIRA.

Edited April 14, 2019 by animats
Submitted JIRA, link.

[Theresa Tennyson]

12 hours ago, animats said:

Script info says: 35 running scripts for that child prim, and using script time at the typical rate for an idle script. Edit shows nothing in Contents. No scripts. That's not right. And that's what to look for. Can anyone find a similar situation in another sim? Thanks.

Submitted JIRA.

As far as the same situation in another region, there was one showing just about exactly the same symptoms in the post you made initially on the script overload topic where you posted screenshots of several regions with low scripts-run percentages.

Edited April 14, 2019 by Theresa Tennyson

[Qie Niangao]

I'm not so sure the Skidz Primz thing is the problem it may seem. I have a script I use for investigating stuff like this* and as far as I can tell that whole house, all 232 land impact worth, contains 35 scripts total, using 560 KB and 68 μs/frame. And I think I can guess where those scripts are: The house has a "Brennan Homes House Control Remote" on the wall just inside the door, containing "Paskis' Keyserver 1.11", each window prim contains "Paskis' Window Script v3.4.6", the light over the door contains "Paskis' Light Bulb v2.7.0" and there may be others. These are scripts from 2006 and 2007, before the "efficient scripts" initiative brought us link-directed parameter setting, so having such scripts scattered all around the build is the nature of prefabs of that era.

(That it's still the practice in the new Premium Homes, presumably to get per-link touch interaction, is less forgivable.)

That said, I don't know anything about Firestorm's "area search" so I can't begin to interpret what it's telling you. I'll login an alt and let him try to use that viewer and see if it will reveal anything, but I'm guessing there's a more sim-systemic problem here, not so much attributable to any particular build.

[ETA: As usual, I'm stumped by Firestorm. Can't get Script Info to work anywhere, not only inside the Area Search results. I suspect it's reporting the total object script time over and over again, once for each link in the linkset, but I just don't seem capable of learning to operate this viewer.

Anyway, let's see, your Python script summed up the times of 186 of the named child prims for a total time of about 11 ms, and the reported times you saw were about 63 μs each (close enough to the 68 μs I just got for the whole house) and 63 μs * 186 = 11.718 ms so that kinda fits, unless those really were per-link script times.]

_________
*It's every bit as laggy as you imagine, which is why I don't distribute it, and why I try to remember to remove it when I go out in public.

Edited April 15, 2019 by Qie Niangao

[Da5id Weatherwax]

Wow... it IS bad isn't it? Just walked past your place out of nostalgia - my main store used to be on the parcel next to yours (years ago) and its way worse than I ever remember.

[HunniHope]

5 hours ago, Qie Niangao said:

 

[ETA: As usual, I'm stumped by Firestorm. Can't get Script Info to work anywhere,

Try rebuilding your bridge on a  script enabled region. Avatar menu > avatar health > recreate LSL Bridge. & make sure you have it enabled still in your preferences. More info here https://wiki.phoenixviewer.com/fs_bridge & here https://wiki.phoenixviewer.com/fs_bridge_issues

[Love Zhaoying]

You would think the SkidzPrims scripts would not use much if they use standard listeners, and nothing is talking on their channel. Rip out all the scripts.

[Qie Niangao]

38 minutes ago, HunniHope said:

Try rebuilding your bridge on a  script enabled region. Avatar menu > avatar health > recreate LSL Bridge. & make sure you have it enabled still in your preferences. More info here https://wiki.phoenixviewer.com/fs_bridge & here https://wiki.phoenixviewer.com/fs_bridge_issues

Thanks, that's exactly what it was: I'd disabled the LSL Bridge creation while mostly using a different viewer and forgot about it when I started using Firestorm more regularly.

So now I was able to look at how Area Search reports scripts in (my) child prims, and indeed it just repeats the statistics for the entire linkset one time for each child prim (when child prims aren't filtered out of the results). So I'm pretty confident that the "BuiltUsingSkidzPrimz" items really aren't the source of lag here (and that they indeed do not contain any scripts).

I really think the problem here is with whatever is causing that weirdly inflated Sleep Time, accounting for the majority of all Script Time. That's very unusual (in my limited experience) and reminds me of the weird "Pump IO" times in Vallone a year ago. For a relatively nearby sim in more typical script lag, try Broadwater, where there is literally zero Sleep Time and negligible Pump IO. (The source of Broadwater's deep script lag is no mystery: a standard issue breedable factory farm.)

 

[animats]

54 minutes ago, Qie Niangao said:

So now I was able to look at how Area Search reports scripts in (my) child prims, and indeed it just repeats the statistics for the entire linkset one time for each child prim (when child prims aren't filtered out of the results). So I'm pretty confident that the "BuiltUsingSkidzPrimz" items really aren't the source of lag here (and that they indeed do not contain any scripts).

Oh. Then the total script time I computed is meaningless, because I added up the stats Firestorm gave me for each child prim. Firestorm's reporting script info for the entire linkset for each child prim isn't right. It's probably using OBJECT_RUNNING_SCRIPT_COUNT and related params to llGetObjectDetails, which returns info for the entire linkset.

So that's what's wrong with the analysis. But I'm still stuck with a sim that is performing so badly that I can't use or work on my pathfinding characters. And this is a sim where almost nothing is going on. The load wasn't this high when there was an elaborate carnival in the sim, with lots of rides and visitors. Something is still badly broken. But what?

Edited April 15, 2019 by animats

[Qie Niangao]

2 hours ago, animats said:

But I'm still stuck with a sim that is performing so badly that I can't use or work on my pathfinding characters. And this is a sim where almost nothing is going on. The load wasn't this high when there was an elaborate carnival in the sim, with lots of rides and visitors. Something is still badly broken. But what?

Yeah. Let's plead for some information at the Server User Group tomorrow, about what it might mean that Sleep Time is tremendously inflated accounting for most of the Script Time in the sim.

There's a meta issue here, too: we have virtually no insight into how Operations might go about troubleshooting any performance problem, beyond a few notes in the wiki about Estate performance tuning and hints about what the statistics are supposed to measure. I'll betcha Operations has derived something like a diagnostic fault tree to track these things down. Something like that would be a mighty handy thing to have on the wiki, and might save the Lab a lot of wasted support tickets.

I forgot to check uptime on Vallone, but this has been going on for a while, right? It's a problem that's persisted through a restart at least? Otherwise that's sure to be the first thing they'll suggest, reasonably enough, and we won't learn anything.

[animats]

15 minutes ago, Qie Niangao said:

Yeah. Let's plead for some information at the Server User Group tomorrow, about what it might mean that Sleep Time is tremendously inflated accounting for most of the Script Time in the sim.

...

I forgot to check uptime on Vallone, but this has been going on for a while, right? It's a problem that's persisted through a restart at least? Otherwise that's sure to be the first thing they'll suggest, reasonably enough, and we won't learn anything.

It was like this right after a restart. In fact, I was in the sim when I got the yellow "Restart coming" alert, crossed the street, and watched it restart. Went right back in, and it was still overloaded.

[Prokofy Neva]

Scraping data like this on an entire sim and finding all the objects of other private property owners is an invasion of privacy. I remember what havoc Electric Sheep caused when they scraped all of SL in this way and ruined people's privacy and also caused theft (because people casually set things to $0 for another person to buy, and when the whole world is in search, griefers can hunt for $0 objects). Eventually this search was scrapped, but I wasn't surprised to find that Firestorm built it into their viewer, and it's not the first time they've enabled data-scraping.

The response to this kind of concern, although privacy is a new obsession for the techs who hated copyright -- an obsession I don't share -- is usually "but you can do it because you can". "Code is law," and if you can area-search, so what, you area-search. Even so, Firestorm could be persuaded to change if there was enough community awareness. Of course area-search enables cheating on hunts.

What do you do when someone on the sim where you live hogs all the resources? It can be a script eating problem like this (I remember submarines diving up and down that brought a sim to its knees years ago). Or it can be a club that takes all the avatar spaces. If there were an abuse report category called "overuse of resources," then perhaps the Lindens could act and enforce fair use -- but what is fair use?
 

In a world with such cheap land, the obvious thing to do is to move, especially because land in Zindra fetches very good prices, and given that some land was gotten for a song *cough*, moving out would seem to be a wise economic decision.

Of course, a note to the owner to delete the listening scripts from their building operation might also be in order...

[Dean Haystack]

On 4/16/2019 at 12:44 PM, Prokofy Neva said:

(...)In a world with such cheap land, the obvious thing to do is to move(...)

I've had to move several times before when new arrivals drop by and start using all the resources.

I always make a point of telling the land owners I was renting from exactly why I'm leaving. If in one hand they wanna fill all their rentals, on the other they can't just accept anyone in there that will hog all the resources. I can only hope that by complaining they'll think of a vetting process in the future.

With 0.001ms of spare time I'm be packing my bags and looking for greener pastures.

[Kali Scientist]

Please bear with me. I am only aware of "Python" as malware/virus and was freaked when it wanted to open as part of second life new download. Is this a safe program that is required with second life now????

[animats]

14 hours ago, Kali Scientist said:

Please bear with me. I am only aware of "Python" as malware/virus and was freaked when it wanted to open as part of second life new download. Is this a safe program that is required with second life now????

What did you do? Where did you get the viewer?

The only legitimate place to get the Linden Labs viewer is "secondlife.com".

The only legitimate place to get the Firestorm viewer is "firestormviewer.org".

Developers who build the viewer from source code use Python during the build process, but end users do not. Python is also used by creators using Blender, a 3D design program. Unless you're doing viewer development or moderately advanced content creation, Python should not be involved with Second Life at all.

If you got a fake viewer from another source, please post the link so that Linden Labs or the appropriate viewer team can be informed.

[Kali Scientist]

That is exactly what freaked me out! I DID download the new viewer from second life.com when I got a new imac. I have removed second life and Python which was showing on my dock bar went with it. Before I removed it, I found second life viewer in Finder but NOT python. Again this program just came in with viewer. What now?

[Whirly Fizzle]

2 hours ago, Kali Scientist said:

That is exactly what freaked me out! I DID download the new viewer from second life.com when I got a new imac. I have removed second life and Python which was showing on my dock bar went with it. Before I removed it, I found second life viewer in Finder but NOT python. Again this program just came in with viewer. What now?

The LL viewer uses Python for the SLVersionChecker - this is an app launched as a child process of the viewer to check for updates etc.

[animats]

1 hour ago, Whirly Fizzle said:

The LL viewer uses Python for the SLVersionChecker - this is an app launched as a child process of the viewer to check for updates etc.

Good grief. I asked a Firestorm dev, and they said "no Python". Didn't realize the LL viewer had that. Does the LL viewer install Python just for that purpose?

[Resi Pfeffer]

On 4/13/2019 at 9:37 AM, animats said:

 Vallone is running incredibly slow. I used Firestorm's area search to get a list of all objects, including child objects, which turned out to matter. Selected them all, then right clicked and selected "Script Info". That produces over a thousand lines or so of data in local chat, so I copied everything in chat and pasted it into a text file. Then I wrote a Python program to reduce that data. Here's the result:

I dont know python at all, but can i get the same results copying it into a excel sheet in order to sort it? I just need to know, whats the biggest millisecond value, right?

Edited May 16, 2019 by Resi Pfeffer
typo