Overload

[animats]

I and others are seeing more overload situations. Unusually slow sim crossings, bad enough that it had Firestorm developers looking for a Firestorm bug. Sikkima sim seems to be intermittently difficult.  My home sim, Vallone, dropped to 25% scripts running. I turned off all my NPCs and escalators, and the load barely changed. Submitted a ticket. No new big builds there.

These are sims with one avatar present. It's not a crowd problem.

Edited May 20, 2020 by animats

[Ardy Lay]

13 minutes ago, animats said:

I and others are seeing more overload situations. Unusually slow sim crossings, bad enough that it had Firestorm developers looking for a Firestorm bug. Sikkima sim seems to be intermittently difficult.  My home sim, Vallone, dropped to 25% scripts running. I turned off all my NPCs and escalators, and the load barely changed. Submitted a ticket. No new big builds there.

These are sims with one avatar present. It's not a crowd problem.

It's a problem for crowds. 😉 We often have 30 to 44 and on the rare occasion 45 people in a region.  Poor scripts get no love on those nights.  I'll keep an eye on it and see if it's gotten worse there too. 

[animats]

LL support chat made this remark in response to a ticket about the overload:

Quote

One thing I would heavily advise is taking a look at the objects that have URLs; based on what I am seeing in the region there may be a URL that is not functioning as well as it should or has gone bad. ...The behavior of this region and some others I have frequented with "bad" or funky URLs is quite similar and that may be why it seemed to sort of spring-up. .

Now that's an unexpected source of load. "Having a URL" presumably means an object calling llRequestURL, which is what you do to set up a web server within SL. That's uncommon. Why that would result in high script load isn't clear.

CasperVend vendors might do that. They take commands from CasperVend HQ;  when you load your dropbox the vendors are told to update. So, maybe. But LL support said CasperVend wasn't the problem. What to look for? Anyone seen this problem before?

[Theresa Tennyson]

46 minutes ago, animats said:

LL support chat made this remark in response to a ticket about the overload:

Now that's an unexpected source of load. "Having a URL" presumably means an object calling llRequestURL, which is what you do to set up a web server within SL. That's uncommon. Why that would result in high script load isn't clear.

CasperVend vendors might do that. They take commands from CasperVend HQ;  when you load your dropbox the vendors are told to update. So, maybe. But LL support said CasperVend wasn't the problem. What to look for? Anyone seen this problem before?

You have - you've had issues with Vallone's performance a few times now over the last couple of years.

[animats]

On 5/22/2020 at 2:09 PM, Theresa Tennyson said:

You have - you've had issues with Vallone's performance a few times now over the last couple of years.

Yes, and I've looked hard for the cause. Shutting down all my stuff makes a small difference, maybe. It's smaller than random variation in load. Restarting the sim usually helps, and moving it to another server helps immediately, but then performance slowly drops. I've looked at what scripts are running all over the sim, and no one script is using much time. But there are about 6000 scripts in the sim, mostly idle. That in itself is a big load. Each idle script uses about 0.003ms per frame, which is tiny, but adds up. There's about 16-18ms of script time available per frame.

Known bug - overhead for scripts that do nothing is a major cause of load.

Edited May 23, 2020 by animats
typo

[Qie Niangao]

16 hours ago, animats said:

... "Having a URL" presumably means an object calling llRequestURL, which is what you do to set up a web server within SL. That's uncommon. Why that would result in high script load isn't clear.

CasperVend vendors might do that. They take commands from CasperVend HQ;  when you load your dropbox the vendors are told to update. So, maybe. But LL support said CasperVend wasn't the problem. What to look for? Anyone seen this problem before?

I'm not aware of ever encountering a URL-based problem, but I certainly use URLs here and there. As it happens, you currently have 3 URLs of your own in Vallone, according to About Land / General / Script Info. I think these data are pooled for all land in the region with the same owner. As parcel owner, you should be able to get details of the objects, including those with URLs. A few banlined parcels in Vallone don't seem to ever populate that table (but one of those owners also has a non-banlined parcel that does populate the table -- no URLs from that one).

Looking around some of my groups' parcels, on one there's an old 7Seas Gear Vendor that uses one URL, elsewhere a few get set up to generate browser-viewable reports, some "traffic servers" that use http messaging with telemetry probes... not sure any of that is a hint for what you might find in Vallone (assuming that's even the region where the URL "overload" was mentioned by Support).

[animats]

7 hours ago, Qie Niangao said:

you currently have 3 URLs of your own in Vallone, according to About Land / General / Script Info. I think these data are pooled for all land in the region with the same owner. As parcel owner, you should be able to get details of the objects, including those with URLs. A few banlined parcels in Vallone don't seem to ever populate that table (but one of those owners also has a non-banlined parcel that does populate the table -- no URLs from that one).

I have two CasperVend vendors, a dropbox, and a redelivery terminal, all working. Those expose URLs so CasperVend HQ can talk to them. But LL support claims CasperVend vendors are not a problem. If they were, we'd see far worse problems.

Moving the sim to another server raised Scripts Run from around 25% to around 80%. I suspect problems with some other sim on the same server.

This has happened before. Over weeks and months, Scripts Run decreases, I complain, LL switches the sim to a different server, things get better, slowly they get worse again, and the cycle repeats. I'm on iteration 3 of this in 18 months.

Edited May 23, 2020 by animats
typo

[Ardy Lay]

This makes me curious how simulator hosts are defended against general Internet mischief and if adding scripts with llRequestURL results in some exposure to additional mischief.

[steeljane42]

3 hours ago, animats said:

 

Moving the sim to another server raised Script Run from around 25% to around 80%. I suspect problems with some other sim on the same server.

That's what it probably is. I have my own region (full island, not a homestead), no adjacent regions (so no child agents to affect stats), no weird objects from anyone, exactly one URL (CasperSafe orb, for stats purposes mostly). Scripts run do vary quite a lot in the exactly same conditions on my side. Now I'm slowly rebuilding the whole thing, so I didn't seen less than 99% scripts run for a while, but when it was full I'd normally see between 75 and 90% scripts run, and sometimes it would go as low as 45% and not after a week of uptime, but right away after the restart. I also seen simulation time being off as well. In normal conditions it was around 1-1.3ms for my region, but sometimes after restart it would be around 2.5ms.

Restart, or two, or three always help, though. So I usually just do it and go check some event/store while it restarts, then repeat again if needed. But yeah, I always felt like it's just being unlucky and ending at the server host with some crowded event, popular club or some adult afk place.

[animats]

57 minutes ago, Ardy Lay said:

This makes me curious how simulator hosts are defended against general Internet mischief and if adding scripts with llRequestURL results in some exposure to additional mischief.

The system seems to be well protected against individual scripts using too much time. A compute-bound script does not cause trouble. Huge numbers of idle scripts are more of a problem.

[Ardy Lay]

38 minutes ago, animats said:

The system seems to be well protected against individual scripts using too much time. A compute-bound script does not cause trouble. Huge numbers of idle scripts are more of a problem.

That is not what I am wondering about.  How well is 'the system' protected from abuse from outside calling upon the HTTP service that exists to service request to the URLs given to scripts?

[animats]

Sikkima sim is still having slow or failing region crossings. Still no idea why.