Kama Center down, Vallone overloaded worse than ever

[animats]

My home sim, Vallone, is overloaded worse than ever, and Kama Center, the sim next door, went offline 15 minutes ago and hasn't come back. I know there's a new server installation today. Did that not go well?

[Mazidox Linden]

Hi there Animats,

I can't speak to your concerns regarding overloading, but Kama Center has been up for over an hour. Have you double checked what you're seeing at all?

[Qie Niangao]

This isn't looking good. Tussock is on Second Life RC LeTigre 19.08.23.530380 and is periodically falling into states where the entire frame is devoted to Pump IO.

ETA: Oh, Vallone is interesting; it's running the anonymized "Second Life Server 2019-08-23T17:54:14.530353" and is back in that state where most of the Script Time is showing as "Sleep Time" and less than 25% of scripts are run... we've seen that on this sim before, right? (So, not surprisingly, a tiny share of Pathfinding characters get updated.)

Edited August 28, 2019 by Qie Niangao

[Qie Niangao]

Further on Tussock: I haven't seen the Pump IO anomaly again for a while. The region is in the condition where it burns over 19 msec Script Time but still can't get through the paltry 3300 scripts on this full sim. We're growing accustomed to seeing this on some random share of restarts, luck-of-the-draw, and other restarts that give a solid 6 msec or more Spare Time and 99% scripts run -- with nothing changing in the region from one of those restarts to the next.

[animats]

3 hours ago, Qie Niangao said:

This isn't looking good. Tussock is on Second Life RC LeTigre 19.08.23.530380 and is periodically falling into states where the entire frame is devoted to Pump IO.

ETA: Oh, Vallone is interesting; it's running the anonymized "Second Life Server 2019-08-23T17:54:14.530353" and is back in that state where most of the Script Time is showing as "Sleep Time" and less than 25% of scripts are run... we've seen that on this sim before, right? (So, not surprisingly, a tiny share of Pathfinding characters get updated.)

Yes, what's with all that "Sleep Time"? You see that on "Homestead" sims, which are throttled, but this is a "mainland" sim.

Four months ago, this sim had spare time. Not happy about the continual reduction in performance.

[animats]

4 hours ago, Mazidox Linden said:

Hi there Animats,

I can't speak to your concerns regarding overloading, but Kama Center has been up for over an hour. Have you double checked what you're seeing at all?

Yes. See above, about "sleep time".

Just asked support for a region restart. I'm the only avi there right now, so I'd like to get a restart ASAP.

Edited August 28, 2019 by animats

[animats]

Still waiting for a restart. Ticket filed earlier today.

[Whirly Fizzle]

The fastest way to request a mainland region restart  is to file your support ticket under Land & Region -> Region offline, even if the region is not offline, as long as you have a valid reason for requesting a region restart.

The "Region Offline" tickets are fast tracked.
Usually LL restart the region in less then an hour.

 

[animats]

I got an immediate restart via live chat. It helped a little; there's still unexpected "sleep time", but about half as much.

30 seconds after sim restart.

I then tried deleting my two pathfinding characters. That cut the "Physics time" to 0.6ms, as expected, but didn't change the "sleep time" , and the "script time" went up.

 

Edited August 29, 2019 by animats

[animats]

Nothing I can do helped. I deleted all my NPCs, both the old-style pathfinding ones and the one on the new system. I pushed the red button that stops the escalators. Still at 29% scripts run.

[Alyona Su]

I really hate to say it, but maybe it's time to consider a move to a new place? I wouldn't have put up with what you are putting up with, so I have to salute you on your perseverance!

[animats]

I feel that way

4 hours ago, Alyona Su said:

I really hate to say it, but maybe it's time to consider a move to a new place?

I've considered it, but I own four parcels on a corner in a city, which is hard to get. Besides, script overload is the new normal in SL.

Support restarted the region. That didn't help. They moved it to another server. That helped, and the "sleep time" went to zero. Still overloaded, though. And this morning, big sleep time is back. I dunno.

It's almost like this mainland sim is being set to "Homestead" mode, where there's forced sleep time to allow two sims to share one CPU.

Vallone's big problem is too many scripts not doing anything. There's little avatar activity but about 7000 scripts. The new "Cake" server release is supposed to help with that.

In some ways, the overload is useful. Because I do scripting in an overloaded area, my bikes, escalators, and now NPCs are all coded to be well-behaved under script overload.

[Profaitchikenz Haiku]

(Reply to Alyona) There is no way the be certain that the script problem won't pop up at another place. It's come and gone at my home parcel several times. Moving a complex build can take several days, I think we should be pushing for a general resolution to this problem.

Edited August 30, 2019 by Profaitchikenz Haiku

[Alyona Su]

9 minutes ago, animats said:

In some ways, the overload is useful. Because I do scripting in an overloaded area, my bikes, escalators, and now NPCs are all coded to be well-behaved under script overload.

I didn't think of that aspect. That's smart! Build it in worst-case scenario and it can only be better than what you've got when building it! This is why I'm not smart enough for the deep-scripting stuff.

[animats]

Just now, Profaitchikenz Haiku said:

There is no way the be certain that the script problem won't pop up at another place. It's come and gone at my home parcel several times. Moving a complex build can take several days, I think we should be pushing for a general resolution to this problem.

Yes. That's why I keep pushing for fixing the script execution system so that idle scripts with no events have zero overhead. Each sit script and color changer currently uses a little time on each frame, and it adds up. Around 3000 to 6000 scripts, that alone eats all the sim time. Several people, including me, have now verified this in empty sims.

If you need to see how much load an object is producing, go to Arowana sim. This is a Linden test sim which has zero scripts, allows rezzing, and has 20 minute autoreturn. Useful for when you want clean statistics for your own objects. I've taken most of my scripted objects there to make sure they're not using time when idling. (Some vehicles continue to run fast timers in scripts while parked. Don't do that.)

[Qie Niangao]

I happened upon another region with nearly all the Script Time given to Sleep Time. "RIOT DISTRICT" is a full region, not Homestead (actually, one of those 30k extra-prim-packed regions). To be fair, this one has over 12,000 active scripts, so it's hardly surprising it only gets round to running about a fifth of them each frame, but still of its 15.1 msec Script Time, 13.7 msec is reported as "Sleep Time".

[Lucia Nightfire]

Having to move away from a sim because of other land owner's content consuming more % of available script timing than % of land they own in the region and then after finding land in another sim with available timing only to have the same problem happen again after some clueless moron moves in next door and rezzes a breedable farm or whatever else that consumes all available script timing is something that has plagued many land owners in SL for years.

It only got worse when LL gave mainland regions 50% more prims, which is essentially 50% more potential hosts for scripts.

It shouldn't have to be this way.

LL should have implemented fair script resource enforcement of some kind just like there is fair prim use enforcement.

Another thing LL could explore with Project ArcTan is make content cost exponentially more land impact the more scripts it contains. Right now there is only a less than 0.5 LI per scripted link hit. If there was something like a Power(total # of scripts, 2) and/or Power(total script memory / 65536, 2) factor included with the Max(physics cost, streaming cost, server cost) calc you'd start to see more efficient scripting of content because the cost/penalty will actually demand it, but ATM the cost/penalty for script count/memory use is a complete joke.

For those that don't know what the math is, it would look like this if based on script count:

#Scripts | Land impact

0	0
1	1
2	4
3	9
4	16
5	25
6	36
7	49
8	64
9	81
10	100

Edited August 31, 2019 by Lucia Nightfire

[Lucia Nightfire]

Yes a power of 2 might seem kind of high, but as long as it is something effective, it can also be slightly lower.

It would also warrant LL to give us more efficient script alternatives than ones that force people to use multiple scripts or scripts in multiple links.

[MBeatrix]

5 hours ago, Lucia Nightfire said:

[...] LL should have implemented fair script resource enforcement of some kind just like there is fair prim use enforcement. [...]

Not a practical thing to to, as it would almost kill many SL activities, like using vehicles, sales events, clubbing, etc.

Scripts are what make SL tick. If we can't use the scripts we need, then what's the point of being in SL at all? Just for sight seeing?

I'm sorry, but I can't agree with any of that.

Edited August 31, 2019 by MBeatrix

[Qie Niangao]

The first order of business needs to be to reduce the unpredictable script effects we're seeing. This needs to be first because it will not play well on the cloud for regions to mysteriously burn ridiculously excessive script time every random number of restarts. Whatever they are doing when they're not doing what they're supposed to be doing is going to be a real expense in the cloud, not merely hidden in crappy performance for a share of randomly chosen regions running on dedicated hardware.

Next, it would be nice if scripts themselves were more efficient, which seems to be what Rider is working on now. This may include improvements to the scheduler.

Meanwhile, yeah, there's ARCTAN, which may be an opportunity. I think I'd start with helping Estates manage their own script load by having control of how many scripts can be attached to an avatar that tries to teleport into the region. This doesn't prevent abuse if a committed script abuser loads up after arrival, but it would sure help remind folks that their attached scripts might be detached if they want into an event venue for example.

Unattached scripts are already something Estates can manage (or could, if there were predictable load effects); the big problem arises on Mainland. I'm not very confident of the proposed exponential LI contribution of scripts. I'd expect scripters to get around that by simply distributing scripts across more objects -- and doing more resource-intensive inter-object communications among the scripts. Also, unless this comes with another big boost in LI capacity, folks will see it as a "taking" and however justified in theory, in practice that's very bad business now.

[Profaitchikenz Haiku]

2 hours ago, MBeatrix said:

Scripts are what make SL tick. If we can't use the scripts we need, then what's the point of being in SL at all? Just for sight seeing?

Agreed.

 

But what Animats and some others are pointing out is that 'pointless' scripts consume far too much resource, these are idle scripts, prims with a particle-setting script or sitpos script still in them, scripts with event-handlers for events they will never receive. There needs to be some mechanism to avoid them consuming resources.

Edited August 31, 2019 by Profaitchikenz Haiku

[MBeatrix]

44 minutes ago, Profaitchikenz Haiku said:

Agreed.

 

But what Animats and some others are pointing out is that 'pointless' scripts consume far too much resource, these are idle scripts, prims with a particle-setting script or sitpos script still in them, scripts with event-handlers for events they will never receive. There needs to be some mechanism to avoid them consuming resources.

I know, I read them. But maybe I should remind everyone that the "scripts issue" began like it was overnight, about one year ago. One day all was normal at my home sim, and on the next bringing up a menu to change the radio station was taking ages. And until recently, LL let it go like nothing wrong was going on, which had some people speculating about the number of regions per hardware server.

As I'm sure you know, many creators can't afford paying for a whole region, so they do whatever they do at their small parcels around SL. Should we tell them "hey, you can't pay for your own sim, so you better stop messing the place where your store is with your scripts"?

I'm very comfortable saying this because the scripts usage at my parcels is quite low, and I don't sell anything — I give it away for free to those people I care about.

This issue was created by LL, and bringing up the idea of limiting scripts accordingly to the amount of land you own is mostly unfair. It's up to LL to fix what they broke, and not limiting scripts usage — it definitely would not help "keeping SL's economy healthy".

[Profaitchikenz Haiku]

1 hour ago, MBeatrix said:

But maybe I should remind everyone that the "scripts issue" began like it was overnight, about one year ago. One day all was normal at my home sim, and on the next bringing up a menu to change the radio station was taking ages

This was my experience exactly, in February the train on my parcel was juddering along, there had been no gradual deterioration or any warnings that things were deteriorating. I am concerned that this has been given so little attention by the Lindens who attend the server user group meeting. (Or they actually know full well what was the reason but for various reasons cannot discuss it with us). My worry is that it is a step-change that cannot be reversed, such as shifting to different servers, so the challenge now is to find a way to restore performance on a new and relatively unknown system.

[Qie Niangao]

52 minutes ago, MBeatrix said:

I know, I read them. But maybe I should remind everyone that the "scripts issue" began like it was overnight, about one year ago. One day all was normal at my home sim, and on the next bringing up a menu to change the radio station was taking ages. And until recently, LL let it go like nothing wrong was going on, which had some people speculating about the number of regions per hardware server. [...]

Yes, this corresponds to the "scripts issue" about which I've been most concerned. I think we should recognize that it's not the only "scripts issue" that affects SL. Some issues have been around a long time and are pretty serious.

Among the issues, In no particular order, there's:

1. Griefing by loading-up with attached scripts after arrival in a region
2. Arriving with too many attached scripts 
3. Rezzing too many unattached scripts
4. Scripts using more resources than they should, per script:

• 4a. because they're poorly written
• 4b. because all sims run scripts less efficiently than they should
• 4c. because the sim is in some unpredictable "bad at scripts" state

As you rightly point out, the Lindens are finally taking some of these issues seriously, especially 4b and maybe 4c. In another thread, Oz mentioned that the Lab aims to collect more sim data (as part of the same initiative that will hide Release Channel names) which might have made it easier for them to more quickly recognize the recent "hell in a handbasket" effect on script performance.

Again, all these are going to be relevant to business success in moving to the cloud.

As I understand @Lucia Nightfire's proposal, it would be most effective addressing #3.* And ARCTAN is going to change calculation of LI (and avatar complexity -- see #2), so if there are ever to be penalties for too many scripts, now would be the time to get it right.

________________
*Credit where credit's due: Lucy has thought about #1 more deeply than I'll ever even be able to understand.

[animats]

For now, Vallone is back to its usual mediocre level of performance.  About 45% of scripts running per frame. I can work with that.

I look forward to the "Cake" server release, which is supposed to address the idle script problem.