Is LL putting more sims on fewer servers?

[Qie Niangao]

1 hour ago, CoffeeDujour said:

The meetings are absolutely worth going to and are not Linden fan clubs.

Regions do get moved from sim to sim for load balancing, there is the possibility that the one in the chart above had some unfortunate sim buddies for a day.

Also, to be clear.

Multiple regions run on a simulator (sim). A sim is not a section of SL, it represents many parts that may or may not be geographically close to each other on the grid.

Well, no, sorry, a "sim" is the software process that simulates a region. Multiple sims run on a single host, but it's one region per sim. (It's very likely that the co-hosted sims share common program memory, but they're still separate processes. Well, separate sets of processes, I expect.)

Anyway, when a region restarts, you're absolutely right, it typically gets rehosted. The thing is, the virtualization software should make it very rare that one (full primmed) region affects another's performance despite them being on the same host. This used to happen -- disastrously -- when Mono was brand new and regions would need so much memory when new avatars rezzed-in that the host had to swap virtual memory to disk. I'm pretty sure that isn't happening anymore, but there are other shared resources not so tidily virtualized. Network, memory bus... could be lots of stuff, but operations should be seeing that if it's happening (and assuming they're watching host performance at all).

The one thing about Joeey's statistics: Can we assume it's a full-primmed region, not a Homestead? Homesteads have more dependency between one region and another because they're actually packed in at four per host core, so their virtualization is probably more like ol' school timesharing, and a Homestead's Statistics bar is difficult to interpret.

[Fionalein]

40 minutes ago, Qie Niangao said:

Well, no, sorry, a "sim" is the software process that simulates a region.

Well in that case the term sim as used by the nefarious unknowing residents is often accidentaly right anyways.

"The sim is laggy" actually refers to the process running the region rather than the region itself...

Edited March 15, 2019 by Fionalein

[Phil Deakins]

12 minutes ago, Fionalein said:

"The sim is laggy" actualls refers to the process running the region rather than the region itself...

That's like saying that such-and-such a game is excellent, but not meaning the game you see on the screen, but meaning the programme that runs the game and creates what you see on the screen.

As is often pointed out here, language is created by users of it and not by dictionaries. 'Sim' means what the person saying it means by it, and different people mean different things, but users in general understand 'sim' to mean the 64k sqms of land section of Second Life. To LL one meaning is correct, and to users as a whole, the 64k meaning is correct.

In other words, 'sim' has more than one correct meaning.

ETA: I remember many years when a Linden explained that a region could be a sim or a group of sims - a region of them. Sim being what most of us mean by it - a 64k area of SL. So the word 'region' doesn't necessarily refer to a 64k area. At least it didn't back then.

Edited March 15, 2019 by Phil Deakins

[Qie Niangao]

1 hour ago, Phil Deakins said:

users in general understand 'sim' to mean the 64k sqms of land section of Second Life. To LL one meaning is correct, and to users as a whole, the 64k meaning is correct.

Yeah, I think it's only very rarely important to make a distinction between the region and the sim. I do think it's confusing, however, to conflate "sim" and "host" which is why I got all pedantic about it.

Another distinction that's only rarely significant: folks often use "avatar" when they really mean what's technically called an "agent." An avatar is really only the collection of stuff that can be animated and appears visually on the screen, whereas an "agent" is the sim's underlying representation of that collection. This difference may seem abstract, but it's a distinction that's sometimes important: when I login an agent is instantiated for my login session; that agent will be visually manifested by some avatar -- and then I can completely change the avatar but the underlying agent remains constant. Sometimes one needs to refer to that thing that remains constant in this scenario, separately from that thing which changes.

[Phil Deakins]

And at the end of the day, words mean what the user intends them to mean. With the words under discussion, it's usually very obvious what the user means, except for the word 'host' when it's about SL servers. It's not a commonly used word for SL. You were right to explain that one, Qie.

[Joeey Aura]

3 hours ago, Qie Niangao said:

The thing is, the virtualization software should make it very rare that one (full primmed) region affects another's performance despite them being on the same host. This used to happen -- disastrously -- when Mono was brand new and regions would need so much memory when new avatars rezzed-in that the host had to swap virtual memory to disk. I'm pretty sure that isn't happening anymore, but there are other shared resources not so tidily virtualized. Network, memory bus... could be lots of stuff, but operations should be seeing that if it's happening (and assuming they're watching host performance at all).

yes i have done tests on this as well, as i once suspected that other regions would lag mine, So i made a script to get the host of a region and flew around the mainland to find the 4 others that share the same host. with the help of some friends positioned on those regions we looked at the stats (and some times greatly affected those stats) lagging 3 of the regions scripts times, had no noticeable effect on the 4th. So LL has done a good job at separating them as far as script time goes.

3 hours ago, Qie Niangao said:

The one thing about Joeey's statistics: Can we assume it's a full-primmed region, not a Homestead? Homesteads have more dependency between one region and another because they're actually packed in at four per host core, so their virtualization is probably more like ol' school timesharing, and a Homestead's Statistics bar is difficult to interpret.

yes its a full region not a homestead. 

 

[Wulfie Reanimator]

21 minutes ago, Joeey Aura said:

with the help of some friends positioned on those regions we looked at the stats (and some times greatly affected those stats) lagging 3 of the regions scripts times, had no noticeable effect on the 4th. So LL has done a good job at separating them as far as script time goes.

You probably (should) already know this, but scripts are the last thing to get any processing time per frame, and only the spare time allotted per frame after everything else. And since sims have their dedicated core, it's pretty hard (if not impossible) for one sim to slow down another.

What you could do to test that is to break something more important, like physics. While scripts alone can't crash a sim, physics certainly can. Even if you don't (and shouldn't) go that far, you'll observe all kinds of other lag in other areas.

[Aishagain]

I'm quite interested by the knowledge/ignorance of LL's sim-hosting procedures in SL shown on this thread and it is significant that homesteads, since they are sharing cores, suffer from more operational cross-talk that mainland full regions or private estates.  I quite frequently have to restart my homestead after rolling restarts, due (I assume) to being moved onto a busy core.  I usually find that one restart is sufficient to move us to a less densely loaded core.  Throughout, however, our region's script run these days never goes above 65%, and is occasionally as low as 25% on a "bad" core.  Previously, and with no change in our homestead's script load or number, we had routinely 99%+ script run: if it dropped below 90% I knew I needed to restart the region.

That being said I do observe similar fluctuations and massively lower script-run percentages on a Mainland full region I help to administer, and these are independent of TP's or login/out either on that sim or on neighbouring (geographically) sims.  I have spent several hours doing these observations in a very non-technical way, simply watching the statistics in my own viewer.  The "child agent" issue is also very observable.

All this being said does not change the observable fact that something has changed within SL to cause this massive reduction in script-run % on nearly all regions.  It is a puzzlement to me as to why some regions are still showing 95+% script-run despite the obvious changes elsewhere.

Edited March 16, 2019 by Aishagain

[steeljane42]

I saw my region's performance being low at times too, usually after the rolling restarts as well. In general I try to keep scripts run around 55-60% and sometimes after the restarts (and not during those, when I assume some things could be a bit unstable) I noticed it dropped to 30%. Same everything else: 1 person in the region, same objects, same script count, no noticeable difference in net, physics, simulation or agent times. It's a standalone island, so no child agents either. Manual restart is enough to get it back to the normal state.

I also did assume it's because LL are saving on hardware before moving to cloud, it would make little sense to mass change/upgrade servers at this point, so I guess only replacements that happen are for hardware that is completely broken. Just speculation, of course, but I really doubt LL are looking for mass new servers upgrades/purchases at this point, they'd have to sell them to someone (and lose bunch of money on it) after migration to cloud is done.

Edited March 16, 2019 by steeljane42
late typos fix

[Gunner Grun]

I have for years restarted my sim a few hours after rolling restarts.

[Qie Niangao]

On 3/15/2019 at 1:48 PM, Aishagain said:

It is a puzzlement to me as to why some regions are still showing 95+% script-run despite the obvious changes elsewhere.

FWIW, I wandered around a bunch of Mainland and found a lot of regions with 95% scripts run each frame.  Every one I found had these facts in common: they had fewer than 4000 scripts in the region and did not have any Pathfinding characters. In fact, I didn't find a single region with even one Pathfinding character that wasn't in script lag, even if there were very few scripts in the region. I know it's not supposed to work that way, though, so I'd be interested if anybody knows an exception.

I did find exceptions in the other direction, though: full-primmed sims with fewer than 4000 scripts, no Pathfinding, few agents, few script events even, and still no spare time. Somehow those few scripts processing few events manage to use up close to 20ms each frame. (Scripts in tight loops, maybe?)

[Coffee Pancake]

Moving away from an always on always 100% simulated grid to having regions spin up and down depending on actual use, makes a lot of sense and if we want to see any major downward push in land prices is pretty much the only way we will get it. Memory, disk use or bandwidth aren't pressure points anymore, but CPU cycles per core are.

Edited March 17, 2019 by CoffeeDujour

[animats]

On 3/15/2019 at 2:07 AM, CoffeeDujour said:

The meetings are absolutely worth going to and are not Linden fan clubs.

Regions do get moved from sim to sim for load balancing, there is the possibility that the one in the chart above had some unfortunate sim buddies for a day.

Also, to be clear.

Multiple regions run on a simulator (sim). A sim is not a section of SL, it represents many parts that may or may not be geographically close to each other on the grid.

As I understand it, a sim runs in one process. Multiple processes may be run on the same computer. Usually about one per CPU core.

[animats]

On 3/17/2019 at 4:20 AM, Qie Niangao said:

FWIW, I wandered around a bunch of Mainland and found a lot of regions with 95% scripts run each frame.  Every one I found had these facts in common: they had fewer than 4000 scripts in the region and did not have any Pathfinding characters. In fact, I didn't find a single region with even one Pathfinding character that wasn't in script lag, even if there were very few scripts in the region. I know it's not supposed to work that way, though, so I'd be interested if anybody knows an exception.

I did find exceptions in the other direction, though: full-primmed sims with fewer than 4000 scripts, no Pathfinding, few agents, few script events even, and still no spare time. Somehow those few scripts processing few events manage to use up close to 20ms each frame. (Scripts in tight loops, maybe?)

Turns out you can look at the script usage of other people's objects. In Firestorm, use World->Area Search to get a list of objects, then right click and get "Script Info". So there's no privacy issue about this. It's just a pain to have to manually look at every object.

Scripts that are doing nothing seem to use about 0.001 to 0.002 milliseconds per frame. That's not much, but if you have 4000 scripts in a sim, there goes 4 to 8 ms of the available script time for doing nothing. So it's possible to waste about half the sim script capacity with idle scripts.

Typical entries from my home sim, Vallone, none very exciting:

• Script info: 'Steps': [1/1] running scripts, 64 KB allowed memory size limit, 0.002022 ms of CPU time consumed.
• Script info: 'Door': [1/1] running scripts, 16 KB allowed memory size limit, 0.001229 ms of CPU time consumed.
• Script info: 'Modern Under Construction Sign': [0/0] running scripts, 0 KB allowed memory size limit, 0.000000 ms of CPU time consumed.
• Script info: 'Smooth Rider - V2.5.2 (REVIEW)': [8/8] running scripts, 464 KB allowed memory size limit, 0.005806 ms of CPU time consumed.
• Script info: '3C Chrome Bench': [4/4] running scripts, 64 KB allowed memory size limit, 0.003121 ms of CPU time consumed.

"Steps" is the moving component of one of my escalators, in operation. "Door" is a stock door script, and nobody is using the door right now. "Modern Under Construction Sign" has no scripts. "Smooth Rider" is one of my bikes, parked. Lots of scripts on standby. If I get on the bike, it disappears from the Area Search list because it's now tied to an avatar and comes out of avatar time. Pathfinding characters are also accounted for separately. "3C Chrome Bench" is a Linden street bench. It's sittable, but as far as I can tell, it does nothing else. Wonder what the 4 scripts do.

I can see objects from others, but I'm not giving their details here, per policy.

I'm not finding any big script time users.

I did find a slow sim with thirty "'M-268 Predator Turret" units installed. (They shoot you if you get too close to the parcel. Even on Linden water.) But they are only using 0.0047ms of CPU each.

 

[Wulfie Reanimator]

An important thing to remember about Script Info, is that the CPU time shown is an average over the past (up to) 30 minutes, or less if the script has been rezzed/restarted. This has two side-effects.

1. The CPU time will not reflect the script's current usage. If the CPU time spikes after 30 minutes of idle time, you won't see a great increase especially if it isn't sustained.
2. Likewise, if the object is recently rezzed/restarted, the CPU time shown will be quite high (it can even be as high as 1-5 ms right after a reset) before it "calms down" and averages out.
   (#2 is hilarious and terrifying in cases where a script works by resetting itself. For example, I've seen full-auto guns that have 3-4 scripts with infinite rez loops in them, which are turned on/off by a central script. The script time while the gun is firing is ridiculous and anybody doing that should stop.)

[Aishagain]

Qie

What puzzles me is that my homestead has over 16ms free time, no pathfinding  and yet script-run never exceeds 65% now

[Israel Schnute]

Still doesn't explain it.  My sim went from terrible in Dec/Jan to running pretty well at the beginning of March, to back to terrible last week, to okay today.  Nothing in the sim or the region itself has changed.  I like that I can watch the performance visually by even going to the adjacent sims, negating any concern it is client side lag.  I have a very old script in all of my trains which was the original ones.  I can gauge sim performance by if they lag or not, or even if I had a restart.

If it doesn't have anything to do with shared sims on the host, then performance is completely random.   

[Qie Niangao]

23 minutes ago, Aishagain said:

Qie

What puzzles me is that my homestead has over 16ms free time, no pathfinding  and yet script-run never exceeds 65% now

I freely admit that I'm easily confused by the statistics bar on Homesteads. What I think is happening is that those 16 ms aren't spare for your use, but rather they're time sliced for other Homesteads on the same core. If that's correct, your share of a 22.7 ms frame is about 5.7 ms, leaving like 17 ms that the other Homesteads must be able to use if they need it. (Or maybe it's supposed to be reserved for them whether they need it or not; as I said, Homesteads hold many mysteries for me.)

[Edit: Just in case it sounds as if I'm minimizing the apparent decline in performance, grid-wide: Not at all. I'm pretty convinced that something is going wrong. I'm just not making much progress as a user, tracking down what the heck is going wrong.]

Edited March 20, 2019 by Qie Niangao

[animats]

8 hours ago, Qie Niangao said:

[Edit: Just in case it sounds as if I'm minimizing the apparent decline in performance, grid-wide: Not at all. I'm pretty convinced that something is going wrong. I'm just not making much progress as a user, tracking down what the heck is going wrong.]

Me either. I'm seeing slow sims that don't have any obvious cause.

Huge numbers of idle scripts can stall out a sim all by themselves. Here's a particularly bad case.

Corchalo sim stats

Corchalo is a "double prim" sim in Zindra. About half the sim is Linden land, with trees. The other half is stacked up skyboxes. At least three clubs. 14000 scripts! Now that's overdoing it. About 5% to 15% of scripts are getting some time on each frame.

Maybe LL needs to restrict skyboxes on mainland. There's no real point to skyboxes on mainland, anyway; anyone who goes there does so with a teleport. Skyboxes let a parcel owner hog sim resources. Private regions, fine; the region owner can hash that out with their tenants. Most of the big estate landlords like Lionheart forbid skyboxes. They'e also forbidden over Linden Homes.

Maybe charge double LI above the ban line level, 128m above ground.

[Qie Niangao]

I thought "Corchalo" sounded familiar, and yeah, it's long been a basket case.

There's Agent and Simulation Time each frame corresponding to the 20-some agents (plus child agents) in the region, so there's less time available for scripts. Most of those agents (and, I suspect, most of the lag) is just one little parcel in the southeast corner of the region with a tiny club in the sky, full of exceptionally fake-looking "lesbians" arrayed in suspiciously orderly fashion. I couldn't explore further because my "female" alt was rejected by their no-men-allowed security device. Besides all the rest of the junk that parcel scatters around the sky, it does some temp-rezzing of scripted objects in at least one location, so spikes in dilation are to be expected.

There are plenty of scripts, and not idle ones either: check the Script Events-per-second. I looked around and found what seemed like the usual density of scripts on other parcels, so I expect the agents on that corner are wearing more than the rest of the sim combined.

My hunch is that this "club" is really just an excuse to keep a flock of heavily over-scripted bots, all for the very purpose of generating lag to torment some other, larger landowner(s) in the region.

Anyway, I definitely disagree about limiting skyboxes because I am 100% sure that those very expensive double-prim parcels would be using the same amount of resources without skyboxes if they were forced to concentrate it all on the ground -- and then the viewer-side lag would be impenetrable. I do wish skyboxes were banned below 800m or so, but above that, they serve to spread out the geometry, and are of significant value to landowners. The Lab does not want to diminish the customer-perceived value of the Land product, whatever else they do.

[Israel Schnute]

" Maybe LL needs to restrict skyboxes on mainland. There's no real point to skyboxes on mainland, anyway; anyone who goes there does so with a teleport. "

 

Yes there is a point.  It allows  the "heart" people to ruin the boating and flying regions of SL for excessive profits.

[MBeatrix]

18 hours ago, animats said:

[...] Maybe LL needs to restrict skyboxes on mainland. There's no real point to skyboxes on mainland, anyway; anyone who goes there does so with a teleport. Skyboxes let a parcel owner hog sim resources. Private regions, fine; the region owner can hash that out with their tenants. Most of the big estate landlords like Lionheart forbid skyboxes. They'e also forbidden over Linden Homes.

Maybe charge double LI above the ban line level, 128m above ground.

I would never agree with that for the simple reason that at my 1024 sqm parcel I have several different buildings and windlight settings, distributed by sky platforms (none is below 2000 m altitude) plus the building on the ground. If you want to limit skyboxes, then why not simply limit the sky altitude itself?

I would agree with a minimum altitude allowed for skyboxes, though. I know it can sound funny flying an airplane against one, but it really isn't for the pilot.

[EDIT] I don't get what you mean with "There's no real point to skyboxes on mainland, anyway; anyone who goes there does so with a teleport". Are you forgetting we can set a landing point? (which I have set to the ground at my parcel where I have sky platforms)

Edited March 21, 2019 by MBeatrix
adding more

[MBeatrix]

4 hours ago, Israel Schnute said:

[...] It allows  the "heart" people to ruin the boating and flying regions of SL for excessive profits.

I like them as much as you do, and I probably sail and fly around more than you do, but LL needs those "excessive profits" not only to keep stuff running as also to make improvements to it.

[Blush Bravin]

On 3/20/2019 at 9:50 PM, animats said:

They'e also forbidden over Linden Homes.

That is changing with the new Premium Homes. Skyboxes are allowed at 2000 or higher. 

[LittleMe Jewell]

On 3/20/2019 at 8:50 PM, animats said:

Maybe LL needs to restrict skyboxes on mainland. There's no real point to skyboxes on mainland, anyway; anyone who goes there does so with a teleport. Skyboxes let a parcel owner hog sim resources.

Me living in a skybox far above the ground, with just a nice park being at ground level, does not have me using any more resources than if my home was actually on the ground with that small park.