Jump to content
animats

Is LL putting more sims on fewer servers?

Recommended Posts

Posted (edited)

I'm seeing many sims in script overload that never used to be overloaded. Looking at Advanced->Performance Tools->Statistics Bar, I'm seeing "Scripts Run" at 40% and "Spare Time" near 0. About half of the sims I've visited are in script overload like that. Even the Builders Brewery sandbox is in script overload, with one house rezzed. My own home sim, Vallone, has been overloaded for several days. I've been looking for the load and can't find it. Turned off my pathfinding characters and escalators. Most of the sim is vacant, and I've checked mine and the neighbors' stuff for obvious problems. Not seeing it.

Is LL putting more sims on fewer servers to save money? I'm getting suspicious.

(Obvious stuff: could it be my avi? No. I went to a sim which has no scripts, and with my avi there, there are zero script events per second.)

Edited by animats
  • Like 1

Share this post


Link to post
Share on other sites
2 minutes ago, Wulfie Reanimator said:

My wild conspiracy theory is that this is totally related to the migration to AWS. /mild sarcasm

Might not be a joke. With the migration to AWS coming, maintaining and replacing the existing servers may not be happening.

Share this post


Link to post
Share on other sites
13 minutes ago, animats said:

Might not be a joke. With the migration to AWS coming, maintaining and replacing the existing servers may not be happening.

Hence why only mild. I can't prove one way or another and I'm not knowledgeable enough to look into it or make an educated guess.

Share this post


Link to post
Share on other sites
Posted (edited)
39 minutes ago, Wulfie Reanimator said:

Hence why only mild. I can't prove one way or another and I'm not knowledgeable enough to look into it or make an educated guess.

Three experienced creators I know have also noticed this. Let's hear from others.

I've been testing my own stuff and my avi in a little-used Linden region that has zero script events per second. That lets me see how much load each object generates. None of my stuff is generating script events when they shouldn't. So it's not me.

GridSurvey has a list of which sims are on which IP addresses, but it's way out of date. Could someone get the scan rerun? Then we can tell if there are too many sims on a server.

Edited by animats

Share this post


Link to post
Share on other sites

I started looking at this late last week when I noticed the steam engine and carriage in our parcel was crawling along at half the usual speed. Scripts run was around 38%. I've been used to seeing 15%. However, as this is a private island where we rent a parcel, I'm not sure if it's going to be subject to the same policies as mainland?

Share this post


Link to post
Share on other sites
18 minutes ago, Profaitchikenz Haiku said:

I started looking at this late last week when I noticed the steam engine and carriage in our parcel was crawling along at half the usual speed. Scripts run was around 38%. I've been used to seeing 15%. However, as this is a private island where we rent a parcel, I'm not sure if it's going to be subject to the same policies as mainland?

"Scripts run was around 38%. I've been used to seeing 15%". "100% scripts run" means all scripts got enough CPU time. That's the normal situation. There should be "spare time" in each frame after 100% of scripts have run, to cope with sudden needs for CPU resources.

If "scripts run" is less than 100%, time-critical scripted objects start having problems. If a sim is at "15% scripts run", everything will be very sluggish and some things will break.

The estate manager for the island can use the estate manager tools to get a list of the scripts using the most time. LL support can do this for mainland sims.

The question is, is the script usage higher than usual, or is less CPU time available for each sim program?

Share this post


Link to post
Share on other sites

Handing out 50% more available prims without implementing any form of fair script use enforcement policy didn't help things either.

Share this post


Link to post
Share on other sites
4 hours ago, animats said:

Even the Builders Brewery sandbox is in script overload, with one house rezzed.

The BB Sandbox has a lot more has "one house rezzed". There are currently ~4650 active scripts, and looking at the performance I would say it's about the same as I would expect (oddly BB's sandbox seems a little higher than it should be but only a little).

1 hour ago, animats said:

The question is, is the script usage higher than usual, or is less CPU time available for each sim program?

It seems the script usage is just higher than it used to be. Unfortunately, with mesh becoming more and more common, people tend to have more free prims. What do they do with those free prims? Use them to rez more scripted objects. 

Share this post


Link to post
Share on other sites
5 hours ago, animats said:

I'm seeing many sims in script overload that never used to be overloaded. Looking at Advanced->Performance Tools->Statistics Bar, I'm seeing "Scripts Run" at 40% and "Spare Time" near 0. About half of the sims I've visited are in script overload like that. Even the Builders Brewery sandbox is in script overload, with one house rezzed. My own home sim, Vallone, has been overloaded for several days. I've been looking for the load and can't find it. Turned off my pathfinding characters and escalators. Most of the sim is vacant, and I've checked mine and the neighbors' stuff for obvious problems. Not seeing it.

Is LL putting more sims on fewer servers to save money? I'm getting suspicious.

(Obvious stuff: could it be my avi? No. I went to a sim which has no scripts, and with my avi there, there are zero script events per second.)

You can see exactly how much time is available for scripts in the statistics panel. It hasn't changed for full regions recently. The only way something like this could be a server issue independent of the scripts is if something is making the same scripts take up more time than they would usually use.

As far as Vallone, I'm currently standing in one of the "empty" lots and it's showing 561 land impact used - I assume they're just up in the air.

  • Like 1

Share this post


Link to post
Share on other sites

Just a thought. Recently there was a longish thread (mine) called something like "world is a blur - are we there yet? " (Cloudfront and web not working right now and I only get 404 errors :D)  about texture loading.  That was never solved but is also a NEW (for me and some others anyway) problem. Doesn't seem to be sim specific and I have a very good connection and typically no problems. Wondering if the two things could be related.   

I am a "low script" gal and have on very basic scripts (like doors opening and furniture animation) in my sim so wouldn't really notice any script problems.   

Hopefully this will post. Taking a copy in case it doesn't :D

Share this post


Link to post
Share on other sites
Posted (edited)

In response to the OP, yes, LL is putting more regions on NEW servers.  There has been a rolling replacement of older "less capable" servers for some time now and while I am not sure just how many full regions we now get per server it HAS increased.  We've known for a while (or at least I have) that more mainland full regions are packed on a server than private estate full regions and needless to say many more Homesteads are crammed onto single servers than was the case a few years ago.  I'm sure there's an LL Wiki entry that gives the details.

Having said all that, I too have noticed a big drop in script-run percentage on my home region, from over 99% to around 40-60% some time ago NOT concurrent with the extra-prim allowance roll-out.  As to WHY, I don't know, it MAY be the increased density of regions, it MAY be something else completely, but with all the other issues (child-agents is just one of several performance hits), it is hard to know.

I would suspect that one of the "internal changes" in a software version change has introduced a new calculation for script-resource use and it is this that has so drastically changed the run-time calculations.  I was always a little surprised that the number of scripts we ran on out Homestead had so little effect, despite us being very careful with out script-use!

Edited by Aishagain

Share this post


Link to post
Share on other sites
8 hours ago, Theresa Tennyson said:

You can see exactly how much time is available for scripts in the statistics panel. It hasn't changed for full regions recently. The only way something like this could be a server issue independent of the scripts is if something is making the same scripts take up more time than they would usually use.

As far as Vallone, I'm currently standing in one of the "empty" lots and it's showing 561 land impact used - I assume they're just up in the air.

That's clock time, not CPU time. If the server computer has more sims than CPUs, or the CPUs are slower, or something server side is using a lot of CPU time, the same scripts could appear to use more CPU time.

We could use a scripting benchmark to measure this. Does anyone have one that's been run in the past and data recorded?

Share this post


Link to post
Share on other sites

Some mainland regions with Scripts Run < 50%:

  • Molymar. Scripts Run 32% - 37% and steady. Server 19.01.25.523656. (Trouble ticket submitted.)
  • Vallone: Scripts Run 38% - 45%. Server RC BlueSteel 19.02.27524. (Trouble ticket submitted.)
  • Charlesville: Scripts Run 38% - 55%. Server 19.01.25.523656.

For comparison, some nearby sims not in overload.

  • Kama Center. Scripts Run 100% Server RC LeTigre 19.02.27.52482
  • Bronlen. Scripts Run 100%. Server 19.01.25.523656. (Mostly a park.)
  • Burns. Scripts Run 100%. Server 19.01.25.523656.
  • Calleta. Scripts Run 97%. Server RC Magnum 19.01.25.5236. (A classic heavily loaded sim, running fine.)

So, doesn't seem to be related to server version

I'm listing mostly sims I've had occasion to check before and have seen running well.

 

  • Thanks 1

Share this post


Link to post
Share on other sites

Oz Linden insisted today at Server User Group that LL is not putting more sims on fewer servers. But there was a consensus from many at the meeting that script load has increased recently. Nobody knows why.

steadyload.png.3614d2293277254c274792d8185ef55a.png

Server statistics, Vallone sim. No spare time, 60% of scripts running.

I've shut down my escalators and pathfinding animesh. I have the only avi in the sim. My own avi has nothing running; I've tested in an empty region. Nothing much is happening in this sim; it's more than half vacant, and I've looked at all the skyboxes. There's a big and very steady load coming from somewhere. One parcel has some decorative vehicles flying around, and I've been trying to reach the owner. But there's no reason their stuff should use resources like this unless it's very badly coded.

I've seen strange steady loads in other sims. I have support tickets in. Waiting on LL for responses since last week.

Let me know if you see a region with a pattern like this - constant, steady script load for no good reason.

Share this post


Link to post
Share on other sites
Posted (edited)

I am glad you posted this.  My city on the Atoll is nearing the 10 year mark.  My "new" section is over 3-4 years old.  For several months I have had intense lag in a section of the sim.  My automated trains started crawling.  This resulted in objects that are on timers to be mistimed.  Such as, automated doors would close just as a train reached it, instead of seconds afterwards.  I couldn't drive in the corner of the sim, buses became unusable. 

After months of deterioration,  I wrote a blog post about it on January 1st.  I turned off anything that was running in the region.  Improvement was minimal.

Most of the sim was bought and traded hands after an auction.  But, it was no the neighbors, as this started when half the region was abandoned.  Therefore, I couldn't find anything on sim that has changed, and I ruled out anything locally as I haven't added anything from last summer to this winter.  My only conclusion too was that someone on another sim on my server must have had a horse farm or something similar. 

Most mysterious of it all, I stopped having the problem a few weeks ago around the time other sims got EEP.  Maybe it's a coincidence, or maybe whoever was hogging the server did abandon.  But I used to have a regular that said almost all of the mainland was getting unusable laggy at the time I complained.

However, you are right, I'm still showing a 0 spare time on that sim.  The non-lagged sim does have spare time.

Edited by Israel Schnute

Share this post


Link to post
Share on other sites
Posted (edited)

Another overloaded sim. This one is on the beta grid.

stygianoverload.thumb.png.06bc319f8bdbbb29745d7acc1daaa6cc.png

Fort Stygian, on Aditi Not much happening there as shown by object updates, but scripts are in overload.

This is a good place for LL to work on the problem. It's on the beta grid, and is an underutilized testing area.

There's some stuff up in the sky, but it's not showing any object updates. Any significant scripting in this sim is just wasting time.
Irana Pey covered the discussion of this at the last Server User Group meeting.

Edited by animats

Share this post


Link to post
Share on other sites

I can say with some certainty that something has been going on with script times. I have been tracking this for some time now, and i have a performance meter on my sim which effectively gives me the Script Run % through a bench marking script. being a scripter i am a stickler for script performance and  try to optimize any script i write.i have logged this % every 5 min for the past several years. While its difficult to look at all the data at once. The number of scripts on the region rarely changes by much besides what visitors bring into the region. To me from what data i am able to gather, it would seem that not all sims run the same. This is a recent graph of the past few days, the region ran fairly well before the tuesday restart, after the restart it ran horribly while nothing on the region had really changed at all. after a day and a half i found an opportunity to do another restart and performance is back up again. while you may be tempted to say.. well that's just tuesday things don't run as well on tuesday, i can assure you that's not the case, any given restart has randomly caused a drop or increase in performance. There are times when i can do a restart and come back to the region and check the script time, sometimes it will be 8ms, other times it will be 17ms. It will vary widely when its exactly the same scripts on the region with no other visitors.

performance.jpg.5be401e9bf6c0e3782b3cbd171fea667.jpg

blue line is performance, green line is number of avatars on the region, blue line at the top would be 100% scripts run.

  • Thanks 3

Share this post


Link to post
Share on other sites

Come to Server User Group on Tuesday with those numbers, if you can. 12:00 SLT, in Denby.

Share this post


Link to post
Share on other sites
Posted (edited)
39 minutes ago, animats said:

Come to Server User Group on Tuesday with those numbers, if you can. 12:00 SLT, in Denby.

This should not be neccessary, why have a Jira systen (And most likely pay for it) when you have to visit the Linden Lab Cheer Club sessions to get stuff done anyways? 

Edited by Fionalein
  • Haha 1

Share this post


Link to post
Share on other sites

The meetings are absolutely worth going to and are not Linden fan clubs.

Regions do get moved from sim to sim for load balancing, there is the possibility that the one in the chart above had some unfortunate sim buddies for a day.

Also, to be clear.

Multiple regions run on a simulator (sim). A sim is not a section of SL, it represents many parts that may or may not be geographically close to each other on the grid.

Share this post


Link to post
Share on other sites

Got my home set up on a homestead. Checked the "scripts run" quite a few times and always have about 98-100%. 
Same numbers on the sim nearby.

my 2 cents...

  • Like 1

Share this post


Link to post
Share on other sites
46 minutes ago, CoffeeDujour said:

Regions do get moved from sim to sim for load balancing, there is the possibility that the one in the chart above had some unfortunate sim buddies for a day.

I was going to suggest that. I'm pretty sure that restarts shuffle which sims are on which machines, and there's no way to make sure that a sim is coupled with the same sims as before.

I'm using the word 'sim' as we users have used it for centuries.

Share this post


Link to post
Share on other sites
22 minutes ago, Phil Deakins said:

I'm using the word 'sim' as we users have used it for centuries.

A character in a series of games originally developed by Maxis?

Share this post


Link to post
Share on other sites
11 minutes ago, Theresa Tennyson said:

A character in a series of games originally developed by Maxis?

Even before that. I think it was the VisiGoths who refered to a 64k sqm field of land as a sim.

  • Haha 2

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.


×
×
  • Create New...