Deploy Plan for the Weeks of 2021-11-22 and 2021-11-29

[Mazidox Linden]

Today we restarted all simhosts on the main channel, and tomorrow we'll be restarting all RC channels. No versions or configurations will change as a result of these restarts; you can expect that most weeks if we don't have a new build to deploy to a channel, we'll restart it during the usual deploy windows. Individual region downtime should be on order of minutes for most of these restarts, and we generally finish restarting all of the Second Life Server channel in under 2 hours.

Second Life Server, 2021-11-29

Next Monday 2021-11-29 we will be moving all simulators currently running on the old configuration previously mentioned here to our new configuration. After this change, all simulators running on our Agni grid will be running on the new configuration, regardless of version. In the coming days we'll have a blog post talking a bit more about the improvements we made with this configuration change, so keep an eye on the Second Life blog roll for that.

Again, to emphasize, this change will happen on Monday 2021-11-29 at 7:00 AM Pacific

Monday Morning Update: We've had to tweak the schedule slightly, new details will be figured out this morning/afternoon Pacific Time. The best bet for up to date information today is the Second Life Grid status blog at status.secondlifegrid.net (as I injured my hands over the weekend so I'm much slower at getting this forum post updated, oops 😐)

Second Life RCs, 2021-12-01

Next Wednesday 2021-12-01 we will be deploying a new version to some RCs. This version's primary change is that when a viewer attempts to toggle the value of Preferences->Chat->Email me IMs when I'm offline, viewers will get a system IM informing them that the correct way to change their IM and Group Notice settings is via the accounts page at secondlife.com. A future viewer release will remove this checkbox.

Scheduled for Wednesday 2021-12-01 at 7 AM Pacific

Edited November 29, 2021 by Mazidox Linden
Monday Morning Update

[bigmoe Whitfield]

well I will be waiting,  wondering how much you changed on the aws vm's.

[Phoebe Avro]

SInce the restart on Wednesday my region is unstable people keep crashing last night (Wednesday), I have restarted the region myself today but people including myself are seeing viewer stuttering, and other problems, I have submited a support ticket also.

Region name is  End of Days

[Lucia Nightfire]

3 hours ago, Phoebe Avro said:

SInce the restart on Wednesday my region is unstable people keep crashing last night (Wednesday), I have restarted the region myself today but people including myself are seeing viewer stuttering, and other problems, I have submited a support ticket also.

Region name is  End of Days

Fitting region name for the occasion.

[Aishagain]

@Mazidox Lindentsk! Playing with matches again, Mazidox?

[Hyperplexed]

Seems like it was postponed for tomorrow... https://status.secondlifegrid.net/incidents/njrxdnzh2htr

Quote

We will be performing rolling restarts for regions on the main Second Life Server channel on Tuesday, November 30th, beginning at approximately 6:00 AM PDT.

In the meantime, here I was all day waiting for the restart! LOL. Nini everyone 😃 

[Kaleb Nightingale]

With the performance of the internet the last few days, including SL, probably because of all the black sales, I think they were right to postpone it.

I see others have been commenting on performance issues in the SL Server Forum.

[Aishagain]

@HyperplexedYes it was postponed, even Maxidox's edited notes in the OP give no detail (no indication at what time the original plans were abandoned but according to the GSP it was stopped before it started..!) and Main Server is officially "under Maintenance" still until the job is attempted again tomorrow.

@Harry WainwrightYes I do wonder if LL know what the issue is (it is very likely a general internet borkage) but they aren't telling.

The baking issue seemed much better today but was still noticeably slow in the PST afternoon.

Steeljane42's comments on

are worthy of note.

Edited November 30, 2021 by Aishagain
Additional information

[arabellajones]

The Status reports show the restarts as running but I have not seen any of the usual signs of restarts happening. I have been doing other things but I know of in-world indicators I can check.

[Drake Darkheart]

i see another roll fail coming lol

[Monty Linden]

Unbelievers!

[Aishagain]

All well and good @Monty Linden, but my region, Woods of Heaven, which almost unfailingly is among the first to roll has still not rolled. So you'll excuse me for being a sceptic.

 

ETA 9am SLT:  At least your Grid Status page is now admitting that it is taking longer than expected.

ETA 2 11am SLT : STILL it is grinding on.

Edited November 30, 2021 by Aishagain

[MBeatrix]

Lotsa regions down for maintenance.

[Phoebe Avro]

It took 4 days for LL to reply to my support ticket and the linden that replied obviously never read the ticked, saying i should restart the region often because its a busy region (I had stated in the ticket I HAD RESTARTED THE REGION) who are you employing? LL and SL are not what they used to be!

 

[VenKellie]

3 minutes ago, Phoebe Avro said:

It took 4 days for LL to reply to my support ticket and the linden that replied obviously never read the ticked, saying i should restart the region often because its a busy region (I had stated in the ticket I HAD RESTARTED THE REGION) who are you employing? LL and SL are not what they used to be!

 

was it Vix Linden? he seems to never read tickets and just replies like he is a bot with a standard pre-writen reply

[Phoebe Avro]

It was via Ellie Linden and was basically a rubbish support reply on key with "Try turning it off and back on again", since I used to be an IT Analyst its quite quaint to see such replies are still used

 

Edited November 30, 2021 by Phoebe Avro

[Profaitchikenz Haiku]

I'm intrigued, after a lengthy downtime today Zephyr restarted and where it previously had a scripts run figure of 35% I'm now seeing 68-70%, and my train is running the smoothest I've seen in almost 2 years. Here's hoping it lasts.

[Aishagain]

OK @Linden Lab: I am calling you out this time.  YET AGAIN my region (Woods of Heaven) has NOT been restarted.  So, either the Roll was abandoned or it is NOT yet complete!  The Grid Status Page announced 25 minutes ago that the Roll had been "resolved"...what on earth do you mean by that?

If I had 100dollars for every time you announce that a Rolling Restart is complete and my region (and mine is NOT the only one) has not restarted/rolled, I would be rich...(well richer than I am).

This has happened far too often for it to be "just a minor oversight".  It is sloppy.

A brief conversation via Live Chat told me nothing other than some tech somewhere will have his ears boxed for this.  I just have the feeling that LL are no longer in control of SL in any meaningful sense.

Edited November 30, 2021 by Aishagain

[MBeatrix]

1 hour ago, Profaitchikenz Haiku said:

I'm intrigued, after a lengthy downtime today Zephyr restarted and where it previously had a scripts run figure of 35% I'm now seeing 68-70%, and my train is running the smoothest I've seen in almost 2 years. Here's hoping it lasts.

Nuggy, my home sim, had over 90% Scripts Run for a few months, after some announced improvements. Then, suddenly, it dropped to around 60% average for the whole November till today before maintenance. Now it shows way over 90% average again. No builds have changed in the region nor the scripts amount, so this is good news. ☺️

[Profaitchikenz Haiku]

I looked back and I think Mazidox was hopeful that the configuration changes they were introducing would improve things. Two other people I know who's parcels were down on the scripts run have confirmed what I'm seeing the percentage figure is a lot healthier now for those region that id get restarted. Two regions I went to that hadn't got resterted are still at low percentage figures, s it is the restart process that is giving the improvement.

[Monty Linden]

23 minutes ago, Aishagain said:

YET AGAIN my region (Woods of Heaven) has NOT been restarted.

This has been on the new hardware for the past week (since 11/23).  No need to restart it.

[Crim Mip]

Well I guess the land I own and sims I help manage are now on the new config because Wow!. Scripts running at 100% with spare time up to like 9ms. Those took a major nosedive after prodject uplift so it's nice to see it's been fixed however it was fixed.

 

[steeljane42]

29 minutes ago, MBeatrix said:

Nuggy, my home sim, had over 90% Scripts Run for a few months, after some announced improvements. Then, suddenly, it dropped to around 60% average for the whole November till today before maintenance. Now it shows way over 90% average again. No builds have changed in the region nor the scripts amount, so this is good news. ☺️

Same experience here. My own region usually had over 80% scripts run and with a few manual restarts after rolls it was pretty easy to get ~90+%. Then suddenly without any changes it dropped to ~65% on a good restart, and I saw as low as 45%. No difference in builds/script amount were made (not counting a few extra avsitter scripts). And after today's rolls it seems to be even better than before the drop happened. Seeing 100% scripts run now. Hopefully it'll stay as is.

 

P.S.

Blog post about improvements made that is mentioned in the deploy plan will be interesting to read. Especially if LL will mention what it was that caused such a massive scripts run % drop until they fixed it again.

Edited November 30, 2021 by steeljane42
typos and more typos

[Aishagain]

4 hours ago, Monty Linden said:

This has been on the new hardware for the past week (since 11/23).  No need to restart it.

That is news to me, @Monty Linden.  If that is so then not only do I not see any performance improvement, our script-run, which was usually 60% or better, it is now under 45%.

I must say I cannot recall seeing that only a part of Main Server was being migrated to the new format. I just reread Mazidox's post regarding last week's roll and there is no mention of it being only a part of SLS that was receiving a "new configuration", which, by the way, does not mean "new Hardware" in my lexicon.

Put bluntly your communication of this change was  as clear as mud.

Further comment: I see in the above posts that many of the contributors are seeing substantial improvements in % script-run.  How unusual then, that I am seeing the converse!  If I were a suspicious person I must be inclined to doubt the veracity of your assertion that my region was moved to the new configuration last week.

That said you have never given me the slightest hint that you are duplicitous.

Edited December 1, 2021 by Aishagain

[Jaylinbridges]

3 hours ago, Aishagain said:

If that is so then not only do I not see any performance improvement, our script-run, which was usually 60% or better, it is now under 45%.

Maybe it takes a week to degrade to 45%.

My main region jumped from about 45% to 98% Scripts Run, after the migration restart today.