Deploy Plan for the Week of 2021-10-18

[Mazidox Linden]

Second Life Server

We'll be restarting all Second Life Server simhosts on Tuesday, October 19th, between 06:00 and 11:00 Pacific Daylight Time. As is usual recently we'll be using a new simulator version (in this case version 2021-10-15.564759). I saw some comments regarding the lack of release notes for versions we use to do large scale rolling restarts, and while I can't promise it will be successful I'm going to try something that should let me put up release notes for this restart-only version. Hand-made release notes page for 564759 is here!

Second Life RCs

We'll be deploying version 2021-10-08.564610 to all Second Life RCs on Wednesday, October 20th, between 07:00 and 10:00 Pacific Daylight Time.

We're going to be restarting all Second Life RC hosts between 07:00 and 10:00 Pacific Daylight Time. The version number will not change as part of this restart.

On Region Restarts

Everything is being restarted this week for that new simulator feeling, yay!

Edited October 19, 2021 by Mazidox Linden
Changes to Wednesday plans

[Aishagain]

@Mazidox Linden that's haha for the first paragraph and the release notes comment and thanks for the whole thing!

[JoyofRLC Acker]

Ok Im really not getting the joke in the "release note".  Is this just 'maintenance' or something more?

Especially i the context of :

"

Update - Scheduled maintenance is still in progress. We are aware some regions are taking longer than normal to come back online. Thank you for your patience!
Oct 19, 07:50 PDT

"

(emphasis added)

[Mazidox Linden]

This is just restarting the machines the simulators run on so they work better, unfortunately we had an issue come up where regions would not start on the new version, so we had to re-start the roll with an earlier version we know works. That should be finishing up just a bit later than expected, and the grid should be back to normal operation soon.

[Aishagain]

@Mazidox LindenSo this will, I assume, be why my region, Woods of Heaven, never did get a restart today despite the GSP stating that the roll is complete a few minutes ago?

Frankly the excuses reasons for these "unexpected" failures are beginning to wear thin.

 

Edited October 19, 2021 by Aishagain
Additional information

[MBeatrix]

@Mazidox Linden Nuggy hasn't been restarted.

[Mazidox Linden]

It looks like we ended up aborting the roll instead of trying to take down the existing hosts and replace them with new hosts, that was my bad on miscommunicating (I misunderstood what other Lindens were telling me). We are not going to roll version 2021-10-08.564610 tomorrow as we found a showstopper bug (hat tip to Lucia Nightfire for finding it during the Server User Group meeting), but we are instead going to use a new technique we've been testing successfully on Aditi this afternoon to restart the Second Life RC versions with their existing version 2021-10-01.564394 for all RCs. If it works we'll restart all Second Life Server hosts the same way next Tuesday.

[arabellajones]

OK, that all explains the behaviour of the concurrency numbers for Tuesday and Wednesday. There's a very regular pattern, every day, a consistent shape to the curve. A restart shows a very large dip in concurrency (bots, I suppose) Nothing much on Tuesday, a typical smaller dip on Wednesday.

I get the data from here: softhyena.com but it only shows the detailed concurrency for the last 24 hours.