Deploy Plan for the Week of 2021-07-26

[Mazidox Linden]

Second Life Server

This week we will be restarting all regions that were not restarted last week to run version 2021-07-16.561609. This week's restarts will not include any changes to the behavior of simulators, nor to the configuration of the machines running those simulators as we are both still gathering data as to how effective last week's modifications were, as well as hunting down a particularly vexing issue.

Second Life RC Updates

We are still in the final stages of figuring out what (if anything) we can promote to Release Candidate stage for the simulator this week. At this point our contender is version 2021-07-20.561821, which contains some changes to help our engineers better investigate crashes and clean up the crash afterward, as well as a fix for BUG-230960, and the implementation of BUG-228541.

No update tomorrow because of a combination of not enough time to finish testing and the bumpiness of today's roll meaning we want to investigate before we roll again.

On Region Restarts

As mentioned above, we're doing a "super-restart" of all remaining regions that weren't restarted last week so that we clean out any cobwebs that might get missed during a normal region level restart. We are aware that the grid poking bot has continued to derelict its duty, and have noted its permanent record; when we know more about the future of rolling restarts you'll hear it here, and at our Tuesday Server User Group meeting in-world.

Edited July 27, 2021 by Mazidox Linden
No Wednesday Roll

[Lucia Nightfire]

It would sure be nice if LL could roll out a server version that allows logging on to aditi without issues.

Every time I want to log on aditi, I get the "despite our best efforts" timeout on the first attempt, followed by "Connecting to region..." > "we are having trouble connecting" timeout on the second attempt.

A third attempt either finally goes through or it is a repeat of the "Connecting to region..." > "we are having trouble connecting" timeout.

This has been the case since January.

[Aishagain]

@Mazidox LindenTsk!  This is another fine Mesh you have gotten us into!

[arabellajones]

The region Restarts have started, certainly before 06:50 SLT. May not be typical, but the restart process had taken at least 20 minutes for the region I had been in. Current users online are about 4000 down on the usual. That doesn't mean so much. I got logged out because of an erratic teleport.

[arabellajones]

At least we are getting info about what is happening, but I do feel a little grumpy about the erratic updates. I'm not in the same timezone as Linden Lab, and some elements of the system, here in the forums and the Status announcements, were horribly last-minute.

It did seem as though the restart failures during the Tuesday restarts are something that should be detectable. Whatever shell script is used to restart a region should be sending a signal to a logging system, and when the sim server does restart, it should be sending a signal. I know it's not trivial, but should the Lindens really have to depend on customers to tell them?

As it is, one region was down for about three hours, and restarted almost instantly when a resident contacted live support. Why didn't the Lindens appear to notice?