Deploy Plan for the Week of 2022-07-04

[Mazidox Linden]

Second Life Server and Events

The SLS Channel will be restarted on Tuesday, 2022-07-05, from 06:30 - 11:30 PDT, with no version change (2022-06-15.572665).

Second Life RC

This week we'll be rolling version 2022-06-22.572768 to the BlueSteel Release Candidate channel - this is the first part of a series of changes to make rolling the grid easier for our Engineers. The other RC Channels will be restarted with no version change.

Coming Soon*…

Another maintenance release is likely our next candidate for promotion

• As always, Coming Soon represents our best guess at what will be moved from Aditi to RC in the coming weeks. We may need to delay, reshuffle, or remove upcoming releases for operational reasons.

[Coffee Pancake]

Crosses fingers that this is the week 'Horizon - Bianca' come back up without taking an hour and getting rolled back in the process.

[Aishagain]

I wonder if the update rolled to Bluesteel regions (a subset now known only to the Lindens, of course) is the cause of this elevated Region crash rate?

 

[Mazidox Linden]

Nope, otherwise we would have rolled those regions back. This particular issue was a great example of how changes to "unrelated" parts of Second Life's underlying technologies can break core Resident facing services like the simulators. It's an extremely complex system!

[Aishagain]

Rolling Restarts for most Second Life RC Channels Jul 7, 13:00-19:00 PDT

We will be performing rolling restarts for most regions on the Second Life RC Server channels today (Thursday, July 7th) beginning at approximately 1:00 PM PDT. Please refrain from rezzing no copy objects before and during the restart, and remember to save all builds. Please check this blog for updates.
Posted on Jul 7, 12:29 PDT

hmm.  Methinks thou speakest to soon, or is this rolling the changes made to Bluesteel to the other RC channels?

And now Main Server as well what DID you lot break?? (non judgemental semi-ironic wibble).

 

Edited July 7, 2022 by Aishagain
removing some tinfoil hat stupidity

[Logan Elf]

Version being rolled today (7/7) is 2022-07-06.573149 (as usual release notes not available)

[Aishagain]

So, now that we are to be on a new software version, when do we see release notes?

[Mazidox Linden]

Hey all, release notes are now available at https://releasenotes.secondlife.com/simulator/2022-07-06.573149.html
As previously mentioned we updated another part of the system that through a ripple effect caused the simulators to sometimes crash. We rolled the affected component back on Tuesday evening, scrubbed all the corrupted data we could find by Wednesday morning (this reduced our mid-incident crash rate by about 2/3s) and have spent the rest of the week figuring out how to clean everything up satisfactorily. Ultimately we came to the conclusion that the simulator update we had planned for next week was the fastest and most comprehensive way to stop crashes that were still happening, so we rolled that out yesterday and today.

[Lucia Nightfire]

4 hours ago, Mazidox Linden said:

Hey all, release notes are now available at https://releasenotes.secondlife.com/simulator/2022-07-06.573149.html
As previously mentioned we updated another part of the system that through a ripple effect caused the simulators to sometimes crash. We rolled the affected component back on Tuesday evening, scrubbed all the corrupted data we could find by Wednesday morning (this reduced our mid-incident crash rate by about 2/3s) and have spent the rest of the week figuring out how to clean everything up satisfactorily. Ultimately we came to the conclusion that the simulator update we had planned for next week was the fastest and most comprehensive way to stop crashes that were still happening, so we rolled that out yesterday and today.

What "component" was this and what data got corrupted?

[Logan Elf]

Bluesteel RC simulator version rolled today (8th July) is 2022-07-07.573176 (release notes not available at time of posting)

[Coffee Pancake]

If you did anything to Horizons Bianca .. thanks, it rebooted nice and fast with no roll back this week. 

[Coffee Pancake]

52 minutes ago, Lucia Nightfire said:

What "component" was this and what data got corrupted?

This might be referring to the CDN serving up PNG data in-place of JPEG2000 .. one can be processed progressively, the other not so much. Gummed up texture processing, we had to rush out an emergency Catznip.

[Lucia Nightfire]

1 hour ago, Coffee Pancake said:

Gummed up texture processing, we had to rush out an emergency Catznip.

Why? Was Catznip doing something different with images than other TPVs? If not, are they also at risk as well in some way?

What kind of damage are we talking here? heh

[Ardy Lay]

1 hour ago, Coffee Pancake said:

This might be referring to the CDN serving up PNG data in-place of JPEG2000 .. one can be processed progressively, the other not so much. Gummed up texture processing, we had to rush out an emergency Catznip.

I didn't notice.

[Coffee Pancake]

1 hour ago, Lucia Nightfire said:

Why? Was Catznip doing something different with images than other TPVs? If not, are they also at risk as well in some way?

What kind of damage are we talking here? heh

The issue also affected others but not to anything like the same degree, not sufficient for anyone to notice .. why this clobbered us so badly I'm honestly not sure.

Basically the viewer asked for the start of some image data, got a partial file, which is normal and fine for JPEG2000 but not PNG. So the viewer would try to process the incomplete PNG, fail and the retry, forever. Opening your friends list would flood the texture queue with so many of these stuck PNG files that any other textures just couldn't get a shot at being processed, so everything would get stuck untextured.

We poked LL who confirmed this wasn't supposed to happen and then rushed out a mandatory update as we didn't know how long it would take LL and Kitty would be out of reach till after the weekend. 

But then if this hadn't happened, I wouldn't have had the opportunity to tease kitty with the R13.2 release notes