Jump to content

Deploy Plan for the Week of 2022-07-04


You are about to reply to a thread that has been inactive for 82 days.

Please take a moment to consider if this thread is worth bumping.

Recommended Posts

  • Lindens

Second Life Server and Events

The SLS Channel will be restarted on Tuesday, 2022-07-05, from 06:30 - 11:30 PDT, with no version change (2022-06-15.572665).

Second Life RC

This week we'll be rolling version 2022-06-22.572768 to the BlueSteel Release Candidate channel - this is the first part of a series of changes to make rolling the grid easier for our Engineers. The other RC Channels will be restarted with no version change.

Coming Soon*…

Another maintenance release is likely our next candidate for promotion

  • As always, Coming Soon represents our best guess at what will be moved from Aditi to RC in the coming weeks. We may need to delay, reshuffle, or remove upcoming releases for operational reasons.

  • Thanks 2
Link to comment
Share on other sites

I wonder if the update rolled to Bluesteel regions (a subset now known only to the Lindens, of course) is the cause of this elevated Region crash rate?

 

Link to comment
Share on other sites

  • Lindens

Nope, otherwise we would have rolled those regions back. This particular issue was a great example of how changes to "unrelated" parts of Second Life's underlying technologies can break core Resident facing services like the simulators. It's an extremely complex system!

  • Thanks 3
Link to comment
Share on other sites

Posted (edited)
We will be performing rolling restarts for most regions on the Second Life RC Server channels today (Thursday, July 7th) beginning at approximately 1:00 PM PDT. Please refrain from rezzing no copy objects before and during the restart, and remember to save all builds. Please check this blog for updates.
Posted on Jul 7, 12:29 PDT
hmm.  Methinks thou speakest to soon, or is this rolling the changes made to Bluesteel to the other RC channels?
And now Main Server as well what DID you lot break?? (non judgemental semi-ironic wibble).
 
Edited by Aishagain
removing some tinfoil hat stupidity
  • Like 1
Link to comment
Share on other sites

  • Lindens

Hey all, release notes are now available at https://releasenotes.secondlife.com/simulator/2022-07-06.573149.html
As previously mentioned we updated another part of the system that through a ripple effect caused the simulators to sometimes crash. We rolled the affected component back on Tuesday evening, scrubbed all the corrupted data we could find by Wednesday morning (this reduced our mid-incident crash rate by about 2/3s) and have spent the rest of the week figuring out how to clean everything up satisfactorily. Ultimately we came to the conclusion that the simulator update we had planned for next week was the fastest and most comprehensive way to stop crashes that were still happening, so we rolled that out yesterday and today.

  • Thanks 3
Link to comment
Share on other sites

4 hours ago, Mazidox Linden said:

Hey all, release notes are now available at https://releasenotes.secondlife.com/simulator/2022-07-06.573149.html
As previously mentioned we updated another part of the system that through a ripple effect caused the simulators to sometimes crash. We rolled the affected component back on Tuesday evening, scrubbed all the corrupted data we could find by Wednesday morning (this reduced our mid-incident crash rate by about 2/3s) and have spent the rest of the week figuring out how to clean everything up satisfactorily. Ultimately we came to the conclusion that the simulator update we had planned for next week was the fastest and most comprehensive way to stop crashes that were still happening, so we rolled that out yesterday and today.

What "component" was this and what data got corrupted?

Link to comment
Share on other sites

52 minutes ago, Lucia Nightfire said:

What "component" was this and what data got corrupted?

This might be referring to the CDN serving up PNG data in-place of JPEG2000 .. one can be processed progressively, the other not so much. Gummed up texture processing, we had to rush out an emergency Catznip.

Link to comment
Share on other sites

1 hour ago, Coffee Pancake said:

Gummed up texture processing, we had to rush out an emergency Catznip.

Why? Was Catznip doing something different with images than other TPVs? If not, are they also at risk as well in some way?

What kind of damage are we talking here? heh

Link to comment
Share on other sites

1 hour ago, Coffee Pancake said:

This might be referring to the CDN serving up PNG data in-place of JPEG2000 .. one can be processed progressively, the other not so much. Gummed up texture processing, we had to rush out an emergency Catznip.

I didn't notice.

Link to comment
Share on other sites

1 hour ago, Lucia Nightfire said:

Why? Was Catznip doing something different with images than other TPVs? If not, are they also at risk as well in some way?

What kind of damage are we talking here? heh

The issue also affected others but not to anything like the same degree, not sufficient for anyone to notice .. why this clobbered us so badly I'm honestly not sure.

Basically the viewer asked for the start of some image data, got a partial file, which is normal and fine for JPEG2000 but not PNG. So the viewer would try to process the incomplete PNG, fail and the retry, forever. Opening your friends list would flood the texture queue with so many of these stuck PNG files that any other textures just couldn't get a shot at being processed, so everything would get stuck untextured.

We poked LL who confirmed this wasn't supposed to happen and then rushed out a mandatory update as we didn't know how long it would take LL and Kitty would be out of reach till after the weekend. 

But then if this hadn't happened, I wouldn't have had the opportunity to tease kitty with the R13.2 release notes :P

  • Like 1
  • Haha 1
Link to comment
Share on other sites

You are about to reply to a thread that has been inactive for 82 days.

Please take a moment to consider if this thread is worth bumping.

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share

×
×
  • Create New...