You may have noticed – or fallen victim to -- the unfortunate increase in server crashes last week. We apologize to everyone affected! 

Here’s what happened:

• On Thursday, June 30, just before the July 4 holiday weekend in the U.S, we released a change to my.secondlife.com.
• Early on the following Tuesday, we noticed and started investigating a significant increase in server crashes. We also started receiving Support reports about these.
• We determined that the web update in my.secondlife.com  included a library update which, unknown to us, dropped support for the texture format our servers expect.
• This led to Viewers being unable to deal with newly updated textures, and under some circumstances the Server would attempt to be more helpful in opening those, than it should have been - and crashing. This is why some residents saw problems with their profile photos. 

On Tuesday July 5, after we investigated and confirmed what was happening, the team quickly rolled back the web site change to prevent the creation of more broken textures. The web team identified these textures, then ran a script to convert them. This was completed by Thursday morning, July 7.

The server team fast-tracked a fix to prevent servers from crashing. We were able to QA and release the fix to RC channels on the afternoon of Thursday, July 7 and then to the main SLS + Events channels on Friday, July 8.

Lessons:

• Second Life is big and complicated and fails in unexpected ways (we knew that!)
• Library upgrades may need additional testing for dependencies we couldn’t imagine.
• We work really well together when Second Life fails in unexpected ways                                                    

Again, we apologize to everyone affected!