Jump to content

Tools and Technology

  • entries
    105
  • comments
    916
  • views
    212,818

Contributors to this blog

Increase in Server Crashes: June 30 to July 8


Linden Lab

1,365 views

 Share

You may have noticed – or fallen victim to -- the unfortunate increase in server crashes last week. We apologize to everyone affected! 

Here’s what happened:

  • On Thursday, June 30, just before the July 4 holiday weekend in the U.S, we released a change to my.secondlife.com.
  • Early on the following Tuesday, we noticed and started investigating a significant increase in server crashes. We also started receiving Support reports about these.
  • We determined that the web update in my.secondlife.com  included a library update which, unknown to us, dropped support for the texture format our servers expect.
  • This led to Viewers being unable to deal with newly updated textures, and under some circumstances the Server would attempt to be more helpful in opening those, than it should have been - and crashing. This is why some residents saw problems with their profile photos. 

On Tuesday July 5, after we investigated and confirmed what was happening, the team quickly rolled back the web site change to prevent the creation of more broken textures. The web team identified these textures, then ran a script to convert them. This was completed by Thursday morning, July 7.

The server team fast-tracked a fix to prevent servers from crashing. We were able to QA and release the fix to RC channels on the afternoon of Thursday, July 7 and then to the main SLS + Events channels on Friday, July 8.

Lessons:

  • Second Life is big and complicated and fails in unexpected ways (we knew that!)
  • Library upgrades may need additional testing for dependencies we couldn’t imagine.
  • We work really well together when Second Life fails in unexpected ways                                                    

Again, we apologize to everyone affected!

  • Like 8
  • Thanks 6
 Share

0 Comments


Recommended Comments

There are no comments to display.

×
×
  • Create New...