To all our Residents impacted by the bumpy grid roll Tuesday, we’re sorry - and we want you to know what happened.

Timeline:
It started out as a normal roll to Second Life Server (SLS) when, around 7am SLT, the team noticed some regions were not coming back as swiftly as they should. Alarms were raised and we got a team together to tackle the issue immediately.  It was quickly determined that we could not continue with the roll without causing more extensive pain to Residents, and the decision was made before 9:00 AM SLT to halt the in-progress grid roll. That left approximately 3800 regions in an offline state, while we continued the investigation.

The team continued to bring the regions back manually, while simultaneously working to identify the cause of the problems. By noon there were still approximately 350 regions stuck offline. 

Cause:
We’ve identified the root cause of the trouble. The server code itself is not the issue, but infrastructure communications between various AWS services broke down and prevented the normally smooth weekly grid roll. 

Next steps: 
We’ve canceled the RC Roll on Wednesday while we ensure this problem doesn't happen again. Next week will be a quiet week with no SLS roll. Things should return back to normal with the following week’s server updates. 

Thank you for your patience while we banged on things yesterday.