You may have been among those who had problems seeing avatars yesterday; this is a quick note to share with you what went wrong and what comes next.

Early yesterday morning, we deployed an update to the backend service that creates the baked textures and other data that make up your avatar’s appearance. The time was chosen to be at a low point in concurrency because we knew the update would create unusual load (making all-new appearances for every active avatar). It turned out that the load was a little higher than we expected; that probably would have been OK, but we had two simultaneous system failures in the servers that provide some of the key data to the service. By the time we had diagnosed those failures, the backlog of work to be done had grown considerably and the number of active users had also increased. It took a few hours for the backlog of avatar appearances to get caught up.

We realize how disruptive events like these can be (our own inworld meetings were full of mostly-gray Lindens and clouds yesterday), and very much regret the inconvenience.

We're increasing the redundancy of these services and modifying our deploy procedures to avoid a repeat of this kind of failure in the future.

The good news is that this was one of the last backend changes needed for us to be able to roll out the new Bakes-on-Mesh feature on agni; it will, with the updated viewer, allow you to apply system clothing and skins on your avatar even when you're using mesh body parts. Look for the viewer release announcement soon.

        - Oz Linden