Somewhere below the regolith in the Linden Lab secret lunar base, some of the Lab’s top minds have been tackling performance issues in Second Life. The areas of focus range from infrastructure improvements to server-side texture compositing to object caching.
This year, Linden Lab is making the single largest capital investment in new server hardware upgrades in the history of the company. This new hardware will give residents better performance and more reliability. Additionally, we are converting from three co-locations to two co-locations. This will significantly reduce our inter-co-location latency and further enhance simulator performance.
The Shining project is the name given to a recent Lab effort to identify and measure delays and other problems with streaming avatars and objects and to propose and implement solutions. During the Shining project, several small improvements have been identified and deployed. The next small improvement from Shining to be deployed will be pre-rendering the starter Avatars to improve the new resident experience.
As a result of the Shining investigation, the project has been split into three larger performance projects: Project Sunshine, Object Caching, and HTTP Library.
All together, the hardware upgrades and the Shining projects should improve avatar and object streaming speeds significantly. Depending on your unique situation, your mileage will vary.
Project Sunshine (Server-side Baked Texture Generation & Storage)
Currently, all client viewers are responsible for compositing their own Avatar textures, then sending the results back to the Sim for other viewers to access. This method can lead to slowdowns and errors. The actual calculations for compositing textures are straightforward and not particularly time-consuming. However, in order for the viewer to do the calculations it first has to download a lot of individual assets from the Sim, and must then upload large results back to the Sim. This pushes a lot of bits through the Sim / Viewer connection, which can be slow and unreliable.
Depending on client hardware to do the compositing and uploading of the resulting baked textures can introduce erroneous results (like way too much pink). In order to handle these errors, a number of retry and fallback mechanisms have been put in place. This adds further load and overhead to the whole system.
Project Sunshine stands up a Texture Compositing server that is separate from the Sims servers. When a Viewer needs to render an Avatar, it sends a message to the Sim, which in turn sends a message to the Texture Compositing Server. The Texture Server then performs the texture compositing and sends the results back to the Viewer.
The Texture Server has access to a database of previously generated Avatar textures. If the Sim is asking for the baked textures for an Avatar that has been previously calculated, the Texture Server can pull the results out of the database instead of recalculating. Even if it’s not the same Avatar, but the set of clothes is the same, the Texture Server can use the previously calculated results.
An additional advantage of having a Texture Generation server is that the heavy HTTP communication about the individual assets that make up an Avatar will happen on reliable, fast internal network connections instead of external connections.
Object Caching & Interest Lists
Caching objects on a resident’s hard drive is intended to provide faster drawing of objects that are most likely to be redrawn later in a resident’s session or in the next session.
The current object cache implementation has room for improvement. Currently, the viewer only caches objects at the point the resident disconnects from a region and only objects currently in view are cached. The current caching communication between Sim and Viewer when a scene is initially being loaded is inefficient and resource intensive.
The new system is designed to make sure that a viewer never has to download object information that it already has and is still valid. This will reduce the load on the Sim and on the Sim / Viewer connection.
With the new system, when the Viewer connects to a region, the Sim first says to the Viewer, “Here’s a list of all my object groups by UUID.” The Viewer checks its cache and replies, “Here are the timestamps for the object groups that I found in my cache.” (If an object group is not found in the cache, the timestamp is 0.) The Sim checks the Viewer’s timestamps against its own data and responds, “Here’s a list of object groups in your cache that are still valid.” While the Viewer starts drawing valid objects from its cache, the Sim starts streaming the new object data that the Viewer needs, based on the particular Viewer’s object priorities.
Top performance for the Shining initiatives depend on the speed and reliability of HTTP messages initiated by both the Viewers and the Sims. The third part of the Shining project, the new HTTP Library, implements modern best practices in messaging, connection management, and error recovery.