At one time or another, many of us have suffered a Viewer crash that interrupted our time in Second Life. At Linden Lab, we collect crash data to help us in our ongoing efforts to identify and fix issues in the Viewer that can cause crashes. Thanks to that information, nearly every release we make includes fixes of this sort.
The data we collect also reveals that many users can greatly reduce their risk of Viewer crashes by taking a few steps to update their software outside of Second Life.
The nature of Second Life as a platform for user creativity means that the Viewer faces different challenges than client software for an online game, for example, which would just need to handle the limited and carefully optimized content created by the game’s developer. This can make Second Life a demanding application for your computer and can mean that if your operating system is out of date, your Viewer is more likely to crash.
The good news is you can take steps today to help this! Here are a couple of tips:
Upgrade your Operating System
There is a very clear pattern in our statistics - the more up to date your operating system is, the less likely your Viewer is to crash. This applies on both Windows and Macintosh (Linux is a little harder to judge, since "up to date" has a more fluid meaning there, and the sample sizes are small). Some examples:
Windows 8.1 reports crashes only half as often as Windows 8.0
Those of you who stuck with Windows 7 (roughly 40% of users of our Viewer right now) rather than upgrade to 8.0 made a good choice at the time; version 7 still has a much better crash rate than 8.0, but not quite as good as 8.1 (now about 15% of users), so waiting is no longer the best approach.
Mac OSX 10.9.3 reports crashes a third less than 10.7.5
OSX rates do not have as much variation as Windows versions do, but newer is still better, and there are other non-crash reasons to be on the up to date version, including rendering improvements.
Upgrading will probably also better protect you from security problems, so it's a good idea even aside from allowing you to spend more time in Second Life.
Use the 64 bit version of Windows if you can
For each version of Windows for the last several years, you have had a choice between 32 bit and 64 bit variants; if your system can run the 64 bit variant, then you will probably crash much less frequently by changing to it. While we don't have a fully 64 bit version of the Viewer yet, you can run it on 64 bit Windows, and statistically you'll be much better off if you do.
Generally speaking the 64 bit Windows versions report crashes half as often as the 32 bit versions.
According to the data we collect, a little more than 20% of users are running 32 bit Windows versions; most of you can probably upgrade and would benefit by it.
If you bought your computer any time in the last 5 years, chances are very good that it can run the 64 bit version of Windows (as will some systems that are even older). Microsoft has a FAQ page on this topic; go there and read the answer to the question "How do I tell if my computer can run a 64-bit version of Windows?". That page also explains how to do the upgrade and other useful information.
We'll of course continue working hard to find and fix things that lead to Viewer crashes. Even as we do that, though, you can decrease your chances of crashing today by taking the steps above.
We're ready to start a limited beta test of an exciting new tool for creators: Experience Keys. These are new LSL functions and calls that make it possible to bypass the multiple permissions dialogs that you encounter with scripted objects today. Experience Keys will make it possible for users to create more immersive experiences inworld, because those interacting with the experience will be able to grant all the permissions necessary to participate just once, instead of having the experience interrupted by multiple permissions requests. To learn more, check out this brief video.
We used this technology when creating the Linden Realms game, and we're now ready to start putting this tool in the talented hands of creators in the Second Life community. Experience Keys is a powerful tool, and we need to be sure we test and roll out the feature carefully, so the first step will be a limited beta, then the viewer and server releases shortly after.
If you’d like to participate, send an email to email@example.com with “Experience Key Beta” as the subject along with:
Your experience name.
What genre does it fit in?
Give us a brief description of your experience.
How would your customers benefit from Experience Keys?
When I came to Linden Lab over five years ago, Second Life had gone through a period of the coveted hockey-stick growth, and we had just not kept up with the technical demands such growth creates. One or more major outages a week were common.
In my first few months at the Lab, we removed more than a hundred major single points of failure in our service, but several major ones still loomed large, the granddaddy of them all being the core MySQL database server. By late Winter 2009 we were suffering from a core database outage a few times each week.
With a lot of hard work and countless long nights we stabilized the service and started making major improvements to the overall stability and performance of Second Life. However, despite our continued improvements, and the relative tranquility they have created, the spectres of technical debt and single points of failure still loom over our operations. In recent weeks some of them have struck and disrupted Second Life. So much so that I want to explain the outages that have occurred, how we addressed them, and what we are doing going forward.
First, that core MySQL database cluster still exists. It is still the core of many of our central functions. When the write server fails it takes a minimum of thirty minutes to promote a new server into position. The promotion itself is actually relatively quick, but its numerous dependent services must all be taken down and brought back up carefully to ensure that they are all functioning properly.
In the last two months the core MySQL write database has hit two different fatal hardware faults, driving us to temporarily halt most Second Life operations. In some sense, two major write database failures close together is bad luck, but we cannot depend on luck to ensure the reliability of Second Life. In the very near future, we are moving the core MySQL write server to a new hardware class, on which production read servers are already running. Moving the write server will further improve overall database performance and make failures less frequent. It does not of course solve the root single point of failure problem so in the coming days, weeks, and months we will be reducing the impact of database failures even more. This includes continued improvement to the rotation process, extracting more functions out of the core database cluster, and further reducing the number of features that depend on the single write server.
The core MySQL database, however, has not been our only problem recently. A few weeks ago there was a massive distributed denial of service attack on one of our upstream service providers that affected most of their customers, including us, and inhibited the ability of some to use our services. We have since mitigated future potential impact from such an attack by adding an additional provider. There have also been hardware failures in the Marketplace search infrastructure that have impacted that site, a problem that we are continuing to work through. Most seriously though was this week’s four and a half hour long login outage.
On Tuesday morning, users stopped being able to get into Second Life. The root cause was created over ten years ago in a system designed to assign a unique identifier to the hand-off of sessions from login to users’ initial regions. At 7:40AM Pacific Time, that system quietly ran out of possible numbers to assign. It took us four hours to isolate the problem, test a fix, and deploy the change. Users could immediately log in at that point, but it took an additional two hours for systems to settle out. When tens of thousands of users rush back into Second Life following an outage, we have to deliberately throttle some services to prevent further breakage.
Having such a hidden fault in a core service is unacceptable, so we are doing a thorough review of the login process to determine if there are any more problems like this lurking. Our intent at this point also is to remove the identifier assignment service altogether. It not only was the ultimate source of this outage, but is also one more single point of failure that should have been dispatched long ago.
We want to apologize for all of the recent problems and the frustration they have caused. We too are frustrated and are intent on making our service better. Few things give me more pleasure than every day helping to make Second Life a happy and fun place. Thank you for your patience and support. We simply could not have a more devoted user base and for that we owe you better.
We have made some changes to the Second Life System Requirements to bring them more up to date, and are making some related changes to the Viewer:
We have removed Windows XP and Mac OSX 10.6 from the list of supported operating systems. Microsoft has announced the end of support for XP, and it has been some time since Apple has released updates for 10.6. For some time now, the Viewer has been significantly less stable on these older systems, and the lack of security updates to them make them more hazardous to use.
We have no plans to actually block those systems, but problems reported on them that cannot be reproduced on supported systems will not likely be fixed.
The Windows installer has been modified to verify that the system has been updated with the most recent Service Packs from Microsoft. While we will not block installation on Windows 8 at this time, we strongly recommend upgrading to 8.1 for greater stability. Our data shows that the Viewer is significantly less stable on systems that have not been kept up to date, so the installer will now block installation until the updates have been applied. This change will be effective in a Viewer version to be released in the next few weeks, so it would be a good idea to get your system up to date before then. You can find information on how to install the latest updates at the Microsoft Windows Update page.
Last week, we made a new page available as a replacement for the old Transaction History page. Due to your feedback, we rolled back the changes to this page to allow us to gather more feedback, and we are now providing this new page for review, without removing the old Transaction History page.
We have not yet made any changes to the new page, because we would like time to collect your feedback and review it. We have created a wiki page giving background on why changes were made to this page, where the new page is, and how to provide feedback. We will be closing feedback on April 30, 2014, so please take a look before then.
Many of you may have read about the Heartbleed SSL vulnerability that is still affecting many Internet sites.
You do not need to take extra action to secure your Second Life password if you have not used the same password on other websites. Your Second Life password was not visible via Heartbleed server memory exposure. No secondlife.com site that accepts passwords had the vulnerable SSL heartbeat feature enabled.
If you used the same password for Second Life that you used on a third-party site, and if that third-party site may have been affected by the vulnerability, you should change your password.
Supporting sites such as Second Life profiles are hosted on cloud hosting services. Some of these sites were previously vulnerable to Heartbleed, which may have exposed one of these servers' certificates. As an extra precaution, we are in the process of replacing our SSL certificates across the board. This change will be fully automatic in standard web browsers.
Thank you for your interest in keeping Second Life safe!
We’re happy to report that the Photo Upload feature of SL Share is once again working (as we previously blogged, that portion of the service had been temporarily disabled by Facebook). Thank you again for your patience as we worked to resolve this issue.
The restored feature no longer automatically includes SLURLs when you share a picture to Facebook, but it’s still possible to let your Facebook friends know where you are inworld by using the Check-In function in SL Share.
As you may have seen, we’re expanding the functionality of SL Share to include not only Facebook, but also Flickr, and Twitter. You can read about that work here and try it out with the project Viewer now available here.
UPDATE - April 3, 2014: this issue is resolved and the Photo Upload feature is once again working. Thank you for your patience!
Facebook recently contacted us to let us know that the Photo Upload feature of SL Share is not permitted to automatically include location SLURLs in posts made from the application. We’re working with them to get a hotfix out ASAP, but in the meantime the Photo Upload feature in SL Share will not work, as Facebook has temporarily disabled that part of the application. SL Share’s Status Update and Check-In features will continue to work.
When SL Share’s full functionality is restored, SLURLs will no longer be included when you share a picture using Photo Upload, but you will still be able to let your Facebook friends know where to join you in Second Life by using the Check-In feature.
We apologize for the inconvenience this may cause you and are working to get a fix out ASAP. We’ll use this blog to keep everyone posted with any updates and will of course let you know once the issue is resolved as well. Thank you for your patience.
The Oculus Rift offers exciting possibilities for Second Life - the stereoscopic virtual reality headset brings a new level of immersion to our 3D world, making Second Life a more compelling experience than ever before.
Though a consumer version of the headset isn’t available yet, we’ve been working with the development kit to integrate the Oculus Rift with the Second Life Viewer. We now have a Viewer ready for beta testers, and if you have an Oculus Rift headset, we’d love to get your feedback.
If you have the Oculus Rift development hardware and would like to help us with feedback on the Viewer integration, please write to firstname.lastname@example.org to apply for the limited beta.
As we blogged about last week, we’re making some changes to our JIRA implementation to make our bug reporting system a more transparent and productive experience. We just wanted to take a moment to let everyone know that these changes are now live!
One of the questions we’ve seen in the past week is how previously submitted issues would be treated - namely, will those also be viewable by everyone and open for comment prior to being triaged?
While we want to make issues visible for the reasons described in our last post, we’re not going to extend this to old issues, because at the time they were created, users knew that those reports would have limited visibility and they may have included sensitive and/or private information. We don’t want to take information that someone thought would be private and suddenly make that visible to everyone, so the new visibility settings will apply only to new issues.
Today, we’re happy to announce some changes to our JIRA implementation - the system we use to collect, track, and take action on bugs reported by users. You’ll see these changes take effect next week.
Recently, this system was working in a way that wasn’t very transparent, and it frankly wasn’t a good experience for the users who care enough about Second Life to try to help improve it, nor was it the best set-up for the Lindens tasked with addressing these issues. So you can see why we’re happy to be changing it!
Moving forward, we’re going to make our JIRA implementation a more transparent experience. All users will be able to see all BUG issues, all the time. You’ll be able to search to see if there are duplicates before submitting an issue, and if there’s a bug that’s particularly important to you, you can contribute your info to it and see when it’s been Accepted and imported to the Linden team.
You’ll also be able to comment. Before an issue is triaged, everyone can comment to help isolate and describe the issue more clearly. Do remember, there are some basic guidelines for participation that need to be followed. Once an issue is Accepted and imported by Linden Lab’s QA team, the original reporter will still be able to comment, as will Lindens and a small team of community triagers - a group that includes some third party Viewer developers and others selected by Linden Lab for having demonstrated skills in this area. This group has been invaluable in helping to keep the bug database orderly and cross-referenced as well as troubleshooting bugs before they’re triaged, and we’re glad to have their continuing help with this process.
Lastly, “New Feature Request” is back! If you’ve got a great idea for a feature, you don’t need to slip it through the system disguised as a bug report - just select the “New Feature Request” category when you submit. Commenting for this category will work just like for bug reports, and submitting improvements through this category will make things much easier for the Linden team reviewing these. Please remember that JIRA is an engineering tool - it’s not meant for policy discussions and the like nor is it a replacement for the Forums, where you can have all kinds of stimulating discussions.
If you’re one of the many who have taken the time to submit a bug report through the JIRA system - thank you! We really appreciate your work in tracking down the issues, and it’s a significant help to us as we continue to improve Second Life.
We think these changes will make for a better, more transparent and more productive experience for all of us, but if you have additional ideas on ways to improve our implementation, you can share them with us in this Forum thread.
As we continue to work on improving the Second Life experience, one challenge we’ve been tackling is the length of the Viewer installation process. No one likes waiting, and now with Project Zipper, you don’t have to!
With the project Viewer available today, there’s really only one thing different - the installation is super fast. Rather than waiting for install to complete, you’ll quickly be in Second Life doing what you love.
Try out Project Zipper with the project Viewer here.
This is still a project Viewer, and if you find bugs while testing it out, please let us know by filing them in BUG project in JIRA.
On deck for immediate release are more improvements to Project Sunshine. New updates include better stability and performance enhancements in retry logic, redundant requests, and better detection for appearance conditions. This release also includes lots of bug fixes and quite a bit of cleanup of old code from the old client-side baking framework.
Additionally, we’ve integrated support for the latest AISv3 code (which will improve how back-end processing of inventory items work). We are just awaiting the latest server components to go live before this feature is fully functioning. More on this soon.
Update to the latest Viewer today and take advantage of the new Sunshine refinements!
Since late in 2012, there's been a project to improve HTTP communications between Viewer and grid services. One of the metrics we've tried to improve is maximum request rate: the number of HTTP requests that can be issued and responded to over a time interval. Improving this figure has been challenging. There are and will always be factors beyond our control, such as the characteristics of the home network, ISP policies and capacities, physical distance to services, and transient network problems, to name a few. However, we can do something about other factors that are under our control.
These factors can be fixed limits intended to implement a rate throttle. Or they can also be side-effects of other decisions, such as connection handling. The impact of fixed limits and connection handling is shown in the diagram below taken from texture and mesh fetching. The lowest curve (red) represents the highest possible request rate when using a unique connection for each HTTP request for a maximum of five (5) concurrent connections. The curve above it (blue) shows the doubling of the request rate limit when HTTP keepalives are used. And both of these limits are strongly affected by packet round-trip (ping) times. Finally, the horizontal line at 100 RPS represents an approximate limit on request rate that's independent of ping time.
At the beginning of this project, most HTTP traffic, texture fetching in particular, was constrained to region 'A'. This was the result of a combination of factors. One was the lack of connection keepalives. This forced a TCP connection handshake prior to every HTTP request. Another was the delay in launching new requests on completion of previous requests.
Released in early 2013, Viewer 3.4.3 included a new library, llcorehttp, to begin to address the above issues. It was foundational in that it didn't attack the above problems directly (nor other problems not discussed here), but it established a structure that would allow the problems to be addressed over a series of releases.
While its goals were modest, the efficiency and latency improvements yielded some nice gains. Texture fetching was the first component to use the new library and throughput and robustness increased while tendencies to stall have disappeared. With these changes, texture fetching solidly entered the 'B' region of the diagram.
Deployed in April, 2013, the DRTSIM-203 server release included the first support for HTTP keepalive connections between viewers and simulator hosts. For small transfers, it halved the number of round-trips required to fetch an object. For larger transfers, the impact of TCP slow-start was also reduced. There were several other beneficial side-effects; the greatest of these was creating fewer TCP connections for a given workload. A high rate of connection creation, called connection churn, is one factor in destabilizing many home routers. Symptoms of this include: router reboots, loss of connectivity, timeouts, and DNS failures.
Keepalive connections were enabled for texture fetches and SSL-encrypted HTTP between viewers and simulator hosts. This raised the ceiling into region 'C,' increasing texture download throughput. Mesh downloads continued to use the original connection scheme, and monitoring and enforcement watchdogs were added to protect common services from monopolization.
We continue to evaluate our HTTP protocols with goals of improving our user experiences. Here are some of our most recent areas of interest.
DRTSIM-216/229 + Viewer DRTVWR-329
Mesh fetching, like texture fetching, is a relatively heavy-weight activity in the viewer. It takes time to acquire these assets and display a scene. So mesh fetches will be the next area to receive the above treatment, putting them in region 'C' as well.
The challenge is that mesh fetching has been built around extremely high connection concurrency. A new capability service, called 'GetMesh2', will supersede the existing 'GetMesh' service. It is designed around low connection concurrency. That, in turn, will permit connection keepalive and even HTTP pipelining while promising higher throughput, lower overhead and improved router stability.
As significant as the above changes are, they still leave most users under a ceiling strongly influenced by ping time. Europe and Asia are usually over 100 ms and South Africa often reports 250 ms. There will always be a 'ping tax' on those who are far from grid services. But there is something we can do: HTTP pipelining.
Pipelining reduces the impact of ping time by issuing multiple HTTP requests at once without waiting for responses. This will allow our servers to keep filling the download stream rather than stalling between requests. Mesh and texture fetching will get first crack at pipelining. There will be new problems to discover. An essential library, libcurl, has only just started supporting pipelining. We will need to update some of our services as well. But those will be solved and asset fetches will begin to operate in region 'D'.
Sky's the Limit
Progress doesn't stop with pipelining. There are more opportunities to push upwards into 'E' and 'F' regions. A few possibilities:
If you participated in the DRTSIM-203 beta test, you had an opportunity to try our 'Happy' regions (TextureTest2H, MeshTest2H, etc.). These regions ran on hosts with elevated limits to see what would happen. The results were positive and invite a permanent increase as HTTP behavior improves.
Updating other HTTP-based services, moving them up the throughput curve. This would include services such as inventory operations and display name lookups, which make heavy, bursty demands.
Looking into other transport protocols. SPDY is interesting and tries to solve some of the challenges Second Life presents. How well it solves them is yet to be seen.
These shouldn’t be taken as commitments or written-in-stone plans, as priorities and goals may shift, but we’ve seen good results from our work on the HTTP project and are looking forward to continuing to improve Second Life’s infrastructure.
A new set of Materials work has become a Release Candidate Viewer. This Viewer includes support for Materials underwater, render support for impostors, and fixes that will address issues with transparency and alpha layers. See the Release Notes for full details.
A quick coral study with Normal and Specular maps shown underwater. Land Impact is 14.
As of today, approximately one fifth of the regions in Second Life have at least one object using Materials, and this is increasing at a rate of about 2% per week. Users with high-end graphics cards are seeing a significant increase in frames per second when Advanced Lighting Model (ALM) is turned on -- we highly recommend that those users turn ALM on. In addition, we will be updating the GPU tables to include newer graphics cards, allowing more users to take advantage of advanced graphics features like Materials.
Thanks to those who have already tried out Materials and submitted a bug report to the MATBUG JIRA project! Once this release candidate has completed the release process, we will be moving any active MATBUGs to the BUG project and track any future issues from the BUG project.
If you’d like to experiment with Materials, download the Materials Release Candidate Viewer today. And don’t forget to check the Good Building Practices wiki for more tips and tricks and the Knowledge Base for updated user documentation.
We are making some improvements to how we release new Viewer features. Our goal is to accelerate the pace at which we bring new features and improvements to all Second Life users by removing what had previously been a bottleneck in our release process.
In a couple of weeks, we’re going to begin following a process similar to what we already do with the simulator. We’ll release more than one new version at the same time in parallel to subsets of users for final validation, and then promote the most important of the best of those to the default Release Viewer when that testing shows it to be ready.
In the past, we have used the 'Second Life Beta Viewer' channel for this final user validation, but because we had only one Beta channel, some features and fixes have been significantly delayed waiting for their 'turn' in the channel. We will now be able to put those Viewers in the regular Release channel to a smaller randomly chosen group, allowing us to test and release new features more quickly.
If a development project wants to put out an early version for testing prior to it being ready for the Release channel, a channel specific to that project (either 'Project <project>' for very early versions, or 'Beta <project>' for more mature ones) will be created, just like we do today. These will be shut down when the project is ready to move to final testing in the Release channel, and users in the early project test channels will automatically be upgraded to the corresponding Release candidate version.
Because several different beta Viewers will be tested concurrently in the Release channel, the ‘Second Life Beta Viewer’ channel will no longer be used; there will be a web page where users who want to try early versions can find them. The existing inworld 'Second Life Beta' group is not affected.
Following a similar process has enabled us to accelerate the pace at which we release improvements to the simulator, and we’re looking forward to using this new process to more quickly introduce new features to the Second Life Viewer.
The Snowstorm project is happy to announce that we have a Materials Project Viewer available for alpha testing of support for Normal and Specular maps on Second Life objects! When generally available, this will enable content creators to make much richer looking materials with less geometric complexity.
This is a very early test version - there are still known bugs, and certainly some that we don't know about yet. We do not recommend using it to manipulate content that you care about. Bugs should be reported in the Materials Project Viewer (MATBUG) project on Jira. If you use it, please leave automatic updates enabled, because we expect to release updates relatively frequently.
Information on how to prepare textures in a way that's compatible with the Materials Project Viewer available on the wiki.
This project is a great example of how we can improve Second Life through collaboration: it includes contributions from members of the Exodus, Firestorm, and Catznip development teams, as well as from Linden Lab. We don't yet have a target date for when it will be in the regular Release Viewer, but we're working toward that, and your help testing will make that happen sooner and with higher quality.
Engineering and Operations at Linden Lab are constantly striving to make performance improvements to Second Life and we want to share the results of a recent one.
An optimization on a single query against the read pool of one of the main core database clusters has resulted in nearly a 75% reduction in load on that cluster, improving numerous systems including group functions and teleports. There has been a 7% improvement in teleport performance in peak concurrency hours and an 86% reduction in group query times.
This is only one of the performance changes we will be making over the next few months, so keep an eye out.
User-submitted bug reports help improve the Second Life experience for all Residents, so we greatly appreciate all of you who take the time to provide this invaluable information to us.
Because we want to make it even easier to report bugs, today we are making some changes that will streamline the bug reporting process, allowing us to more quickly collect information and respond to issues.
Following is a summary of the JIRA changes:
- All bugs should now be filed in the new BUG project, using the more streamlined submission form.
- Second Life users will only see their own reported issues. When a Bug reaches the "Been Triaged" status, they will no longer be able to add comments to their issue.
- Once a Bug reaches the “Accepted” or “Closed” status, it will not be updated. You can watch the Release Notes to see when and if a fix has been released for your issue.
- Existing JIRAs will remain publicly visible. We will continue to review and work through these.
To those of you who have taken the time to alert us to bugs and provided the information we need to fix them -- thank you! We hope that you will continue to help us improve Second Life, and this new process should make it easier for all of us. Ideas about how we can continue to improve the bug reporting process can be shared here.
For more information, visit:
How to report a BUG (Knowledge Base Article):
Bug Tracker (wiki page):
Bug Tracker Status/Resolutions (wiki page):
One of the challenges that virtual world creators face is the trade-off between rich visual detail and geometric complexity. Ideally, by adding more and smaller faces to an object, a designer can model different surface textures and create realistic variations in the interplay of light and shadow. However, adding faces also quickly increases the size of the model and its rendering cost. Normal and Specular Maps are ways to address this by allowing for the appearance of a complex surface without actually modeling fine scale geometry.
A Normal Map is an image where the color codes indicate how the renderer should reflect light from each pixel on a surface by modifying the direction that the pixel "faces" (imagine that each pixel could be turned on tiny pivots). This means that pixels on a simple surface can be rendered so that they appear to have much more detail than the actual geometry and at much lower rendering cost. Light and shadow are rendered as though the surface had depth and physical texture, simulating roughness, bumps, and even edges and additional faces.
Similarly, a Specular Map allows each pixel to have its own degree of reflectivity, so that some parts of a single face reflect sharply, while adjacent pixels can be dull.
The open source developers of the Exodus Viewer are contributing Viewer support for Normal and Specular Maps, as well as some additional controls for how light reflects from faces. Linden Lab is developing the server side support so that this powerful tool will be available in Second Life.
Design and development are under way. Watch this blog and the Snowstorm Viewers page for information on when test Viewers with these new capabilities become available.
For additional information, or to learn more about how you can participate in the open source program, please contact Oz@lindenlab.com.
Somewhere below the regolith in the Linden Lab secret lunar base, some of the Lab’s top minds have been tackling performance issues in Second Life. The areas of focus range from infrastructure improvements to server-side texture compositing to object caching.
This year, Linden Lab is making the single largest capital investment in new server hardware upgrades in the history of the company. This new hardware will give residents better performance and more reliability. Additionally, we are converting from three co-locations to two co-locations. This will significantly reduce our inter-co-location latency and further enhance simulator performance.
The Shining project is the name given to a recent Lab effort to identify and measure delays and other problems with streaming avatars and objects and to propose and implement solutions. During the Shining project, several small improvements have been identified and deployed. The next small improvement from Shining to be deployed will be pre-rendering the starter Avatars to improve the new resident experience.
As a result of the Shining investigation, the project has been split into three larger performance projects: Project Sunshine, Object Caching, and HTTP Library.
All together, the hardware upgrades and the Shining projects should improve avatar and object streaming speeds significantly. Depending on your unique situation, your mileage will vary.
Project Sunshine (Server-side Baked Texture Generation & Storage)
Currently, all client viewers are responsible for compositing their own Avatar textures, then sending the results back to the Sim for other viewers to access. This method can lead to slowdowns and errors. The actual calculations for compositing textures are straightforward and not particularly time-consuming. However, in order for the viewer to do the calculations it first has to download a lot of individual assets from the Sim, and must then upload large results back to the Sim. This pushes a lot of bits through the Sim / Viewer connection, which can be slow and unreliable.
Depending on client hardware to do the compositing and uploading of the resulting baked textures can introduce erroneous results (like way too much pink). In order to handle these errors, a number of retry and fallback mechanisms have been put in place. This adds further load and overhead to the whole system.
Project Sunshine stands up a Texture Compositing server that is separate from the Sims servers. When a Viewer needs to render an Avatar, it sends a message to the Sim, which in turn sends a message to the Texture Compositing Server. The Texture Server then performs the texture compositing and sends the results back to the Viewer.
The Texture Server has access to a database of previously generated Avatar textures. If the Sim is asking for the baked textures for an Avatar that has been previously calculated, the Texture Server can pull the results out of the database instead of recalculating. Even if it’s not the same Avatar, but the set of clothes is the same, the Texture Server can use the previously calculated results.
An additional advantage of having a Texture Generation server is that the heavy HTTP communication about the individual assets that make up an Avatar will happen on reliable, fast internal network connections instead of external connections.
Object Caching & Interest Lists
Caching objects on a resident’s hard drive is intended to provide faster drawing of objects that are most likely to be redrawn later in a resident’s session or in the next session.
The current object cache implementation has room for improvement. Currently, the viewer only caches objects at the point the resident disconnects from a region and only objects currently in view are cached. The current caching communication between Sim and Viewer when a scene is initially being loaded is inefficient and resource intensive.
The new system is designed to make sure that a viewer never has to download object information that it already has and is still valid. This will reduce the load on the Sim and on the Sim / Viewer connection.
With the new system, when the Viewer connects to a region, the Sim first says to the Viewer, “Here’s a list of all my object groups by UUID.” The Viewer checks its cache and replies, “Here are the timestamps for the object groups that I found in my cache.” (If an object group is not found in the cache, the timestamp is 0.) The Sim checks the Viewer’s timestamps against its own data and responds, “Here’s a list of object groups in your cache that are still valid.” While the Viewer starts drawing valid objects from its cache, the Sim starts streaming the new object data that the Viewer needs, based on the particular Viewer’s object priorities.
Top performance for the Shining initiatives depend on the speed and reliability of HTTP messages initiated by both the Viewers and the Sims. The third part of the Shining project, the new HTTP Library, implements modern best practices in messaging, connection management, and error recovery.
With the initial release of mesh in August, Linden Lab introduced an entirely new way to measure the impact that objects have in Second Life. Since then, we’ve seen that providing customers with this additional insight has allowed them to design with more performance efficiency.
Now, with this new release of mesh, creators get a more detailed understanding of the actual impact of objects on the world around them. Being able to see Resource Weights, as well as how they drive Land Impact, helps creators more easily build responsive and low-lag locations, improving the overall resident experience.
To better show what effect an object will have inside Second Life, we have developed four weight values: Download, Physics, Server, and Display. Each of these values can be used by creators to determine the impact that a given object will have on performance from different perspectives (e.g. client vs. server) and under different circumstances.
Here’s how they break down:
- Download measures how much bandwidth we need to send out the information about an object so that everyone can see it.
- Physics is a value that shows how difficult the object is for the physics and Havok processes to create, track, and update the object.
- Server is a value that shows you how difficult the object is for the server processes to create, track, and update the object.
- Land Impact (formerly PE or Prim Equivalence) is the highest of the above server-side weights. It defines a theoretical limit under which performance remains acceptable for residents.
- Render is a value that shows you how difficult it is for the client renderer (your viewer) to draw the object. Because it’s related to the client, this is informational only. This is the same calculation used to create Avatar Rendering Weight.
Knolwedge Base Article on Weights and Impact.
How you see it in the viewer:
We hope understanding the relationship between objects and performance will help land owners and creators make even more efficient content. As always, we welcome feedback to help us make this information even easier to understand.
Linden Lab is excited to announce the latest updates on one of our coolest new features — mesh. With mesh, you’ll be able to create, build and beautify even more incredible objects inworld! You’ll find a new Mesh Enablement functionality tab in the My Account section of your Account screen. It’s just another part of the innovation and imagination that makes Second Life the magical place it is.Read more...
Today, we released the latest version of the SL Viewer into general availability with Voice included in Basic mode. All that new users have to do is click on a simple microphone button--a well-understood symbol for voice/sound. For those seasoned Residents, the microphone button is where the Speak button is in Advanced mode.
Look for more improvements to the SL Viewer coming soon. In the meantime, download the latest SL Viewer and give Basic mode a try.
• Download SL Viewer with Basic and Advanced:
◦ Windows | Mac | Linux
• SL Viewer Release Notes
• Feedback Forum on SL Viewer Basic Mode
• Twitter hashtag #slviewer
Enhanced Avatar Physics in the new SL Viewer Beta gives your avatar more realistic body movement. Yes, bouncing and jiggling.Read more...
Today we launched Viewer 2.5, now out of Beta. The most significant update is a new, web-based profile system, which allows profiles to be viewed and edited both on the web and in the Viewer. For example, here's mine. Please note this is just a starting point for the web-based profiles; we’ll be doing a lot of work to refine the usability and make them richer over time.
In response to your feedback from the early beta versions of Viewer 2.5, we've added some privacy settings that will allow you to control just how public your profile is. Once you’ve logged in, click on “Privacy Settings” in the upper right corner of your profile. Group settings set in the Viewer will be overridden by these group privacy settings.
- "Everyone" means that the information is available to the whole Internet and can be picked up by search engines.
- "Second Life" means that the information is available to all Second Life Residents who are logged into the website or inworld. This is the default for all existing Residents using Viewer 2.5.
- "Friends" means that only your Second Life friends can see the information on the web and inworld.
This is why we have a beta process--to address concerns and improve your user experience. We will continue to iterate as we get more feedback. Thank you for all your help and comments. Please attend the Viewer 2 User Group meetings if you would like to share your thoughts and feedback directly with me and the Snowstorm team.
Viewer 2.5 also has some other new features. The one I like best is that you can now have your Favorite landmarks also appear on the login screen, so that you can log directly into your favorite locations. Torley made a video about this, so check it out! We've also improved some texturing performance and fixed another batch of bugs. Watching the internal data, we've already seen a noticeable improvement in stability and performance--on par with Viewer 1.23.
Download Viewer 2.5, try it out, and keep the feedback coming! And, if you Twitter, please use the hashtag #slviewer2.
In addition to many grid enhancements that FJ shared last week in his technology update blog post, the Viewer 2.5 beta cycle begins today. Viewer 2.5 Beta 1 is now available and includes several new features, usability improvements, and bug fixes.
Improved Web and Viewer Resident Profiles
The biggest change in Viewer 2.5 Beta 1 is that we've replaced the Viewer-based profiles with web-based profiles. You can access the redesigned profiles on the web using your Second Life user id and in the Viewer itself. For example, here’s mine. In the Viewer, profiles now open in a web browser and you can open as many profiles as you like at the same time. And, if you wish, you can even connect other social identities from Twitter, Facebook, and LinkedIn to your profile.
Logging In to your Favorite Locations
A long-standing feature request has been the ability to access some of your favorite Landmarks from the login screen, so you can quickly teleport to places inworld. In the Viewer 2.5 Beta, we've added a preference that gives you access to the Landmarks on your Favorites Bar from the Login Screen. To use this feature, go to Preferences > Privacy and select the check box labeled, "Show my Favorite Landmarks at Login." If you log off and then restart your Viewer, then you'll see a list of your Favorite Landmarks in the "Start At" drop-down box on the login screen. Also, when this feature is enabled and you share a computer account (login) with other people, they will see your list of Favorites if they run the Viewer 2.5 Beta.
Decompression Performance Improvements
For a long time, we’ve used an older version of the Kakadu library known as KDU to load textures and images in the Viewer. The KDU Library has been updated to version 6.4 in the Viewer 2.5 Beta, which allows the Viewer to take advantage of significant decompression performance gains; our preliminary measurements gauge the performance gain at 30% for decoding images, which should translate to a visible improvement in rezzing time for complex scenes.
Please note that per our support policy, we’re also deprecating Viewer 2.1.1 today. If you’ve been running Viewer 2.1.1, you’ll be prompted to download the latest Viewer release. To stay up to date on future deprecation announcements, please subscribe to @secondlife on Twitter.
As always, we want to know what you think of the latest beta so please keep that feedback coming. Share your impressions with us on Twitter and use the hashtag #SLViewer2.
As we begin 2011, I want to share the progress that we’re making on several important technology enhancements that I discussed in my last post. As I mentioned, we are focused on improving the overall performance of Second Life while addressing some long standing limitations such as raising group limits, improving the chat system, and reducing lag.
Group Limits Raised to 42 Today
In October, we committed to increase group limits from the current 25 up to 40 in the first quarter of 2011. As of today, group limits have been raised to 42! To add groups beyond the previous limit of 25, you must be using Viewer 2.4 (or a more recent version). And if you’re still using Viewer 1.23, or a third-party viewer based on Viewer 1.23 code, then you can add more groups in Viewer 2.4 and they will still be accessible when you switch back to Viewer 1.23.
That said, if there is an unexpected load, then we may need to lower the group limitation to maintain acceptable performance levels across the grid. If we decide to do that, then any Residents who have up to 42 groups will not lose their memberships. But, other Residents will not be able to exceed the new limit.
Group Chat System Will Launch Gridwide By March 31st
We were set to deploy a prototype of the new group chat system in December, but last minute licensing issues were found with our chosen open source library. Now that a solution is in place, we expect to have the prototype available by the end of this month and an industry standard and high performing group chat system available by the end of this quarter.
Performance Improvements When Teleporting and Crossing Regions
As you teleport, or cross regions, all of your avatar data (often a very large amount of data) needs to be processed by both source and destination regions. In order to streamline this process, we are now compressing avatar information, making your teleports and region crossings faster and more reliable. In fact, we’ve found that teleport failures, due to avatar complexity, have dropped 40%. See the graph below for a more detailed view showing how much we’ve shortened region crossing time.
Viewer 2.5 Beta Releasing Soon
We will soon launch the latest version of the Second Life Viewer -- Viewer 2.5 Beta. In addition to more performance and stability improvements, we’ve added enhanced web-based profiles, accessible both on the web and in the Viewer. And, if you wish, you can even connect your Second Life profile to other social identities including Twitter, Facebook, and LinkedIn! You will also have the option in Preferences to choose your first inworld destination from the saved Landmarks in your Favorites Bar. This is very handy when you need to get to a specific destination quickly.
Also, one of the most important new features added to Viewer 2.4 was the auto-updater capability. If you're using Viewer 2.4 or higher, then you won’t be inconvenienced by the notification and download process when we release a new Viewer.
Planning to Implement Significant Grid Infrastructure Enhancements in 2011
We’re planning significant grid infrastructure enhancements throughout the year including technologies to speed server-side rendering (SSR) and server virtualization (web and simulator services). We are also exploring new storage and asset delivery systems. Some of the benefits will not always be noticeable, but they are foundational platform changes that set the stage for rapid performance and scalability improvements. We will continue to keep you updated as we roll out these systems.
I’m pleased with the progress that we made across the platform last year and I'm looking ahead to newer technologies that we will deploy in 2011 to enhance your Second Life experience. As always, I'll be watching for your feedback and thank you for making Second Life such an amazing place.
As of today, SL Viewer 2.4 is the default Second Life Viewer download for new Residents! As was mentioned in the Viewer 2.4 Beta blog post, this is largely a maintenance release focused on improving user experience, stability, and performance. This release does, however, have a few important changes and additions, including the following:
A Cleaner User Experience and More Customization Options in Preferences
Throughout 2010, we’ve added many new Preferences to the Viewer and it was time to not only reorganize and clean up the layout, but also add popular customization options, such as:
- Color and transparency options that allow you to change the look of Viewer windows
- Options to easily enable the Advanced and Developer menus
- A new preference that you can use to turn group and IM chat pop-ups on or off whenever you like
Support for External Text Editors when Working with LSL
Here’s a good one for content creators and developers. Now, you can use your favorite script editor to edit LSL scripts outside of the Viewer. In Viewer 2.4, enabling the new ‘ExternalEditor’ debug setting will allow you to specify the path to your own text editor (for example: /Applications/TextMate.app/Contents/MacOS/TextMate
We continue to make numerous improvements to Viewer graphics as we move closer to integrating Mesh Import into Viewer 2. You can see more detail in the Release Notes, but we’ve improved antialiasing, anisotropic filtering, snapshots, and we’ll see even more in upcoming releases.
Viewer 2 performance is improving, as shown by steadily decreasing crash rates that are now close to those for Viewer 1.23, but we know there’s still more to do. So if you do crash, please be sure to send us a crash report, as they are essential to helping us understand where issues occur, which allows us to better prioritize our work.
An Auto-Updater Tool
We now have an auto-updater for Viewer 2! This means that whenever a new Viewer is released, the next one being Viewer 2.5, it automatically downloads the newest software in the background and offers you the ability to upgrade when it’s ready. You can still decide whether you want to install the new version or not, but the auto-updater will help ensure that you’re always using the most up-to-date version of the SL Viewer. Keep in mind, the auto-updater will only work with optional updates. Mandatory updates will continue to work as they have in the past.
Finally, to those of you who gave us feedback on previous versions of Viewer 2, we offer a heartfelt thank you for all your bug reports, forum and blog comments, and Tweets using the #slviewer2 hashtag. Your input is invaluable as we continue to prioritize our 2011 Viewer 2 roadmap.
So download Viewer 2.4 -- and keep that feedback coming!
- Tips for Reducing Viewer Crashes
- Seeking Creators for the Experience Keys Beta
The Recent Unpleasant
Updates to the Second Life System Requiremen
Try Out the New Transactio
n History Page
- Account Safety and the Heartbleed OpenSSL Bug
- SL Share’s Photo Upload Feature Is Back!
SL Share’s Photo Upload Feature is Temporaril
Second Life’s Oculus Rift Integratio
n is Ready for...
- JIRA Changes Are Now Live