Tools and Technology

We've been working hard on the Uplift of Second Life. If you have not been following this project, that's what we're calling the migration of our Second Life simulators, services, and websites from a private datacenter to hosting in The Cloud (Amazon Web Services). It's a massive, complicated project that I've previously compared to converting a steam-driven railroad to a maglev monorail -- without ever stopping the train. This undertaking has at times been smooth sailing, at other times a very bumpy ride. We wanted to share some more of the story with you.

Our goal has been to move SL incrementally to give ourselves the best chance of minimizing awareness among the residents that these changes were happening. We feel we’ve done better than we expected, but of course it’s the bumps in the road that are most noticeable to our residents. We apologize for recent service disruptions, although what’s perhaps not apparent is the progress we’ve made -- and the improvements in performance that have quietly taken place.

First, the rough spots:

• Region Crossings
  One of the first troubles we found was that region crossings were significantly worse between a cloud region and a datacenter region. We did a deep dive into the code for objects (boats, cars, planes, etc) and produced an improvement that made them significantly faster and more reliable even within the datacenter. This has been applied to all regions already and was a good step forward.
• Group Chat stalls
  Many users have reported that they are not able to get messages in some of their groups; we're very much aware of the problem. The start of those problems does coincide with when the chat service was uplifted; unfortunately the problems did not become clear until moving that service back to the datacenter was not an option. We haven’t been able to get that fixed as quickly as we would like, but the good news is that we have some changes nearly ready that we think may improve the service and will certainly provide us with better information to diagnose it if it isn't fixed. Those changes are live on the Beta grid now and should move to the main grid very soon.
• Bake Failures
  Wednesday and especially Thursday of this past week were bad days for avatar appearance, and we're very much aware of how important that is. The avatar bake service has actually been uplifted for some time - it wasn't moving it that caused the problem, but another change to a related service. The good news is that thanks to a great cross-team effort during those two days we were able to determine why an apparently unrelated simulator update triggered the problem and got a fix deployed Thursday night. 
• Increased Teleport Failures
  We have seen a slight increase in the frequency of teleport failures. I know that if it's happened to you it probably doesn't feel like a "slight" problem, especially since it appears to be true that if it's happened to someone once, it tends to keep happening for a while. Measured over the entire grid, it's just under two percentage points, but even that is unacceptable. We're less sure of the specific causes for this (including whether or not it's Uplift related), but are improving our ability to collect data on it and are very much focused on finding and fixing the problem whatever it is.
• Marketplace & Stipend Glitches
  We've had some challenges related to uplift for both the Marketplace and the service that pays Premium Stipends. Marketplace had to be returned to the datacenter yesterday, but we'll correct the problems that required the rollback and get it done soon. The Stipends issues were both good and bad for users; there were some delays, but on the other hand we sent some users extra stipends (our fault, you win - we aren't taking them back); those problems are, we believe, solved now.

Perhaps the above makes it sound as though Uplift is in trouble. While this week in particular has seen some bumps in the road, it's actually going well overall. Lots of the infrastructure you don't interact with directly, and some you do, has been uplifted and has worked smoothly.

For a few weeks, almost all of the regions on the Beta grid have been running in the cloud, and over the last couple of weeks we've uplifted around a hundred regions on the main grid. Performance of those regions has been very very good, and stability has been excellent. We expect to be uplifting more regions in the next few working days (if you own a region you'd like included, submit a Support Ticket and we'll make it happen). Uplift of the Release Candidate regions, which will bring the count into the thousands, will begin soon. When we're confident that uplifted regions are working well at that larger scale, we'll be in a position to resume region sales, so if you've been waiting - the wait is almost over.

Overall, the Uplift project is on track to be complete or very nearly so by the end of this year (yes, 2020… I know I've said "fall" before and people have noted that I didn't say what year 🙂 ; the leaves haven't finished falling at my house yet…). It's likely that there will be other (hopefully small) temporary disruptions during this process, but we promise we'll do all we can to avoid them and fix them as fast as we can. This migration sets the stage for some significant improvements to Second Life and positions us to be able to grow the world well into the future.
 

Hello Residents!

Well, this is not the way we wanted the past week to go -- multiple outages in the same week! Not fun for anyone. Here’s what happened.

Login and Inventory Issues: Monday, 1/24 - Friday, 1/28
On Monday, January 24, residents’ inventories stopped responding to requests. Some residents were unable to log in as a result. We identified the affected accounts, addressed the problem and restored services to full operation in a little under 3 hours. 

Beginning on Wednesday of this week, we received reports of intermittent inventory issues – failures to rez, unpack and delete inventory items; and intermittent errors upon login.

When the number of reports increased, it became clear that this was an outage. Several Lindens jumped in to diagnose the reports. After some digging, we discovered our back end infrastructure was being overloaded. Once this was resolved, the positive impact was almost immediate. The data now looks good -- on the back end and we are no longer receiving inventory reports.

We’re taking steps, including a deploy late last week, to prevent these issues in the future - and we have already seen progress in making the service more robust.

Weekly Rolling Restarts: Tuesday, 1/25
Every week we restart the simulator servers to keep them running smoothly. We only restart a certain number at the same time, allow them to finish, then start another batch. That’s why we call them “rolling restarts.” But on Tuesday a recent upgrade to the simulators meant that the usual number of simultaneous restarts was too much for the system to handle. The result was load spikes and numerous regions going down. Ultimately more than 12,000 regions were stuck in restart mode, which is 40% of the grid. Not good!

The team came together quickly to bring simulators back up in smaller batches, and then manually fixed blocks of regions. After some trials and monitoring, we found a smaller number of concurrent restarts that worked better. By 1:00 PM all regions were restored and operating normally. We apologize for this lengthy outage!

Advance… Retreat!: Wednesday, 1/26
We had planned a deploy to the production grid that would help us gather information on group chat performance. Unfortunately, the procedure used during testing did not work in production. We figured -- OK, we’ll just roll back. But then the rollback itself had problems!  Residents experienced issues with login, group chat, and presence information. The team was able to isolate the problem and complete the rollback, getting a few more gray hairs in the process. We’re going to take what we’ve learned and do a better deploy that will give us the information to improve group chat in the long run.

Last week was no fun for a lot of people at the Lab. Thank you for your patience. We really don’t like interrupting your enjoyment of Second Life. Here’s hoping we don’t have to come back with another one of these blog posts for quite some time! 

With love, 
The Lindens
 

We will be performing an update to the Second Life Login service on Monday, April 11th beginning at approximately 9:00 AM PDT and expected to complete before 10:00 AM PDT. Viewer logins will be unavailable for a portion of this maintenance period.

This update includes a fix to new device email notifications for newly created accounts, performance improvements for the viewer login handshake, and improvements to metrics and diagnostics for our internal tools. Most of these changes are behind the scenes and should not change the login experience for Residents once the maintenance is completed.

If you experience viewer login issues during the deployment window, please wait and try again later. Keep an eye on the Second Life Status Blog for updates. Thank you for your patience!

UPDATE: The maintenance is now completed.

You may have noticed – or fallen victim to -- the unfortunate increase in server crashes last week. We apologize to everyone affected! 

Here’s what happened:

• On Thursday, June 30, just before the July 4 holiday weekend in the U.S, we released a change to my.secondlife.com.
• Early on the following Tuesday, we noticed and started investigating a significant increase in server crashes. We also started receiving Support reports about these.
• We determined that the web update in my.secondlife.com  included a library update which, unknown to us, dropped support for the texture format our servers expect.
• This led to Viewers being unable to deal with newly updated textures, and under some circumstances the Server would attempt to be more helpful in opening those, than it should have been - and crashing. This is why some residents saw problems with their profile photos. 

On Tuesday July 5, after we investigated and confirmed what was happening, the team quickly rolled back the web site change to prevent the creation of more broken textures. The web team identified these textures, then ran a script to convert them. This was completed by Thursday morning, July 7.

The server team fast-tracked a fix to prevent servers from crashing. We were able to QA and release the fix to RC channels on the afternoon of Thursday, July 7 and then to the main SLS + Events channels on Friday, July 8.

Lessons:

• Second Life is big and complicated and fails in unexpected ways (we knew that!)
• Library upgrades may need additional testing for dependencies we couldn’t imagine.
• We work really well together when Second Life fails in unexpected ways                                                    

Again, we apologize to everyone affected!

Photo by Alexa Linden

The idea

Wouldn’t it be cool if you could animate your avatar in real time?  What if you could wave your arm and your avatar could mimic your motions?  Or imagine if your avatar could reach out and touch something inworld or perform animations?  Linden Lab is exploring these possibilities with an experimental feature called “Puppetry.”

We have been working on this feature for some time and now we are ready to open it up to the Second Life community for further development and to find out what amazing things our creators will do with this new technology.

The codebase is alpha level and does contain its share of rough edges that need refinement, however the project is functionally complete and it is possible for the scriptors and creators of Second Life to start to try it out.

See the section below “How to participate” to learn how to use Puppetry yourself.

Take a Look

We have some basic things working with a webcam and Second Life but there's more to do before it's as animated as we want.

Puppetry Technology

Puppetry accepts target transforms for avatar skeleton bones and uses inverse kinematics (IK) to place the connecting bones in order for the specified bones to reach their targets.  For example the position and orientation “goal” of the hand could be specified and IK would be used to compute how the forearm, elbow, upper arm, and shoulder should be positioned to achieve it. The IK calculation can be tricky to get right and is a work in progress. 

The target data is supplied by a plug-in that runs as a separate process and communicates with the viewer through the LLSD Event API Plug-in (LEAP) system.  This is a lesser known functionality of the Viewer which has been around for a while but has, until now, only been used for automated test and update purposes.

The Viewer transmits the Puppetry data to the region server, which broadcasts it to other Puppetry capable Viewers nearby.  The receiving Viewers use the same IK calculations to animate avatars in view.

For more details about the Puppetry technology, take a look at the Knowledge Base article Puppetry : How it Works

Uses and Possibilities

We are excited about Puppetry’s potential to change the way we interact inside Second Life.  For example, using a webcam to track your face and hands could allow your avatar to mimic your face animations and finger movement, or more natural positioning of the avatar’s hands and feet against in-world objects might also be possible.  Alternative hardware could be used to feed information into Second Life to animate your avatar - a game controller or mocap equipment.  There's a lot to explore and try, and we invite the Second Life community to be involved in exploring the direction of this feature.

How to participate

The Puppetry feature requires a project viewer and can only be used on supporting Regions.  Download the project Viewer at the Alternate Viewers page.  Regions with Puppetry support exist on the  Second Life Preview Grid and are named: Bunraku, Marionette, and Castelet.

When using the Puppetry Viewer in one of those regions, if someone there is sending Puppetry data you should see their avatar animated accordingly.  To control your own avatar with Puppetry it's a bit more work to set up the system.  You need: a working Python3 installation, a plug-in script to run, and any Python modules it requires.  If you are interested and adventurous: please give it a try.   More detailed instructions can be found on the Puppetry Development page.

What's next

We look forward to seeing what our creators do with the new Puppetry technology. Compared to other features we have introduced, it’s quite experimental and rough around the edges, so please be patient!  We will keep refining it, but before we go further we wanted to get our residents’ thoughts.

We will be hosting an open discussion inworld on Thursday, Sept 8 1:00PM SLT at the Bunraku, Marionette, and Castelet regions on the Preview Grid.    We're also happy to talk about this at the upcoming Server User Group or Content Creator meetings.  Come by, let us know what you think, and hear about our future plans!

The Puppetry User Group meeting will be a bi-weekly meeting discussing an experimental new feature that allows hardware and other software to connect to Second Life and provide animation data for an avatar. The purpose of these meetings is to connect with developers and coders who might want to experiment with it. 

Learn more about Puppetry in our blog post.

Join the team on Thursday at 13:00 PT on the Aditi beta grid, Castelet region. 
 

The Puppetry User Group meeting will be a bi-weekly meeting discussing an experimental new feature that allows hardware and other software to connect to Second Life and provide animation data for an avatar. The purpose of these meetings is to connect with developers and coders who might want to experiment with it. 

Learn more about Puppetry in our blog post.

Join the team on Thursday at 13:00 PT on the Aditi beta grid, Castelet region. 
 

Linden Lab open-sourced Second Life's client on January 8th, 2007. In the 15 years since, open source contributions have led to countless new features and bug fixes that have markedly improved the life of our virtual world. Without open source, SL would not be as vibrant, enduring and successful as it is today.

As part of an effort to improve and modernize our tooling and better support the open source community, I'm happy to announce that Second Life's source code is moving to GitHub, the world's most popular version control hosting platform. You can find our open source repositories in their new home at github.com/secondlife.

As the de facto home of many open source projects, GitHub likely needs no introduction to anyone who works with open source software. Its features and market position make it the best option for Second Life going forward.

Improved Developer Experience

While migrating code to GitHub is an important step towards improving Second Life open source contributors' development experience, it is not a complete picture. We will be making several changes to streamline and modernize our engineering processes: 

Simpler Contributor License Agreements

A Contributor License Agreement (CLA) is a commonly used legal contract between an individual or group and an organization accepting contributions. This process has been modernized, adopting simpler, more developer-friendly language and allowing contributors to sign the contract as part of pull requests. No PDF, no fax. 😉

GitHub Actions CI/CD

Second Life will be moving its continuous integration and deployment (CI/CD) from a self-hosted TeamCity instance to GitHub Actions. This change, to be completed in 2023, will open up new possibilities for our engineering team and the open source community:

• Our build tooling will be more transparent, allowing OSS contributors to easily fork and build things like autobuild libraries
• Pull requests will feature automated merge checks such as linting and automated testing
• Services can be built using newer technology, such as aarch64 servers

A simple pipeline to test and publish autobuild.

Summary

While these changes will not impact most users, they will make contributing to Second Life easier and allow the platform to evolve faster. Linden Lab is a uniquely open company, holding regular in-world user groups and allowing its residents to contribute in meaningful ways. We are committed to this philosophy and would love to join you in helping make our shared virtual world even better.

Do you want to contribute to Second Life? Attend one of the user groups on our public calendar, submit an idea for a new feature, or, of course, check out our code on GitHub!

Signal Linden
Engineer, Second Life Server

Greetings Residents! Please note that with the release of the Second Life Viewer Maintenance Preferences, Positions and Paste fixes (build 6.6.8.576863) we are making the decision to sort the alpha blending of rigged mesh attachments by the priority assigned to each attachment point. For example, the Chest attachment ID is 1 and is treated with highest priority.

We know this change will require that some alpha blended rigged mesh content will need to be moved to new attachment points. Residents should be able to do this for themselves in many cases, but content creators may need to adjust their product attachment points to account for the new changes.

One of our Residents has written their own blog post explaining this in much greater detail and can be found at: https://beqsother.blogspot.com/2022/11/alpha-blend-issues-get-them-sorted.html

As always, if you believe you have found a bug, please file a Bug report at https://jira.secondlife.com 

The Puppetry User Group meeting is a bi-weekly meeting discussing an experimental new feature that allows hardware and other software to connect to Second Life and provide animation data for an avatar. The purpose of these meetings is to connect with developers and coders who might want to experiment with it. 

Learn more about Puppetry in our blog post.

Join the team every other Thursday at 13:00 PT on the Aditi beta grid, Castelet region. 

Keep up to date with all of our user group meetings on our wiki and public calendar.

Hello!

This Wednesday, February 1st, was a rocky one for Second Life, especially for regions on our Release Candidate (RC) channels. As with many large-scale services, we experience service interruptions from time to time, but yesterday's problems were unique and noteworthy enough that they deserve a sincere apology and more thorough explanation.

Summary

• The incident lasted 6 hours
• 7029 regions were reverted to a previous stable version
• The root cause was a bug in object inventory behavior that was introduced by new code. This bug caused some objects' scripts to enter a non-running state.
• We are making changes to our deploy and testing processes to make events like this less likely to happen in the future.

Timeline

Second Life deploys simulator changes in two major phases: first, to Release Candidate (RC) channels which comprise roughly one-third of all production regions, then to the remainder on a main Second Life Server (SLS) channel. Before changes are deployed to RC, they go through several develop-test cycles, including deployments and testing on staging environments.

RC rolls are regularly scheduled for 7AM Pacific (SLT) every Wednesday. While conducting one of these deployments, we received initial reports of issues relating to script behavior at 8:30AM. We immediately started investigating, and by 9AM it was clear that we needed to stop the roll, because the issue was pervasive and clearly related to the new changes. As we do in these situations, we declared an “Incident” and an Incident Commander took charge. 

By 9:30 we started a rollback, reverting the affected simulators to their previous version. We also evaluated what additional actions we would need to take, as it was unclear that a rollback alone would start the scripts which had stopped. This quickly proved to be the case, and we came up with a new plan - one that would ensure that scripts would perform as expected going forward, but possibly undo the changes our residents had been making since the time we had introduced the bug. Although this meant more downtime, it prevented further content loss, and proved to be the best way to put the grid back in order. The team came up with a quick, clean, and efficient way to achieve this and get everyone back on track. 

By 12PM the decision was made to take this direction. By 1:15PM the code was complete, by 1:40PM we confirmed that it worked. By 2:25PM all regions were brought back up. 

Of course after an event like this there’s cleanup to be done, and we had quite a few Lindens reviewing the fallout and taking care of our Residents. Our amazing support team was working with creators whose operations were affected. 

Trouble doesn’t come alone: we had to deal with another outage at the same time - this one on the web side of things.  Some of our services were getting overwhelmed. We added a few hamsters to those wheels, and they seem to be turning just fine again. 

Technical Details

What happened? To quote part of our internal discussion, “scripts are weird”:

Quote

Sometimes we refer to scripts in an object by a globally unique identifier, and sometimes we refer to scripts in an object by an object-local identifier. There was an issue in the version we deployed on 2023-02-01 where object-local script identifiers would always refer to the first script in an object, causing only that script to be marked as running and receive events like touch and linked_message.

Conclusion

What remains after an outage? We write up an internal “postmortem” report: what happened, what lessons we learned, and what future measures we’re going to put in place. We share it widely, so that we can all learn from it. On days like this, we also write a blog post to let you all know what happened as well. 

Of all the reasons to write a blog, a postmortem ranks pretty low for me, because it means something went seriously wrong. But of all the reasons I love working here, watching everyone come together during an outage ranks pretty high. I’m grateful to know that while we may make mistakes, we will not sweep them under the rug, nor look for someone to blame - we will come together and make it right. 

Thank you for your patience and understanding. This is your world, and we hope to take better care of it. 

Grumpity and the Second Life team

Hello Residents!

The Second Life viewer and servers have added a feature for group chat, providing recent message history. 

Have you ever logged into Second Life and your group chat opens with the punchline of a joke? Now you’ll be able to read back and see what it was. The chat servers save messages from the last hour, so you won’t miss what’s happening.

A viewer update is required to use this new feature, and will be part of a future release (headed to Maintenance-R soon - watch release notes here). 

The viewer code is available now in our public repo - explore here: https://github.com/secondlife/viewer/tree/DRTVWR-573-maint-R 

We look forward to your feedback or any issues with this new feature.

Good morning everyone,

At the end of last year we began the task of moving Marketplace search to an up-to-date version of ElasticSearch, the application we use to do full text searches.  We apologize for some of the bumps you may have experienced in the last few weeks as we make this transition.

The Marketplace search backend was showing its age. We’ve upgraded the technology and we’ve dusted off both our indexing and searching filters. We were running ElasticSearch version 2.3 and needed to update to ElasticSearch version 8.4.  Our goal was to bring Marketplace search results closer to web search and make page loads faster, in order to improve your shopping and selling experience.

In practical terms this means:

• Merchant and store names won’t be included in product-specific searches anymore, increasing the number of relevant results
• We’ve added the ability to do exact matches with quotes around a word or phrase
• You can do asterisk(*) wildcard searches
• Fuzzy matching will help with misspelled words and typos
• Search operators AND, OR, NOT will still work as before. (Note that they must be capitalized. Consult the tooltip next to keyword search for more information.)

We received a number of reports about these issues:

• A bug with how relevance of search results is determined. This bug has been fixed.
• The search results page was displaying an inaccurate count of matching items. Search is returning the same number of matching item as before, but the count may say “10,000+”

We hope this ElasticSearch upgrade will improve everyone’s Marketplace experience. However, we are continuing to fine-tune. If you find that Marketplace search isn’t working the way you’d expect, please let us know! We may very well be able to address your issue.

An example of specific, helpful reporting might be, “I searched for ‘unicorn’, I expected ‘unicorn unicorn unicorn’ in the first page of results, but instead I got ‘horse costumes’.”

From all of us on the Second Life Web team, thank you for your patience and understanding. Happy shopping and selling!