Jump to content

Tools and Technology

  • entries
    126
  • comments
    916
  • views
    286,100

Contributors to this blog

About this blog

Entries in this blog

Linden Lab

We recently made changes to the Marketplace that had some results that our community was not expecting. So we want to clear up confusion about the Marketplace and go over our current and upcoming changes.  

Recently we made a change to the default sort for merchant stores to "newest first" instead of "relevance." This was done to prevent old or outdated products from being the first thing shoppers see when visiting a merchant store, and we have seen increased sales across the Marketplace as a result. 

This is just one of the tweaks we have made to the Marketplace and we have plans for further updates in the near future that we hope will continue to improve the Marketplace experience for both shoppers and creators. 

To clear up some of the confusion around how the Marketplace works and give more insight on how to best use the Marketplace, we have put together some answers to frequently asked questions. 

When using product search 

Your product may not be appearing in the same position as it used to in relevance sort for product search for a number of reasons.

  1. Titles matter. The title of your product needs to contain the search terms that have been entered into search.
  2. How well it sells matters. This looks at the number of sales over a given period of time. 
  3. Keywords included in the product listing. Keywords need to contain the search terms that have been entered into search. 
  4. What features you have matters. Features need to contain the search terms that have been entered into search. 

We have adjusted the weighting of each of these factors with the last search upgrade. This is something we actively monitor and adjust and your product may have ranked higher in relevance under the previous weighting. 

When viewing a merchant store

Relevance sort requires a search term to really function in the way it is intended. When viewing a Marketplace store page or the product search with relevance sort enabled and no search term, you will get unexpected results. 

Prior to the Marketplace search upgrade, when viewing a merchant store with no search term, relevance sort used to take into account the number of sales of your items over periods of time. So high selling items used to sort to the top even if there was no search term applied. Now when viewing a merchant store with no search term the relevance sort will function the same as best selling sort (not the best selling merchant report). Since best selling only indexes the top 50k best selling items across the entire Marketplace, best selling sort will only look up data from that index. If your items do not fall into the top 50k they will all be ranked the same. This is why your best selling sort may not match your best selling reports. In short, your items will only have a sales rank if they fall into the top 50k. Best selling sort has always functioned this way and has not changed. 

With that in mind, we are currently working on a change to allow best selling to now index each merchant’s store individually instead of just the top 50k items. This will allow shoppers to see any creator’s best selling items when viewing a store page. We hope this will be a better long-term solution than the previous relevance sort and clearly communicate to shoppers what are the most popular products in a creator’s store. 

As always, we are also continuing maintenance and have a planned release Monday April 10th to fix a bug impacting merchant product listing enhancements. 

We have seen good sales results with our current changes and are very excited for our upcoming improvements. As is the industry standard for online marketplaces we will continue to improve and make changes based on our own internal data and community feedback. Our top priority is the best experience for our Marketplace creators and shoppers and we hope these changes will bring us closer to that goal. As always, if you find that Marketplace isn’t working the way you’d expect, please let us know!

Happy shopping and selling!

Linden Lab

We’re Improving Marketplace Search


Tools & Technology

Good morning everyone,

At the end of last year we began the task of moving Marketplace search to an up-to-date version of ElasticSearch, the application we use to do full text searches.  We apologize for some of the bumps you may have experienced in the last few weeks as we make this transition.

The Marketplace search backend was showing its age. We’ve upgraded the technology and we’ve dusted off both our indexing and searching filters. We were running ElasticSearch version 2.3 and needed to update to ElasticSearch version 8.4.  Our goal was to bring Marketplace search results closer to web search and make page loads faster, in order to improve your shopping and selling experience.

In practical terms this means:

  • Merchant and store names won’t be included in product-specific searches anymore, increasing the number of relevant results
  • We’ve added the ability to do exact matches with quotes around a word or phrase
  • You can do asterisk(*) wildcard searches
  • Fuzzy matching will help with misspelled words and typos
  • Search operators AND, OR, NOT will still work as before. (Note that they must be capitalized. Consult the tooltip next to keyword search for more information.)

We received a number of reports about these issues:

  • A bug with how relevance of search results is determined. This bug has been fixed.
  • The search results page was displaying an inaccurate count of matching items. Search is returning the same number of matching item as before, but the count may say “10,000+”

We hope this ElasticSearch upgrade will improve everyone’s Marketplace experience. However, we are continuing to fine-tune. If you find that Marketplace search isn’t working the way you’d expect, please let us know! We may very well be able to address your issue.

An example of specific, helpful reporting might be, “I searched for ‘unicorn’, I expected ‘unicorn unicorn unicorn’ in the first page of results, but instead I got ‘horse costumes’.”

From all of us on the Second Life Web team, thank you for your patience and understanding. Happy shopping and selling!

Linden Lab

Viewer Profiles: Everything old is new again!


Tools & Technology

With our latest release we’d like to welcome back Viewer Profiles! The Second Life viewer has made the transition back to an integrated Profile floater. This is the place to find more information about any Resident including a bio, picks, and more!

image1.png

Residents may still use web-based profiles in a web browser. However this change brings back a more responsive experience in the Viewer as well as giving you the information you’re most interested in seeing on the first page. 

As part of this change, Groups now moves to the main tab.  This change also means that hiding Groups no longer requires opening and viewing each one.  Just click on the eye next to the group to hide or show.  

A big thank you to the Firestorm Viewer team for your contribution!

How things have changed

The viewer Profile, as opposed to the web Profile, shows all of your Second Life Bio, Feed, Picks, Classifieds and Real Life bio when viewed by other residents logged in to Second Life.

The Feed will remain a web property displayed in the Profile floater in its own tab.

Interests are accessible on the old web Profiles but will no longer be included in the viewer Profile.

Oz Linden

Uplift Update


Tools & Technology

We've been working hard on the Uplift of Second Life. If you have not been following this project, that's what we're calling the migration of our Second Life simulators, services, and websites from a private datacenter to hosting in The Cloud (Amazon Web Services). It's a massive, complicated project that I've previously compared to converting a steam-driven railroad to a maglev monorail -- without ever stopping the train. This undertaking has at times been smooth sailing, at other times a very bumpy ride. We wanted to share some more of the story with you.

Our goal has been to move SL incrementally to give ourselves the best chance of minimizing awareness among the residents that these changes were happening. We feel we’ve done better than we expected, but of course it’s the bumps in the road that are most noticeable to our residents. We apologize for recent service disruptions, although what’s perhaps not apparent is the progress we’ve made -- and the improvements in performance that have quietly taken place.

First, the rough spots:

  • Region Crossings
    One of the first troubles we found was that region crossings were significantly worse between a cloud region and a datacenter region. We did a deep dive into the code for objects (boats, cars, planes, etc) and produced an improvement that made them significantly faster and more reliable even within the datacenter. This has been applied to all regions already and was a good step forward.
  • Group Chat stalls
    Many users have reported that they are not able to get messages in some of their groups; we're very much aware of the problem. The start of those problems does coincide with when the chat service was uplifted; unfortunately the problems did not become clear until moving that service back to the datacenter was not an option. We haven’t been able to get that fixed as quickly as we would like, but the good news is that we have some changes nearly ready that we think may improve the service and will certainly provide us with better information to diagnose it if it isn't fixed. Those changes are live on the Beta grid now and should move to the main grid very soon.
  • Bake Failures
    Wednesday and especially Thursday of this past week were bad days for avatar appearance, and we're very much aware of how important that is. The avatar bake service has actually been uplifted for some time - it wasn't moving it that caused the problem, but another change to a related service. The good news is that thanks to a great cross-team effort during those two days we were able to determine why an apparently unrelated simulator update triggered the problem and got a fix deployed Thursday night. 
  • Increased Teleport Failures
    We have seen a slight increase in the frequency of teleport failures. I know that if it's happened to you it probably doesn't feel like a "slight" problem, especially since it appears to be true that if it's happened to someone once, it tends to keep happening for a while. Measured over the entire grid, it's just under two percentage points, but even that is unacceptable. We're less sure of the specific causes for this (including whether or not it's Uplift related), but are improving our ability to collect data on it and are very much focused on finding and fixing the problem whatever it is.
  • Marketplace & Stipend Glitches
    We've had some challenges related to uplift for both the Marketplace and the service that pays Premium Stipends. Marketplace had to be returned to the datacenter yesterday, but we'll correct the problems that required the rollback and get it done soon. The Stipends issues were both good and bad for users; there were some delays, but on the other hand we sent some users extra stipends (our fault, you win - we aren't taking them back); those problems are, we believe, solved now.

Perhaps the above makes it sound as though Uplift is in trouble. While this week in particular has seen some bumps in the road, it's actually going well overall. Lots of the infrastructure you don't interact with directly, and some you do, has been uplifted and has worked smoothly.

For a few weeks, almost all of the regions on the Beta grid have been running in the cloud, and over the last couple of weeks we've uplifted around a hundred regions on the main grid. Performance of those regions has been very very good, and stability has been excellent. We expect to be uplifting more regions in the next few working days (if you own a region you'd like included, submit a Support Ticket and we'll make it happen). Uplift of the Release Candidate regions, which will bring the count into the thousands, will begin soon. When we're confident that uplifted regions are working well at that larger scale, we'll be in a position to resume region sales, so if you've been waiting - the wait is almost over.

Overall, the Uplift project is on track to be complete or very nearly so by the end of this year (yes, 2020… I know I've said "fall" before and people have noted that I didn't say what year 🙂 ; the leaves haven't finished falling at my house yet…). It's likely that there will be other (hopefully small) temporary disruptions during this process, but we promise we'll do all we can to avoid them and fix them as fast as we can. This migration sets the stage for some significant improvements to Second Life and positions us to be able to grow the world well into the future.
 

Linden Lab

Updates to Marketplace Search


Tools & Technology

Last week we blogged about moving to an updated version of ElasticSearch, the application we use to do full-text searches on Marketplace. After successfully upgrading from 2.3 to 8.4 we have been closely monitoring usage and feedback from our merchants and Residents. This has led us to make a few tweaks to how our new search system works.   

In the coming days we will be changing the following:

  • A bug where featured listings are not being updated properly.
  • We have removed the 50-page cap on search results. Our statistics showed that few people look at higher-numbered pages. In addition, page load time is increased by loading more result pages into memory. But you have spoken, and we’re removing the cap.
  • A search string such as “blue shirt” will return results that contain both the term “blue” and “shirt” instead of including results that contain “blue” or “shirt.” This behavior is changed if you use advanced features such as quotes, boolean operators (AND, OR, NOT), or asterisks. Make sure to check the tooltip next to the search field for help with boolean operators.

    For example:
    image2.png

    image1.png

    The above search term will now return results in the center of the diagram.

We hope these changes will improve everyone’s Marketplace experience while also enjoying the advantages of ElasticSearch. We continue to monitor how search performs for everyone. If you find that after these changes, Marketplace search still isn’t working the way you’d expect, please let us know! We value everyone’s thoughts and we thank you for your patience as we continue to make search better. 

Happy shopping and selling!

Linden Lab

Upcoming Security Improvements to Second Life


Tools & Technology

Hello Residents!

If you run an inworld service that logs in as a scripted agent (aka, a bot), or maintain a Third Party Viewer, please pay attention. This blog post is for you. :)

We are making some changes to improve the security of Second Life!

On November 1st, 2021 we are going to be discontinuing the use of two older security protocols, TLS 1.0 and TLS 1.1, on our login services. We’re doing this to increase the security of everyone on the grid.

In March of 2021 the Internet Engineering Task Force (IETF) officially deprecated these two older protocols, and now we’re gonna do the same. TLS 1.0 was released in 1999, and TLS 1.1 was released in 2006, and while they’ve had a good run, it’s time for them to enjoy a nice retirement into Internet history.

On Wednesday afternoon this week we inadvertently turned off TLS 1.0 and TLS 1.1, and we received reports that several inworld services (such as older bots and some very old Third Party Viewers) were unable to log in. Since we hadn’t given Residents any warning this was going to happen, we turned them back on this morning. We want to make sure folks have a chance to update their services before we turn them off again permanently on November 1st, 2021.

If you were impacted by the sudden removal of these older security types this week, we’re sorry they were turned off without warning. We should have communicated it better. We also want to thank you for taking the time to improve the security of your service! The grid will be safer for us all as a result.

For everyone else, you have nothing to do! Our viewer, and almost all of the popular Third Party Viewers have been using the latest versions of TLS for years. You’re all set!

Looking forward to a more secure Second Life,
April Linden, Gridbun

Signal Linden

Spring 2023 Scripting Summary


Tools & Technology

As spring winds to an end it may be a good time to stop and smell the Linden Scripting Language (LSL) controlled flowers. In the past year, Second Life's server team has placed an increased priority on delivering useful new functions and enhancements for scripters. This blog post contains highlighted features released in the past six months; maybe there's something you missed!

💾 Linkset Data (LSD)

Likely the most prominent new feature introduced to LSL in recent history, Linkset Data (LSD) provides 128KiB of persistent memory which is shared by all scripts in an object. This functionality scratches several itches which have plagued scripters for some time: a desire for memory that persists between script resets and better conventions for inter-process/script communication. Prior to its introduction, many awkward techniques were used to persist and share data between scripts, including placing data in external HTTP APIs and using the age-old trick of storing data in object descriptions. No more!

At its core, LSD is a key-value database associated with the root object of a linkset. The data stored in LSD is persisted between script resets, object transfer and copying. In its simplest form, a script can write, retrieve and delete data using llLinksetDataWrite, llLinksetDataRead and llLinksetDataDelete while the event linkset_data may be used by any script in the object to be informed of changes.

More information on LSD, including examples, and information about all related library functions can be found on Second Life's wiki, Category: LSL LinksetData and on the great blogpost from resident NeoBokrug Elytis, Linkset Data (LSD) - A New Feature for Second Life.

🔍 List search, now with start and end

Lists can now be searched using llListFindStrided, and manipulated using llList2ListSlice, and llSortListStrided. Notably, llListFindStrided contains start and end parameters, which allow lists to be searched more quickly without many of the out-of-memory situations run into using llListFindList. You can even use llListFindStrided with non-strided data by giving it a stride of one. 🤫

Strided lists are commonly used to store structured data as LSL has no native map type. llListFindStrided and llSortListStrided should make it easier and more performant to work with lists.

✂️ Replace substring

You can now find and replace segments of string in a performant way using llReplaceSubstring.

🔊 Linkset sounds

A handful of functions have been added which make it easier for a linkset to play multiple sounds simultaneously using a single script. This should be a boon for vehicle makers and anyone looking to layer and compose sounds.

🔐 Cryptographic functions

HMAC and RSA functions have been added to LSL. These will make secure communication between systems inside and outside of Second Life easier, and make it easier to do things like create a valid JSON Web Token (JWT).

🙂 Miscellaneous

  • Environment control - Scripts are able to control environment settings using two functions introduced in August 2022: llSetEnvironment and llReplaceEnvironment.
  • Simulator statistics - llGetSimStats has been enhanced with over 25 new statistics, mirroring much of the data displayed in the viewer statistics panel's Simulator section.
  • You can now set an object property that causes objects to die with no warning if they hit a no-entry parcel using llSetStatus(STATUS_DIE_AT_NO_ENTRY). This is useful for bullets and other objects that would otherwise become lodged in the air at parcel boundaries.
  • HTTP custom header size and accounting has been changed. Scripts are no longer limited to 8 256 byte custom header.  LSL Now provides a 4k pool of memory that may be used for custom headers in whatever way the scripter wants.

Wrap up

Phew. In all, more changes have been made to LSL in the last six months than several prior years combined. That's a lot, but we always want to hear more ideas about how best to improve LSL and the tools available to content creators. If you are interested in hearing about scripting and other areas of development, we encourage you to attend one of Second Life's user group meetings or submit your idea to Jira.

Many of the features in this post come from resident submitted requests. Thank you for loving Second Life just like we do.

Happy scripting!

Cosmic Linden

As you may know, Second Life is soon adding the ability to import and edit GLTF Physically-Based Rendering (PBR) Materials. This upgrade to Second Life's graphics engine allows for more realistic surfaces, better reflections, and a more modern workflow for artists. You can learn more about PBR materials on our wiki, or try it out on the beta grid using its Release Candidate Viewer, available here.

We thought we'd give you a sneak peek of one of the features we are working on next: high resolution PBR terrain textures. While this work is early in development, we will discuss ideas and development at the Content Creator meetings, which all residents are welcome to attend. In the near future, a PBR Terrain viewer will be available on the Alternative Viewers page. We look forward to your feedback, and in the meantime, enjoy the pictures!

pbr_terrain_mountain_pass_001.jpg

pbr_terrain_icy_sea_001.jpg

pbr_terrain_muddy_shore_001.jpg

Signal Linden

Shutting Down LSL XML-RPC


Tools & Technology

LSL XML-RPC is a deprecated method for communication between remote systems and LSL scripts. Its use has been cautioned against since the introduction of LSL HTTP-In functionality in 2009. We plan on shutting down LSL XML-RPC completely on June 1st, 2023.

Rationale

As of 2023/03/21, the average amount of traffic served by XML-RPC has dwindled to a few dozen requests per hour. Considering this extraordinarily low amount of traffic and the fact that LSL's more modern HTTP in/out functionality has been available for over 16 years, maintaining LSL XML-RPC infrastructure does not represent a good use of development resources. We'd rather spend time developing new features! 😉

Timeline

  • April-May 2023 - One or more unannounced circuit-breaker exercises will be performed by temporarily blocking incoming LSL XML-RPC traffic. This will give residents and Lindens a chance to experience XML-RPC shutdown and perform any required corrective actions (Upgrading scripts to HTTP-In, etc.) before permanent shut down.
  • May 2023 - XML-RPC will be permanently disabled on RC regions
  • June 2023 - Incoming LSL XML-RPC traffic will be permanently blocked across the grid.

Running LSL scripts that use XML-RPC functionality will continue to function to the extent they can with XML-RPC functions no longer doing anything. If you have functionality that depends on XML-RPC, please migrate your scripts and supporting services to HTTP-In.

If you have questions or concerns feel free to attend the Simulator User Group meeting, held Tuesdays, 12:00 PM Pacific in Denby.

Update: 2023/05/03 - XML-RPC circuit breaker has been left open, as we have not received any indication of XML-RPC impacting production systems. This post's timeline has also been updated to reflect the RC deploy schedule.

Signal Linden

Server Release 2023.07


Tools & Technology

This week we rolled a maintenance update to Second Life's production grid. This release contains a handful of small features and bug fixes as well as some housekeeping and internal improvements to how we address assets.

Scripting

Detect Animesh

You can now detect whether a prim is mesh or has animesh enabled using llGetPrimitiveParams:

Delete, Count multiple linkset values based on pattern

It's now possible to delete and count multiple Linkset Data (LSD) values based on a regex pattern. This should simplify bulk deletion of values:

When multiple keys are deleted using llLinksetDataDeleteFound the linkset_data event will be called with the action LINKSET_DATA_MULTIDELETE.

Estate management

Backend changes have been made to support an expansion of estate manager and ban list entries. This bump will be available to users as part of a future viewer update.

  Old value New value
Estate manager limit 15 20
Estate ban list limit 500 750

Bug Fixes

  • BUG-233830 - Issue with llRequestAgentData() & DATA_PAYINFO never updating.

As usual, see the full list of bugs and features on the release notes.

Happy summer. 🌇

Kyle Linden

Second Life Viewer Maintenance 6.6.8.576863


Tools & Technology

Greetings Residents! Please note that with the release of the Second Life Viewer Maintenance Preferences, Positions and Paste fixes (build 6.6.8.576863) we are making the decision to sort the alpha blending of rigged mesh attachments by the priority assigned to each attachment point. For example, the Chest attachment ID is 1 and is treated with highest priority.

We know this change will require that some alpha blended rigged mesh content will need to be moved to new attachment points. Residents should be able to do this for themselves in many cases, but content creators may need to adjust their product attachment points to account for the new changes.

One of our Residents has written their own blog post explaining this in much greater detail and can be found at: https://beqsother.blogspot.com/2022/11/alpha-blend-issues-get-them-sorted.html

As always, if you believe you have found a bug, please file a Bug report at https://jira.secondlife.com 

Linden Lab

Second Life on GitHub


Tools & Technology

Linden Lab open-sourced Second Life's client on January 8th, 2007. In the 15 years since, open source contributions have led to countless new features and bug fixes that have markedly improved the life of our virtual world. Without open source, SL would not be as vibrant, enduring and successful as it is today.

As part of an effort to improve and modernize our tooling and better support the open source community, I'm happy to announce that Second Life's source code is moving to GitHub, the world's most popular version control hosting platform. You can find our open source repositories in their new home at github.com/secondlife.

As the de facto home of many open source projects, GitHub likely needs no introduction to anyone who works with open source software. Its features and market position make it the best option for Second Life going forward.

Improved Developer Experience

While migrating code to GitHub is an important step towards improving Second Life open source contributors' development experience, it is not a complete picture. We will be making several changes to streamline and modernize our engineering processes: 

Simpler Contributor License Agreements

A Contributor License Agreement (CLA) is a commonly used legal contract between an individual or group and an organization accepting contributions. This process has been modernized, adopting simpler, more developer-friendly language and allowing contributors to sign the contract as part of pull requests. No PDF, no fax. 😉

GitHub Actions CI/CD

Second Life will be moving its continuous integration and deployment (CI/CD) from a self-hosted TeamCity instance to GitHub Actions. This change, to be completed in 2023, will open up new possibilities for our engineering team and the open source community:

  • Our build tooling will be more transparent, allowing OSS contributors to easily fork and build things like autobuild libraries
  • Pull requests will feature automated merge checks such as linting and automated testing
  • Services can be built using newer technology, such as aarch64 servers

image1.png
A simple pipeline to test and publish autobuild.

Summary

While these changes will not impact most users, they will make contributing to Second Life easier and allow the platform to evolve faster. Linden Lab is a uniquely open company, holding regular in-world user groups and allowing its residents to contribute in meaningful ways. We are committed to this philosophy and would love to join you in helping make our shared virtual world even better.

Do you want to contribute to Second Life? Attend one of the user groups on our public calendar, submit an idea for a new feature, or, of course, check out our code on GitHub!

Signal Linden
Engineer, Second Life Server

Linden Lab

Recent Outages


Tools & Technology

Hello Residents!

Well, this is not the way we wanted the past week to go -- multiple outages in the same week! Not fun for anyone. Here’s what happened.

Login and Inventory Issues: Monday, 1/24 - Friday, 1/28
On Monday, January 24, residents’ inventories stopped responding to requests. Some residents were unable to log in as a result. We identified the affected accounts, addressed the problem and restored services to full operation in a little under 3 hours. 

Beginning on Wednesday of this week, we received reports of intermittent inventory issues – failures to rez, unpack and delete inventory items; and intermittent errors upon login.

When the number of reports increased, it became clear that this was an outage. Several Lindens jumped in to diagnose the reports. After some digging, we discovered our back end infrastructure was being overloaded. Once this was resolved, the positive impact was almost immediate. The data now looks good -- on the back end and we are no longer receiving inventory reports.

We’re taking steps, including a deploy late last week, to prevent these issues in the future - and we have already seen progress in making the service more robust.

Weekly Rolling Restarts: Tuesday, 1/25
Every week we restart the simulator servers to keep them running smoothly. We only restart a certain number at the same time, allow them to finish, then start another batch. That’s why we call them “rolling restarts.” But on Tuesday a recent upgrade to the simulators meant that the usual number of simultaneous restarts was too much for the system to handle. The result was load spikes and numerous regions going down. Ultimately more than 12,000 regions were stuck in restart mode, which is 40% of the grid. Not good!

The team came together quickly to bring simulators back up in smaller batches, and then manually fixed blocks of regions. After some trials and monitoring, we found a smaller number of concurrent restarts that worked better. By 1:00 PM all regions were restored and operating normally. We apologize for this lengthy outage!

Advance… Retreat!: Wednesday, 1/26
We had planned a deploy to the production grid that would help us gather information on group chat performance. Unfortunately, the procedure used during testing did not work in production. We figured -- OK, we’ll just roll back. But then the rollback itself had problems!  Residents experienced issues with login, group chat, and presence information. The team was able to isolate the problem and complete the rollback, getting a few more gray hairs in the process. We’re going to take what we’ve learned and do a better deploy that will give us the information to improve group chat in the long run.

Last week was no fun for a lot of people at the Lab. Thank you for your patience. We really don’t like interrupting your enjoyment of Second Life. Here’s hoping we don’t have to come back with another one of these blog posts for quite some time! 

With love, 
The Lindens
 

Linden Lab

Recent February 1st Outage


Tools & Technology

Hello!

This Wednesday, February 1st, was a rocky one for Second Life, especially for regions on our Release Candidate (RC) channels. As with many large-scale services, we experience service interruptions from time to time, but yesterday's problems were unique and noteworthy enough that they deserve a sincere apology and more thorough explanation.

Summary

  • The incident lasted 6 hours
  • 7029 regions were reverted to a previous stable version
  • The root cause was a bug in object inventory behavior that was introduced by new code. This bug caused some objects' scripts to enter a non-running state.
  • We are making changes to our deploy and testing processes to make events like this less likely to happen in the future.

Timeline

Second Life deploys simulator changes in two major phases: first, to Release Candidate (RC) channels which comprise roughly one-third of all production regions, then to the remainder on a main Second Life Server (SLS) channel. Before changes are deployed to RC, they go through several develop-test cycles, including deployments and testing on staging environments.

RC rolls are regularly scheduled for 7AM Pacific (SLT) every Wednesday. While conducting one of these deployments, we received initial reports of issues relating to script behavior at 8:30AM. We immediately started investigating, and by 9AM it was clear that we needed to stop the roll, because the issue was pervasive and clearly related to the new changes. As we do in these situations, we declared an “Incident” and an Incident Commander took charge. 

By 9:30 we started a rollback, reverting the affected simulators to their previous version. We also evaluated what additional actions we would need to take, as it was unclear that a rollback alone would start the scripts which had stopped. This quickly proved to be the case, and we came up with a new plan - one that would ensure that scripts would perform as expected going forward, but possibly undo the changes our residents had been making since the time we had introduced the bug. Although this meant more downtime, it prevented further content loss, and proved to be the best way to put the grid back in order. The team came up with a quick, clean, and efficient way to achieve this and get everyone back on track. 

By 12PM the decision was made to take this direction. By 1:15PM the code was complete, by 1:40PM we confirmed that it worked. By 2:25PM all regions were brought back up. 

Of course after an event like this there’s cleanup to be done, and we had quite a few Lindens reviewing the fallout and taking care of our Residents. Our amazing support team was working with creators whose operations were affected. 

Trouble doesn’t come alone: we had to deal with another outage at the same time - this one on the web side of things.  Some of our services were getting overwhelmed. We added a few hamsters to those wheels, and they seem to be turning just fine again. 

Technical Details

What happened? To quote part of our internal discussion, “scripts are weird”:

Quote

Sometimes we refer to scripts in an object by a globally unique identifier, and sometimes we refer to scripts in an object by an object-local identifier. There was an issue in the version we deployed on 2023-02-01 where object-local script identifiers would always refer to the first script in an object, causing only that script to be marked as running and receive events like touch and linked_message.

Conclusion

What remains after an outage? We write up an internal “postmortem” report: what happened, what lessons we learned, and what future measures we’re going to put in place. We share it widely, so that we can all learn from it. On days like this, we also write a blog post to let you all know what happened as well. 

Of all the reasons to write a blog, a postmortem ranks pretty low for me, because it means something went seriously wrong. But of all the reasons I love working here, watching everyone come together during an outage ranks pretty high. I’m grateful to know that while we may make mistakes, we will not sweep them under the rug, nor look for someone to blame - we will come together and make it right. 

Thank you for your patience and understanding. This is your world, and we hope to take better care of it. 

Grumpity and the Second Life team

Linden Lab

Puppetry User Group Meeting on September 8, 2022


Tools & Technology

The Puppetry User Group meeting will be a bi-weekly meeting discussing an experimental new feature that allows hardware and other software to connect to Second Life and provide animation data for an avatar. The purpose of these meetings is to connect with developers and coders who might want to experiment with it. 

Learn more about Puppetry in our blog post.

Join the team on Thursday at 13:00 PT on the Aditi beta grid, Castelet region
 

Linden Lab

Puppetry User Group Meeting on September 22, 2022


Tools & Technology

The Puppetry User Group meeting will be a bi-weekly meeting discussing an experimental new feature that allows hardware and other software to connect to Second Life and provide animation data for an avatar. The purpose of these meetings is to connect with developers and coders who might want to experiment with it. 

Learn more about Puppetry in our blog post.

Join the team on Thursday at 13:00 PT on the Aditi beta grid, Castelet region
 

Linden Lab

Puppetry User Group Meeting on January 12, 2023


Tools & Technology

The Puppetry User Group meeting is a bi-weekly meeting discussing an experimental new feature that allows hardware and other software to connect to Second Life and provide animation data for an avatar. The purpose of these meetings is to connect with developers and coders who might want to experiment with it. 

Learn more about Puppetry in our blog post.

Join the team every other Thursday at 13:00 PT on the Aditi beta grid, Castelet region

 

Keep up to date with all of our user group meetings on our wiki and public calendar.

Linden Lab

PLEs Making Things Right


Tools & Technology

Good afternoon everyone,

As mentioned in our previous blog we encountered a bug that was preventing PLEs (product listing enhancements) from updating correctly. Since the bug prevented merchant ads from working as intended for an extended period, we have decided to make things right. Today we will be issuing PLE credits for time lost during the outage. So anyone with an active/expiring (which is active but won’t renew) PLE as of Monday, March 13th will get 14 days added to their account. 

You can see the added time on the MKT Subscription page by looking at the new expiration date. For example, if you bought a 30-day sub on say, March 3rd, your end date will show an end date of April 16th. Please let us know if you are having any trouble or didn’t receive the extra time. 

Happy shopping and selling!

Kermit Linden

We have some important planned maintenance for our internal systems coming up on August 10th from approximately 6:00am - 7:30am SLT. In order to upgrade these systems, we will need to take several services down including: new logins, registration, marketplace, payments and payouts. There may be other errors or issues seen across the entire platform.

As soon as the work is complete we will post an update on our Status Blog atSecond Life Status, so be sure to subscribe to updates to be notified as quickly as possible.

We apologize in advance for this disruption and especially appreciate your patience.

While we aren't expecting any in-world problems once the maintenance is complete, if you notice any new issues with everyday operations, please let us know as quickly as possible. You can file a Jira issue to communicate directly with our development team, or contact our support team if that's more convenient. No one knows the world of Second Life as well as Residents, and we value your input and assistance as we continue to update the grid's fundamental technologies. Thank you!

Kyle Linden

PBR Materials Are Here


Tools & Technology

Greetings Residents!

Over the next several weeks PBR Materials will be rolling out across the grid. For those who do not know what PBR Materials are already, this post will explain what they are, how to use them and the benefits they bring.  This announcement is particularly of interest to residents who create inworld objects and avatar attachments.

PBR stands for Physically Based Rendering. PBR is a richer way of specifying the surface of an object that better simulates how objects appear in real life. PBR allows for realistic lighting for metals, plastics, leather, fabric, and more, including realistic reflections and bounce lighting.  The Second Life Viewer will now be able to display these more realistic-appearing objects in world, as well as allow creators to include this more advanced information in the object texturing.

GLTF stands for Graphics Library Transmission Format. This is a standard file format, common in computer graphics, that allows storage of  PBR information. GLTF files can be used by many 3D content authoring tools. Second Life will now support the import and export of GLTF files that contain PBR information.  Note that at this time only PBR Materials will be imported.  Full GLTF support will be available at a later date.

Important note: Until PBR server support is released across the entire Second Life grid, there is a risk of damaging content to which PBR Materials are applied, if rezzed or worn in a region that is not PBR-capable.

At present, you will need to use the latest Second Life release candidate viewer with PBR support 7.0.1.6658224456 and be on PBR capable regions running on BlueSteel and Le Tigre channels with build (2023-10-11.6488746491). To confirm you are using the correct viewer and server, while running Second Life open Help > About Second Life.

What are PBR Materials? Why would I use them in my creations?

The goal behind PBR Materials is Increased visual realism. One method of enhancing realism is to create scenes with real reflections which mimic how our eyes have learned to identify that a surface is metal versus plastic. The Viewer has always been open source and so we sought an open source solution, GLTF, to achieve this realism. GLTF was created by the Khronos Group https://www.khronos.org/, a standards body responsible for open standards such as OpenGL, OpenXR, Vulkan, and others.  

With this new Viewer feature, PBR Materials will automatically generate real reflections when applied to new or existing content. While we cannot go back and add PBR Materials to no-mod content -- only the original content creator can -- every effort has been made to preserve or improve all object appearances in world. The team has been working with content creators for many months, and we thank them all for their dedication to making Second Life the benchmark for beautiful inworld content.

What if you do not create content?

If you do not create inworld content or avatar attachments, you do not need to do anything as a result of these changes. You will begin to see more realistic reflections and lighting on objects inworld and for sale as creators adopt this new feature.

New Features 

The key additions and changes for PBR Materials are:

  • GLTF support for PBR Materials
  • HDR Environments
    • Auto exposure and tonemapping
    • The default environment of Second Life has always been a sunny day with blue skies and this will be more apparent now. 
  • Reflection Probes (automatic and manual)
    • Automatically generated by the PBR Viewer
    • Manually generated by content creators for interior spaces (requires lighting)
  • Materials system folder 

Benefits 

  • GLTF standards are widely accepted and utilized for content creation across many applications
  • PBR Materials objectively and subjectively look better
  • Real reflections automatically generated by the Viewer 
  • Increased compatibility with content creation tools e.g. Substance Painter, Blender

There is a wonderful video from Second Life University on our Youtube channel. We encourage anyone wishing to learn more to take a look: How to Create PBR Materials 

The Second Life Wiki has more technical information about GLTF PBR Materials and how to use them in the content you create.

Known Issues 

While our GLTF PBR Materials project is closer than ever to full release, there are some known issues and we know with broader release, more issues will be uncovered. If you see something, please say something, by filing a BUG report in Jira.

  • PBR material information may be lost from content on non-PBR aware regions
  • Underwater reflections are weird. We hope to fix this in a future release
  • Materials set to alpha mode Blend cannot be changed back to Opaque
  • MacOS performance issues
    • Poor performance on Apple Silicon when “Reflection Coverage” not set to “None”
    • Poor performance on Intel Macbooks when shadows are enabled
  • PBR Material limited permissions issues
  • Transition lines between overlapping manual probes
  • Semi-transparent fullbright objects slightly more opaque
  • Non-PBR enabled Viewers will show high packet loss on PBR aware regions due to the introduction of a new message.

There are inworld Content User Group Meetings we encourage you to attend to discuss your questions and give feedback.  For more information see Content User Group 

IMPORTANT: If content looks good in the Khronos GLTF Sample Viewer or Adobe Substance Painter but looks wrong when you import it into Second Life, please stop and file a bug immediately. Our goal is to adhere to the GLTF specification. Please do not modify content in Second Life to work around situations where the Viewer does not adhere completely to the GLTF spec.

Linden Lab

New Voice Improvements!


Tools & Technology

Happy Wednesday everyone!  

Today we released the latest Maintenance Viewer Build.  Among a slew of fixes and improvements is a key feature we want to highlight: ability to tweak your Voice Activity Level. Lately we’ve had increasing problems with voice cutting out, at events or just talking to friends nearby.  With these changes you will have a way to easily manage the problem. It’s been working seamlessly in our internal testing, including large events. 

Voice Activity Detection
This Viewer exposes 3 VIVOX VAD (Voice Activity Detection) variables via Debug Settings and disables the (previously enabled) automatic mode. By making changes to these variables, we should be able to come up with a collection of settings that we can base new default values on in settings.xml.

The Debug Settings are:

  • VivoxVadAuto
    • Enable (1) or disable (0) automatic VAD - you will almost certainly want this set to 0 [off] and change things using the other settings
  • VivoxVadHangover
    • The time (in milliseconds) that it takes for the VAD to switch back to silence from speech mode after the last speech frame has been detected.
  • VivoxVadNoiseFloor
    • A dimensionless value between 0 and 20000 (default 576) that controls the maximum level at which the noise floor may be set at by the VAD’s noise tracking. Too low of a value will make noise tracking ineffective (A value of 0 disables noise tracking and the VAD then relies purely on the sensitivity property). Too high of a value will make long speech classifiable as noise.
  • VivoxVadSensitivity
    • A dimensionless value between 0 and 100, indicating the ‘sensitivity of the VAD’. Increasing this value corresponds to decreasing the sensitivity of the VAD (i.e. ‘0’ is most sensitive, while 100 is ‘least sensitive’)

The default values (updated) are (using VIVOX names):

  • VivoxVadAuto: 0 (disabled)
  • VadHangover(s): 2000 (Valid values are 1 - 60000 milliseconds)
  • VadSensitivity: 0 (Was 43 - valid values are 0 - 100)
  • VadNoiseFloor: 576 (Valid values are 0 - 20000)

Early testing suggests that VivoxVadNoiseFloor can only be changed by restarting the Viewer or teleporting away and coming back (needs a new voice connection) but the other 2 work in real time as you change them. After some initial testing with VIVOX, we have settled on starting from VivoxVadSensitivity set to 0. This will result in no dropouts because the microphone is sending everything to the voice channel. However, in a noisy environment (talking in background, vacuum cleaner, TV on etc.) it will also transmit that too. With modern microphones with built-in noise cancellation, sending everything may be a good thing as the microphone may have done all the heavy lifting of noise cancellation first.

Please try it out and let us know what you think!  If you’re experiencing any bugs, please let us know!

Linden Lab

Moving From Jira, Pt. 3


Tools & Technology

Friendly Greetings!

As mentioned in our Pt. 2 blog post, we are excited to announce that the process of providing an archive of the Second Life Jira BUG project on GitHub is now complete! 

You can find the public BUG archive at https://github.com/secondlife/jira-archive/issues. The archive currently contains most of the previously public Jira tickets from the original Jira BUG project. However, there are some edge cases that we are still in the process of migrating. It does NOT include tickets that previously had restricted access.

We encourage you to use the Second Life Feedback Portal for any new bug reports or feature requests. Your feedback is crucial to our ongoing efforts to improve Second Life, and the feedback portal provides a convenient platform for you to share your thoughts and insights.

We appreciate your continued support and understanding as we work to enhance the Second Life experience. The migration to feedback.secondlife.com signifies our commitment to fostering a more collaborative and efficient environment for everyone in the Second Life community: providing feature request voting, a public roadmap, tighter integration into the engineering process, and more.

Thank you and see you on the Grid!

Second Life Team

Linden Lab

Mirrors, PBR Terrain, and 2K textures


Tools & Technology

Second Life is getting some new, exciting features that will make the world look even better.

Mirrors are arriving soon! While we love the realistic reflections PBR materials enable, mirrors take reflections to a whole new level.

You can create a mirror using a method similar to reflection probes, but with a twist -- when a reflection probe is set to use the “Mirror” option, all surfaces near the probe will exhibit a real-time mirror reflection, including PBR and legacy materials.

We chose this method to enable maximum flexibility.  For example, you can create a big wall of mirrors and have minimal performance impact regardless of how many mirrors are on that wall. As long as they’re aligned with the mirror reflection probe, they can all show a mirror reflection.

To conserve performance, mirrors are subject to some limitations. The Viewer will show the effect of only one mirror reflection probe at a time -- the one closest to your camera. Mirrors are planar, so they won’t work very well on the body of a car, for example. Like so many good things in life, mirrors are best enjoyed in moderation.

Mirrors-3.jpg

Applying PBR materials to terrain will be possible in the upcoming featurettes release. You will soon be able to apply most GLTF materials to terrain. Initially, PBR terrain will support normal maps and metallic-roughness maps. Emissive textures will not be visible on some machines and transparent materials are not fully supported. We expect PBR materials on terrain to vastly improve the beauty of SL -- because there’s a lot of terrain out there!

PBR terrain uses triplanar mapping, a texture-mapping technique which is new to Second Life. On steep slopes you will notice far less texture-stretching. To conserve performance, triplanar mapping will be enabled only when the graphics quality setting is set to “High” or greater.

PBR Terrain Demo - 3.jpg

With the advances in GLTF PBR Materials and upcoming Terrain materials, we know it is time to increase the texture size limit to 2K (2048x2048 pixels). We are approaching this change  carefully, and are still weighing pricing options for images larger than the current max size. 2K textures will allow artists to add significantly more detail to their creations and do things like efficiently consolidate more UV surface area onto a single asset.

Mirrors and PBR Terrain will be available on the Beta Grid (Aditi) to start with, on the following regions:

  • Rumpus Room 2048
  • Rumpus Room 2049
  • Rumpus Room 2050
  • Rumpus Room 2051

Interested in trying out these new features? You can download the Viewer here. Never accessed the Beta Grid before? Here’s information on how to access the Second Life Beta Grid: https://lindenlab.freshdesk.com/support/solutions/articles/31000156725-accessing-aditi

We can’t wait to see what fun and exciting new things residents create with these featurettes!

Linden Lab

Marketplace offline April 6th, 5am -8am SLT


Tools & Technology

We have some important upgrades planned for our database back end. To ensure the best possible transition to the updated version, we're taking the extra precaution of taking the Second Life Marketplace offline from 5am - 8am SLT on April 6th, 2023. L$ transactions will not be impacted. As soon as the work is complete, we'll post an update on our Status Blog at status.secondlife.com, so be sure to subscribe to updates to be notified as quickly as possible. 

We apologize in advance for this disruption and especially appreciate your patience as we prepare for future projects which depend on this work.

While we aren't expecting any in-world problems once the maintenance is complete, if you notice any new issues with everyday operations that impact Second Life inventory - updating/copying/modifying items, changing outfits, giving an object to another Resident, etc. please let us know as quickly as possible. You can file a Jira issue to communicate directly with our development team, or contact our support team if that's more convenient. No one knows the world of Second Life as well as Residents, and we value your input and assistance as we continue to update the grid's fundamental technologies. Thank you!

Linden Lab

Hello Residents!

We’ve just released an update to our previous Maintenance viewer (here). We introduced a media playing bug last time and this update fixes that.

More information on the previous bug here.

Work around for the previous bug here.

Please feel free to try it out and let us know of specific problems by filing a Jira.

Thank you for your patience as we worked through this problem! 
 

×
×
  • Create New...