Tools and Technology

Second Life's next Server regular maintenance release is scheduled to roll out to the full grid next week (2023/06/27.) This update contains quality of life improvements for builders and several bug fixes.

Building

Max linkset size now 64 meters

The maximum size for a linkset has been 54 meters since 2010. This has posed problems for builders as it is smaller than the maximum object size of 64m. In order to allow creators to do things like create a house out of a single large mesh object and link furniture on either sides of it we have bumped the max linkset size to match max object size.

Old (54m) and new (64m) max linkset sizes.

Scripting

HTTP PATCH and HEAD support

Scripters who interact with external web services and APIs can now use HTTP PATCH and HEAD methods with llHTTPRequest.

Ability to detect bot status

You can now detect if an agent is an identified scripted agent ("bot") using llGetAgentInfo.

llCreateLink sleep time reduced

Scripts can now create links 10x faster: llCreateLink sleep time has been reduced from 1s to 100ms!

Bug fixes

• Fixed an issue where groups could be accidentally disbanded when removing a single agent
• Fixed an issue where group owners could not leave a group
• Fixed an issue that would prevent objects with LinksetData containing certain keys could not be rezzed or attached

For a complete list of features and bug fixes see the release notes: 2023-06-09.580543

Good afternoon everyone,

As mentioned in our previous blog we encountered a bug that was preventing PLEs (product listing enhancements) from updating correctly. Since the bug prevented merchant ads from working as intended for an extended period, we have decided to make things right. Today we will be issuing PLE credits for time lost during the outage. So anyone with an active/expiring (which is active but won’t renew) PLE as of Monday, March 13th will get 14 days added to their account. 

You can see the added time on the MKT Subscription page by looking at the new expiration date. For example, if you bought a 30-day sub on say, March 3rd, your end date will show an end date of April 16th. Please let us know if you are having any trouble or didn’t receive the extra time. 

Happy shopping and selling!

LSL XML-RPC is a deprecated method for communication between remote systems and LSL scripts. Its use has been cautioned against since the introduction of LSL HTTP-In functionality in 2009. We plan on shutting down LSL XML-RPC completely on June 1st, 2023.

Rationale

As of 2023/03/21, the average amount of traffic served by XML-RPC has dwindled to a few dozen requests per hour. Considering this extraordinarily low amount of traffic and the fact that LSL's more modern HTTP in/out functionality has been available for over 16 years, maintaining LSL XML-RPC infrastructure does not represent a good use of development resources. We'd rather spend time developing new features! 😉

Timeline

• April-May 2023 - One or more unannounced circuit-breaker exercises will be performed by temporarily blocking incoming LSL XML-RPC traffic. This will give residents and Lindens a chance to experience XML-RPC shutdown and perform any required corrective actions (Upgrading scripts to HTTP-In, etc.) before permanent shut down.
• May 2023 - XML-RPC will be permanently disabled on RC regions
• June 2023 - Incoming LSL XML-RPC traffic will be permanently blocked across the grid.

Running LSL scripts that use XML-RPC functionality will continue to function to the extent they can with XML-RPC functions no longer doing anything. If you have functionality that depends on XML-RPC, please migrate your scripts and supporting services to HTTP-In.

If you have questions or concerns feel free to attend the Simulator User Group meeting, held Tuesdays, 12:00 PM Pacific in Denby.

Update: 2023/05/03 - XML-RPC circuit breaker has been left open, as we have not received any indication of XML-RPC impacting production systems. This post's timeline has also been updated to reflect the RC deploy schedule.

As mentioned in the forum, we have been working on improvements to the inventory service. Second Life's agent inventory service (AIS) handles all agent inventory operations that happen in Second Life. We uncovered some additional issues last week, fixed (and tested!) them, and will be deploying AIS this Wednesday, March 22nd. We expect downtime to be minimal.

Post deploy, if you notice any new issues with everyday operations that impact Second Life inventory - update/copy/modify items, change outfits, give an object to another Resident, etc. let us know and file a Jira issue!

Hello Residents!

The Second Life viewer and servers have added a feature for group chat, providing recent message history. 

Have you ever logged into Second Life and your group chat opens with the punchline of a joke? Now you’ll be able to read back and see what it was. The chat servers save messages from the last hour, so you won’t miss what’s happening.

A viewer update is required to use this new feature, and will be part of a future release (headed to Maintenance-R soon - watch release notes here). 

The viewer code is available now in our public repo - explore here: https://github.com/secondlife/viewer/tree/DRTVWR-573-maint-R 

We look forward to your feedback or any issues with this new feature.

We have some important upgrades planned for our database back end. To ensure the best possible transition to the updated version, we're taking the extra precaution of taking the Second Life Marketplace offline from 5am - 8am SLT on April 6th, 2023. L$ transactions will not be impacted. As soon as the work is complete, we'll post an update on our Status Blog at status.secondlife.com, so be sure to subscribe to updates to be notified as quickly as possible. 

We apologize in advance for this disruption and especially appreciate your patience as we prepare for future projects which depend on this work.

While we aren't expecting any in-world problems once the maintenance is complete, if you notice any new issues with everyday operations that impact Second Life inventory - updating/copying/modifying items, changing outfits, giving an object to another Resident, etc. please let us know as quickly as possible. You can file a Jira issue to communicate directly with our development team, or contact our support team if that's more convenient. No one knows the world of Second Life as well as Residents, and we value your input and assistance as we continue to update the grid's fundamental technologies. Thank you!

This week we rolled a maintenance update to Second Life's production grid. This release contains a handful of small features and bug fixes as well as some housekeeping and internal improvements to how we address assets.

Scripting

Detect Animesh

You can now detect whether a prim is mesh or has animesh enabled using llGetPrimitiveParams:

Delete, Count multiple linkset values based on pattern

It's now possible to delete and count multiple Linkset Data (LSD) values based on a regex pattern. This should simplify bulk deletion of values:

When multiple keys are deleted using llLinksetDataDeleteFound the linkset_data event will be called with the action LINKSET_DATA_MULTIDELETE.

Estate management

Backend changes have been made to support an expansion of estate manager and ban list entries. This bump will be available to users as part of a future viewer update.

Bug Fixes

• BUG-233830 - Issue with llRequestAgentData() & DATA_PAYINFO never updating.

As usual, see the full list of bugs and features on the release notes.

Happy summer. 🌇

Hello!

This Wednesday, February 1st, was a rocky one for Second Life, especially for regions on our Release Candidate (RC) channels. As with many large-scale services, we experience service interruptions from time to time, but yesterday's problems were unique and noteworthy enough that they deserve a sincere apology and more thorough explanation.

Summary

• The incident lasted 6 hours
• 7029 regions were reverted to a previous stable version
• The root cause was a bug in object inventory behavior that was introduced by new code. This bug caused some objects' scripts to enter a non-running state.
• We are making changes to our deploy and testing processes to make events like this less likely to happen in the future.

Timeline

Second Life deploys simulator changes in two major phases: first, to Release Candidate (RC) channels which comprise roughly one-third of all production regions, then to the remainder on a main Second Life Server (SLS) channel. Before changes are deployed to RC, they go through several develop-test cycles, including deployments and testing on staging environments.

RC rolls are regularly scheduled for 7AM Pacific (SLT) every Wednesday. While conducting one of these deployments, we received initial reports of issues relating to script behavior at 8:30AM. We immediately started investigating, and by 9AM it was clear that we needed to stop the roll, because the issue was pervasive and clearly related to the new changes. As we do in these situations, we declared an “Incident” and an Incident Commander took charge. 

By 9:30 we started a rollback, reverting the affected simulators to their previous version. We also evaluated what additional actions we would need to take, as it was unclear that a rollback alone would start the scripts which had stopped. This quickly proved to be the case, and we came up with a new plan - one that would ensure that scripts would perform as expected going forward, but possibly undo the changes our residents had been making since the time we had introduced the bug. Although this meant more downtime, it prevented further content loss, and proved to be the best way to put the grid back in order. The team came up with a quick, clean, and efficient way to achieve this and get everyone back on track. 

By 12PM the decision was made to take this direction. By 1:15PM the code was complete, by 1:40PM we confirmed that it worked. By 2:25PM all regions were brought back up. 

Of course after an event like this there’s cleanup to be done, and we had quite a few Lindens reviewing the fallout and taking care of our Residents. Our amazing support team was working with creators whose operations were affected. 

Trouble doesn’t come alone: we had to deal with another outage at the same time - this one on the web side of things.  Some of our services were getting overwhelmed. We added a few hamsters to those wheels, and they seem to be turning just fine again. 

Technical Details

What happened? To quote part of our internal discussion, “scripts are weird”:

Quote

Sometimes we refer to scripts in an object by a globally unique identifier, and sometimes we refer to scripts in an object by an object-local identifier. There was an issue in the version we deployed on 2023-02-01 where object-local script identifiers would always refer to the first script in an object, causing only that script to be marked as running and receive events like touch and linked_message.

Conclusion

What remains after an outage? We write up an internal “postmortem” report: what happened, what lessons we learned, and what future measures we’re going to put in place. We share it widely, so that we can all learn from it. On days like this, we also write a blog post to let you all know what happened as well. 

Of all the reasons to write a blog, a postmortem ranks pretty low for me, because it means something went seriously wrong. But of all the reasons I love working here, watching everyone come together during an outage ranks pretty high. I’m grateful to know that while we may make mistakes, we will not sweep them under the rug, nor look for someone to blame - we will come together and make it right. 

Thank you for your patience and understanding. This is your world, and we hope to take better care of it. 

Grumpity and the Second Life team

The Puppetry User Group meeting will be a bi-weekly meeting discussing an experimental new feature that allows hardware and other software to connect to Second Life and provide animation data for an avatar. The purpose of these meetings is to connect with developers and coders who might want to experiment with it. 

Learn more about Puppetry in our blog post.

Join the team on Thursday at 13:00 PT on the Aditi beta grid, Castelet region. 
 

Hello Residents!

We’ve just released an update to our previous Maintenance viewer (here). We introduced a media playing bug last time and this update fixes that.

More information on the previous bug here.

Work around for the previous bug here.

Please feel free to try it out and let us know of specific problems by filing a Jira.

Thank you for your patience as we worked through this problem! 
 

The Puppetry User Group meeting will be a bi-weekly meeting discussing an experimental new feature that allows hardware and other software to connect to Second Life and provide animation data for an avatar. The purpose of these meetings is to connect with developers and coders who might want to experiment with it. 

Learn more about Puppetry in our blog post.

Join the team on Thursday at 13:00 PT on the Aditi beta grid, Castelet region. 
 

You may have noticed – or fallen victim to -- the unfortunate increase in server crashes last week. We apologize to everyone affected! 

Here’s what happened:

• On Thursday, June 30, just before the July 4 holiday weekend in the U.S, we released a change to my.secondlife.com.
• Early on the following Tuesday, we noticed and started investigating a significant increase in server crashes. We also started receiving Support reports about these.
• We determined that the web update in my.secondlife.com  included a library update which, unknown to us, dropped support for the texture format our servers expect.
• This led to Viewers being unable to deal with newly updated textures, and under some circumstances the Server would attempt to be more helpful in opening those, than it should have been - and crashing. This is why some residents saw problems with their profile photos. 

On Tuesday July 5, after we investigated and confirmed what was happening, the team quickly rolled back the web site change to prevent the creation of more broken textures. The web team identified these textures, then ran a script to convert them. This was completed by Thursday morning, July 7.

The server team fast-tracked a fix to prevent servers from crashing. We were able to QA and release the fix to RC channels on the afternoon of Thursday, July 7 and then to the main SLS + Events channels on Friday, July 8.

Lessons:

• Second Life is big and complicated and fails in unexpected ways (we knew that!)
• Library upgrades may need additional testing for dependencies we couldn’t imagine.
• We work really well together when Second Life fails in unexpected ways                                                    

Again, we apologize to everyone affected!

The Puppetry User Group meeting is a bi-weekly meeting discussing an experimental new feature that allows hardware and other software to connect to Second Life and provide animation data for an avatar. The purpose of these meetings is to connect with developers and coders who might want to experiment with it. 

Learn more about Puppetry in our blog post.

Join the team every other Thursday at 13:00 PT on the Aditi beta grid, Castelet region. 

Keep up to date with all of our user group meetings on our wiki and public calendar.