Mazidox Linden

Hello everyone! We're doing some slightly unusual restarts on Tuesday of this week. From your perspective they should look just like a quick rolling restart for the main channel, but they're actually quite a bit more exciting than that. We're finally ready to put all Homestead and Openspace (and 1/6th of all main channel Full) regions on the same configuration change that we put out to RCs during the week of 2021-09-14. There should be some modest performance increases associated with this change in most circumstances. There isn't a version number difference associated with this change, so until we get the rest of Second Life Server's Full regions on the same configuration there will be a mixture of old and new region configurations co-existing. We anticipate that we'll finish converting everything to the new configuration in the next month or so. Second Life Server https://releasenotes.secondlife.com/simulator/2021-10-25.565008.html Tuesday 2021-11-16 06:00-11:00 PST Second Life RCs https://releasenotes.secondlife.com/simulator/2021-10-25.565008.html (existing version) https://releasenotes.secondlife.com/simulator/2021-11-11.565737.html (new version, replaces 2021-10-29.565204) Wednesday 2021-11-17 07:00-9:30 PST On Rolling Restarts All regions will be restarted this week.

It looks like we ended up aborting the roll instead of trying to take down the existing hosts and replace them with new hosts, that was my bad on miscommunicating (I misunderstood what other Lindens were telling me). We are not going to roll version 2021-10-08.564610 tomorrow as we found a showstopper bug (hat tip to Lucia Nightfire for finding it during the Server User Group meeting), but we are instead going to use a new technique we've been testing successfully on Aditi this afternoon to restart the Second Life RC versions with their existing version 2021-10-01.564394 for all RCs. If it works we'll restart all Second Life Server hosts the same way next Tuesday.

This is just restarting the machines the simulators run on so they work better, unfortunately we had an issue come up where regions would not start on the new version, so we had to re-start the roll with an earlier version we know works. That should be finishing up just a bit later than expected, and the grid should be back to normal operation soon.

Second Life Server We'll be restarting all Second Life Server simhosts on Tuesday, October 19th, between 06:00 and 11:00 Pacific Daylight Time. As is usual recently we'll be using a new simulator version (in this case version 2021-10-15.564759). I saw some comments regarding the lack of release notes for versions we use to do large scale rolling restarts, and while I can't promise it will be successful I'm going to try something that should let me put up release notes for this restart-only version. Hand-made release notes page for 564759 is here! Second Life RCs We'll be deploying version 2021-10-08.564610 to all Second Life RCs on Wednesday, October 20th, between 07:00 and 10:00 Pacific Daylight Time. We're going to be restarting all Second Life RC hosts between 07:00 and 10:00 Pacific Daylight Time. The version number will not change as part of this restart. On Region Restarts Everything is being restarted this week for that new simulator feeling, yay!

We were initially planning to release a new version of the simulator to RC this week, but ran into some issues with new crashes last Friday, so we're delaying the release for a week to iron them out. Grid Poking Bot has been brought back online and should be restarting regions with more than 10 days uptime. We're also looking at possibly restarting the non-RC simhosts next Tuesday before our Wednesday RC deploy, as they'll be getting close to 3 weeks up uptime.

We've got top Lindens on the case 🙂

Sorry for the delay in notification, but we won't be rolling this week given we rolled the entire grid last Friday. Enjoy your roll free Tuesday and Wednesday!

Second Life Server No roll is planned for Second Life Server hosts after our promotion of version 563375 to all hosts last week. Second Life RC We are planning a rolling restart this week. Due to technical constraints this rolling restart will show as version 564195, however there are no changes in code or configuration from the currently deployed version 563375. Wednesday 2021-09-29 07:00-10:30 PDT On Region Restarts Grid Poking Bot has been brought back online and should restart regions that reach 2 weeks up continuous uptime. We will continue to do mass restarts via version updates for the time being however, as this allows us to more fully refresh the environment the simulators are running in, not just the individual simulators themselves.

If you're still having problems logging in please contact Support, they'll walk you through some initial troubleshooting steps and escalate it up to an engineer to look at if needed.

That's odd, it's working fine for me on both my work machine and personal machine. You might want to try a force refresh in case it's being cached on your end. If that doesn't work, please do let me know. CO₂ levels are good, but we may need to adjust the template we use for SLS rolls. I'll check with the folks on the blog-side, thanks!

Second Life Server We'll be rolling last week's RC roll, version 563375 to all remaining channels and simhosts this week. Second Life RC No new roll for RC this week, but keep your eyes on the Server Technology forum for some news about upcoming releases! Region Restarts After Grid Poking Bot nearly gained sentience earlier today our engineers will be handling region restarts manually until we've had a chance to check the bot's code and make sure it's set to "friendly helper" 🙂

Badlist Bot came back to life and decided to restart everything, just to be safe. We caught it before it got more than about 5% of the grid restarted, and it's currently sitting in a small dark hole while we wait for an engineer to take another crack at not making it go berserk.

I fixed it, sorry. Caffeine seems to never be enough on Mondays. Edit: No, I don't know why we have a dash in the staging URL but not the production URL. I've asked, it's just a thing 🤷‍♂️

https://releasenotes.secondlife.io/simulator/2021-08-27.563375.html is now live. I had it just about done yesterday but there were merge conflicts that required input from colleagues during business hours. (n.b. 563385 is an older "restart only" version that had no changes, so you won't find the release notes for that version anywhere.)

Second Life Server No roll for Second Life Server this week. Second Life RCs We're deploying version 2021-08-27.563375 to all RCs this week after a successful configuration change with version 2021-08-27.563385 last week. On Rolling Restarts We are not planning any rolling restarts for Second Life Server regions this week.

Second Life Server We will be deploying version 2021-08-27.563385 to all Second Life Server main channel regions. This version is identical to existing Second Life Server version 2021-07-29.562252 with no code changes nor configuration changes being deployed to Second Life Server hosts. Scheduled for: 2021-08-31 06:00-11:00 PDT Please note, the Second Life Server deploy window has been permanently modified to be 06:00-11:00 Pacific Time. The Second Life Grid Status Blog will reflect this shortly. Second Life RC We will be deploying version 2021-08-27.563385 to all Second Life RC regions. As noted above there are no code changes in this version, however we will be changing all RC regions over to our new configuration as mentioned here. This will cause some incidental changes to how mesh land impact is calculated, but the primary change will be a small but noticeable bump in speed for most regions. Scheduled for: 2021-09-01 07:00-9:30 PDT As today's outage is ongoing further configuration changes are suspended until we have ensured the stability of Second Life and the in-world experience. On Rolling Restarts As mentioned above, all simulator hosts are being restarted this week. All Main Channel hosts have been restarted.

We're restarting the main channel. It requires a new version to spin up new hosts, but there are no code or configuration changes being deployed to simulators on 2021-08-31. We will have some configuration changes during our standard RC roll window on Wednesday, look for my weekly forum post in a few minutes.

I've passed this on to engineers examining the issue in-depth, but for posterity could you file a BUG at jira.secondlife.com so we can make sure we capture this failure mode as well?

We have paused today's roll as we investigate an issue affecting less than 0.25% of regions preventing them from starting.

Second Life Server There will be no roll for Second Life Server this week. Ideally we won't need to restart most regions before next week's planned deploy; we've seen a higher tolerance for simulators to go a long period without restarting, and will be keeping an eye out this week to see how it goes. Second Life RC This week we'll be deploying version 2021-08-19.563028 to all Second Life RC. In addition, this update will bring all of our RC simhosts onto the same configuration as mentioned in the deploy plan here . This release has fixes for BUG-231064 and ensures llChar() now properly returns the Unicode Replacement Character (�) for all out of range, even if the input value is negative. On Rolling Restarts As mentioned above, we won't be doing rolling restarts for Second Life Server this week.

There are no rolls planned for the week of August 16th. We're looking at possibly rolling out our performance optimizing configuration changes to all RCs next week and putting a new maintenance release on RCs that are already optimized, but we need some more time to gather and parse the relevant data, plus testing the Maintenance release.

Second Life Server This week we're deploying version 2021-07-29.562252 to the rest of the Second Life Server simulator fleet. As a reminder this version contains a partial fix for BUG-230960, with a fix for negative indicies coming in a future release, as well as a first pass of allowing experiences to unsit agents that have accepted the experience's permissions, plus some internal fixes and changes to make life easier for our engineers. Second Life RC This week we'll be deploying version 2021-07-29.562252 with the same updated machine configuration as mentioned here to approximately 15% of homesteads and openspaces. We do not anticipate any major changes in behavior for the simulator, but there is a chance of both better performance, and the already discovered mesh physics impact calculation issue. If you should encounter a situation where an object has a noticeably different mesh physics impact value, please file a BUG on jira.secondlife.com On Region Restarts With all regions on the grid restarting this week we're pausing our investigation of automated region restart issues for the week while we gather data on performance from the changes we're making to RC Homesteads and Openspaces.

We've got some folks on vacation this week and I was up late finishing the last round of testing for this RC, deploy plan is up now and the release notes should have propagated through our CDN at this point.

Second Life Server No roll for Second Life Server this week Second Life RCs As you may have noticed this week we put out version 2021-07-29.562252 which contains some changes to help our engineers better investigate crashes and clean up the crash afterward, as well as a fix for BUG-230960 *, and the implementation of BUG-228541. * llOrd() currently incorrectly returns 63 instead of 65533 for any (gibberish) character created with llChar() from -1 to -(2^32) On Region Restarts We didn't have any rolling restarts needed this week, we're still investigating what is going on with our Grid Poking Bot not always restarting regions.

Second Life Server This week we will be restarting all regions that were not restarted last week to run version 2021-07-16.561609. This week's restarts will not include any changes to the behavior of simulators, nor to the configuration of the machines running those simulators as we are both still gathering data as to how effective last week's modifications were, as well as hunting down a particularly vexing issue. Second Life RC Updates We are still in the final stages of figuring out what (if anything) we can promote to Release Candidate stage for the simulator this week. At this point our contender is version 2021-07-20.561821, which contains some changes to help our engineers better investigate crashes and clean up the crash afterward, as well as a fix for BUG-230960, and the implementation of BUG-228541. No update tomorrow because of a combination of not enough time to finish testing and the bumpiness of today's roll meaning we want to investigate before we roll again. On Region Restarts As mentioned above, we're doing a "super-restart" of all remaining regions that weren't restarted last week so that we clean out any cobwebs that might get missed during a normal region level restart. We are aware that the grid poking bot has continued to derelict its duty, and have noted its permanent record; when we know more about the future of rolling restarts you'll hear it here, and at our Tuesday Server User Group meeting in-world.