Mazidox Linden

Things are in flux right now, we're building a new version of the release currently on all RC hosts. Our current plan is to put an updated RC with the correct certificates to fix BUG-228850 tomorrow, June 2nd at 3 AM Pacific Time. Then on Wednesday, June 3rd we will put that build on all hosts on the main grid (assuming we don't have any problems with it overnight, we'll be watching for reports of such!). Formal release notes forthcoming, but I wanted to give everyone who frequents this forum for our deploy plans as much notice as possible that it will be a slightly strange week. We'll also be communicating this through other channels.

The answer to that question is "both" and "it depends". Sometimes it's a problem with using an older base version of code to start a project and trampling over fixes that were made in the mean time, other times it's more like the domino effect you describe. I've certainly seen both types of regressions, though I'd be hard pressed to say which is more common. (None of this is EEP specific, I'm just speaking from experience testing various kinds of server changes.)

Per Grumpity's response on https://jira.secondlife.com/browse/BUG-228848 we're aware of the issue and investigating. Please keep up to date with this issue at https://status.secondlifegrid.net/incidents/r64gmw3383q3

Capabilities are the way the Second Life viewer maps a URL that's just for you and your current session to the internal resources on our servers needed to make your connection to the simulator fully functional. It provides things like the ability to download offline IMs reliably, fetch chunks of your inventory, or send an Abuse Report when someone has broken the Terms of Service or Community Standards. The most important capability though is the "Seed Capability" which is the capability to get capabilities. Without that, almost nothing works, as mentioned above. We know some of the ways a server can look when it's not properly handing out capabilities, but I won't say we know what every case of capability failure looks like. We don't have a magic solution to make Capability failures go away (and if we did we'd be releasing it as soon as we could!), but we know it has a big impact on Residents' ability to enjoy their time in Second Life and it's never very far from our minds when we're thinking about ways to improve Resident experience.

That will be 539684 and it should be everywhere on the grid by the end of Wednesday's roll.

Second Life Server: https://releasenotes.secondlife.com/simulator/2020-04-03T19%3A49%3A49.539684.html Scheduled Tuesday 2020-04-14 03:00-09:00 PDT Second Life RCs: https://releasenotes.secondlife.com/simulator/2020-04-10T18%3A15%3A50.540032.html https://releasenotes.secondlife.com/simulator/2020-04-10T18%3A39%3A41.540037.html Probably not an updated release of https://releasenotes.secondlife.com/simulator/2020-03-30T23%3A37%3A53.539435.html as we need to fix BUG-228456 Scheduled Wednesday 2020-04-15 07:00-10:30 PDT On Region Restarts: Regions will be restarted if they have been running for more than 10 days on Tuesday (Main Channel) and Wednesday (RCs) regardless of whether or not new code is being deployed, for the general health and well being of the Simulators. Nothing beats turning it off and then on again ... once in a while. As always please plan events around these maintenance windows. However, if you are hosting an event on Tuesday or Wednesday when we're not deploying new code, you can restart your region ahead of time to avoid interruptions.

Yes you can! It will be up shortly. I was hoping for a build to finish before posting, but it looks like that's not going to happen tonight. Edit: Some of it will. Computers are not being my friends tonight, so I'm going to put the RC notes up in the morning. Sorry about that!

Just to chime in here that we're aware of this issue (as mentioned up-thread it's not particularly new unfortunately, dating back to late last year), and using the latest information you all have provided we're able to confirm that the issue under discussion (object_rez events never fire) reproduces in the regions mentioned, but work fine in other regions running the same code. We're not sure why that is just yet, but we're going to do our darnedest to find out and get a fix for this behavior in your hands as quick as we can.

We've managed to find bugs that are either showstoppers or require a rebuild in all 3 of the RCs we had prepped for this week (shoutouts to my awesome colleagues for working on so many simulators this week!), so please expect a roll-less Wednesday, with restarts sprinkled in for regions that have been running over 10 days. As far as release notes, last I heard a meeroo ate the signing key for the release note generator and they're hoping to make another one soon. Nonetheless, the headlining feature for 536748 is that we fixed several memory leaks, particularly in region crossing and LSL script initialization. If you notice any new and unfortunate behavior in those areas, please file a bug at jira.secondlife.com

Hello everyone, We have a large load of release candidate simulators this week, and we're still testing some of them today. I can share that we are planning on promoting 2020-02-14T20:17:32.536748 to Second Life Server tomorrow, and we'll have further updates on which simulator or simulators make the RC cut either this evening or tomorrow (Pacific Time).

Hey Phoebe, that's my fault. We smudged things copying from the deploy schedule we use to the actual set of instructions for the roll and I didn't catch it, so we started rolling some of the regions that we meant to only restart today. We realized it fairly early on in the process and rolled them back to the version they were on before (535128). (Edited to add) At the end of today's rolls if you were on version 535702 last week you should be on version 536040 this week, and if you were on 535128 you should still be on that version. If that's not the case, please post your region name and we can take a look at it.

Hello everyone! Indeed, today's rolls have been somewhat unusual, and we've spent almost all of our time trying to get the grid back into a good and healthy state. We're just about there, and we anticipate fewer bumps in the road tomorrow. There will definitely be restarts for regions that need it all regions on RCs, and some RC regions may shuffle around which code they're running, but at the end of tomorrow none should be running anything that's new this week.

As Profaitchikenz alludes to, we made a change to our restart procedure aimed at minimizing disruption (trying to limit each region to one restart) when restarting simhosts (not merely simulators, which as Grumpity Linden posted about is handled by Grid Poking Bot) but unfortunately this exposed a bug with restarting these machines that affected saving scripts in-world. In order to avoid having script saving broken, we started the roll from scratch without this change to our procedure, which is as of this post is ongoing and headed toward the finish line.

Actually this is sort of interesting, we are in the process of making some changes to how we handle inventory backend hosts that could be related to this. It should be mostly transparent to Residents, but if you do see issues like this, we definitely want to hear about them. If you could file a BUG jira at jira.secondlife.com and include the region name, environment information from Help->About Second Life, the time you experienced the issue, and the details of what you tried and what happened I would greatly appreciate it!

Second Life Server: 2019-10-26T00:06:48.532192 Release notes Scheduled Tuesday 2019-11-05 03:00-10:00 PST Second Life RCs: 2019-11-01T18:02:37.532376 (This is a rebuilt version of 2019-10-24T19:07:13.532143 which includes features that are going to Second Life Server this week) Release notes Scheduled Wednesday 2019-11-06 07:00-10:30 PST 2019-11-01T18:05:51.532377 Release notes also TBD, this is a new release that includes just two changes, one of which is internal, and the other of which contains a script to monitor additional region data during rolls Somehow we broke scripts uploading in 532377 (‽), so that's definitely not going to Agni this week! If you see messages about "Unable to upload 12345678-90ab-cdef-1234567890ab due to the following reason" on a version that isn't 532377 we'd very much like to know about it; please file a BUG report at jira.secondlife.com and include all the information requested. On Region Restarts: Regions will be restarted if they have been running for more than 10 days on Tuesday (Main Channel) and Wednesday (RCs) regardless of whether or not new code is being deployed, for the general health and well being of the Simulators. Nothing beats turning it off and then on again ... once in a while. As always please plan events around these maintenance windows. However, if you are hosting an event on Tuesday or Wednesday when we're not deploying new code, you can restart your region ahead of time to avoid interruptions.

Should be fixed now LoganPryor, try again and let me know if you still have problems?

Hi everyone! Sorry for the late deploy plan, I've been testing some changes we're making to internal bits and bobs as a result of the outage this weekend. Release Notes will be forthcoming as soon as I've generated them! (And as a to answer a question I've seen asked about those Release Notes, the date in grey next to a simulator's build ID is the date we generated the notes. Sometimes we generate them on a Friday, sometimes on a Tuesday or a Wednesday, and sometimes we have to re-generated them between rolls of the same version to add or change information) Second Life Server (No Roll) Second Life RCs: 2019-10-24T19:07:13.532143 Release Notes: https://releasenotes.secondlife.com/simulator/2019-10-24T19%3A07%3A13.532143.html 2019-10-26T00:06:48.532192 Release Notes: https://releasenotes.secondlife.com/simulator/2019-10-26T00%3A06%3A48.532192.html Scheduled Wednesday 2019-10-30 07:00-10:30 PDT (Reminder that the US switches from Daylight Time to Standard Time this coming Sunday 2019-11-02. The next 4ish months of deploys will occur on the Pacific Standard Time (UTC-8) deploy schedule.) On Region Restarts: Regions will be restarted if they have been running for more than 10 days on Tuesday (Main Channel) and Wednesday (RCs) regardless of whether or not new code is being deployed, for the general health and well being of the Simulators. Nothing beats turning it off and then on again ... once in a while. As always please plan events around these maintenance windows. However, if you are hosting an event on Tuesday or Wednesday when we're not deploying new code, you can restart your region ahead of time to avoid interruptions.

Hi everyone, our Operations engineers are on the case and currently working to troubleshoot the current root cause (which could include symptoms like rez or derez failures, login difficulties, disconnects when teleporting, and problems sending or receiving inventory items).

We're still ironing out the process of the new release notes generator, and it looks like you found a bug in the process. Thanks! I've forwarded that to the appropriate Lindens to examine.

Second Life Server (No Roll) Second Life RCs (No Roll) Due to a late-breaking bug found during the QA process we are unable to put out a fixed version of last week's roll to RC.

Yep, we are rolling the grid right now to revert the most recent update to LSL script improvements. This should prevent further issues with objects communicating. (But if you continue seeing issues with scripts, whether it's communication or something else we want to know about them!)

Hi there Animats, I can't speak to your concerns regarding overloading, but Kama Center has been up for over an hour. Have you double checked what you're seeing at all?

That isn't a bug, it is the precise version identifier for that simulator build. If a different identifier would be better there's always a feature request describing the user case for when it needs to roll off the tongue (but I hope you'll give this a try first and see if as we get a couple of weeks into this new way of doing things it starts to make more sense).

We've updated the first post to point to the new releasenotes.secondlife.com website for simulator releases (https://releasenotes.secondlife.com/simulator/2019-08-23T17%3A54%3A14.530353.html).

Second Life Server (No Roll): http://wiki.secondlife.com/wiki/Release_Notes/Second_Life_Server/19#19.06.14.528215 Second Life RC BlueSteel: https://wiki.secondlife.com/wiki/Release_Notes/Second_Life_RC_BlueSteel/19#19.08.07.529856 Scheduled Wednesday 2019-08-12 07:00-10:30 PDT Second Life RC LeTigre: https://wiki.secondlife.com/wiki/Release_Notes/Second_Life_RC_LeTigre/19#19.08.07.529856 Scheduled Wednesday 2019-08-12 07:00-10:30 PDT Second Life RC Magnum: https://wiki.secondlife.com/wiki/Release_Notes/Second_Life_RC_Magnum/19#19.08.06.529800 Scheduled Wednesday 2019-08-12 07:00-10:30 PDT On Region Restarts: Regions will be restarted if they have been running for more than 10 days on Tuesday (Main Channel) and Wednesday (RCs) regardless of whether or not new code is being deployed, for the general health and well being of the Simulators. Nothing beats turning it off and then on again ... once in a while. As always please plan events around these maintenance windows. However, if you are hosting an event on Tuesday or Wednesday when we're not deploying new code, you can restart your region ahead of time to avoid interru Edit: That last word was supposed to say interruptions, but I got interrupted typing it