Deploy Plan for the Week of 2023-01-30 (Updated Feb 1 @ 11AM)

[Mazidox Linden]

Second Life Server

The SLS channel will be updated to last week's RC release. As the release notes say, this version of the Second Life simulator contains a mixture of stability improvements, bug fixes, and enhancements to LSL's ability to play sounds. 

https://releasenotes.secondlife.com/simulator/2023-01-20.577734.html

Scheduled Tuesday, January 31, between 06:30 - 11:30 PST.

Second Life RC

The RC channels will be rolled during their usual time slot on Wednesday, and will all receive version 2023-01-27.577942. This version adds additional LSL functions for generating RSA and HMAC hashes, and another for replacing a substring of a string, as well as augmenting some existing functions with new parameters.

The release notes can be found here.

Scheduled Wednesday, February 1, between 07:00 - 09:30 PST.

Update:

Due to a regression discovered during the rollback (BUG-233288), the RC roll has been halted, and regions on the new build are being returned to the SLS version of 2023-01-20.577734.  The rollback should be completed sometime before 13:00 PST.

[M Peccable]

My Ban Line HUD does not operate under this new server version 577942. It is hard to describe all the symptoms and the things it is doing wrong.

When editing the HUD and doing a reset of the scripts, nothing happens. It is like the reset request is being completely ignored, because the touch menu still responds after the reset.

Teleporting back to a region running 577734 restores normal operation. Anyone else?

I guess I will go look for a JIRA.

[KT Kingsley]

https://status.secondlifegrid.net/

Investigating - We are receiving reports that scripts are malfunctioning for regions found on RC Server channels. Our engineers are investigating this issue further. We apologize for any inconvenience.

Please follow this blog for updates.
Feb 01, 2023 - 09:07 PST

Update - We will be rolling RC regions back to their previous version. Regions that have already been restarted for today's rolling restarts will be restarted again. We apologize for any inconvenience, and thank you for your patience.

Please follow this blog for updates.
Feb 01, 2023 - 09:19 PST

[Aishagain]

Oops...BOOM!

ETA, 16:30hr PST:  My word, for once BOOM was an understatement.  That such damage to simulators could be done by a code revision shows to me that Linden Lab's QA process is at best flawed and perhaps best described as inadequate!

Edited February 1, 2023 by Aishagain
revised comment

[Maestro Linden]

2 hours ago, M Peccable said:

My Ban Line HUD does not operate under this new server version 577942. It is hard to describe all the symptoms and the things it is doing wrong.

When editing the HUD and doing a reset of the scripts, nothing happens. It is like the reset request is being completely ignored, because the touch menu still responds after the reset.

Teleporting back to a region running 577734 restores normal operation. Anyone else?

I guess I will go look for a JIRA.

For those who are following, we did get a jira for this issue, which triggered the version rollback: https://jira.secondlife.com/browse/BUG-233288

[Love Zhaoying]

Yes, major script issues here. Thanks for the Rollback, @Maestro Linden!

[MBeatrix]

SL is a mess at the moment. Took me a few attempts to login (the message was something about not being able to create a valid session). The dashboard page is gone (bad gateway, it says). And my avatar isn't rezzing.

I'm in an SLS region, by the way.

Well done!

[Mazidox Linden]

The login issues and issues with avatar rezzing/baking are separate from the RC script issue, and it is also being worked on by a separate team of engineers from the RC script issue.

[Henri Beauchamp]

1 hour ago, Mazidox Linden said:

The login issues and issues with avatar rezzing/baking are separate from the RC script issue, and it is also being worked on by a separate team of engineers from the RC script issue.

There are days, like this one, when everything goes wrong...

By the way, I managed to bake my avatar by disabling the AIS3 protocol (it's an option in the Cool VL Viewer)...

AIS3 kept reporting errors such as:

2023-02-01 21:43:52Z WARNING: AISAPI::invokeAISCommandCoro: Inventory error: Gone - Result: <llsd>
    <map>
    <key>error_code</key>
        <integer>9</integer>
    <key>error_description</key>
        <string>Item does not exist or was already deleted.</string>
    <key>error_filename</key>
        <string>item.py</string>
    <key>error_function</key>
        <string>Item.delete</string>
    <key>error_line_number</key>
        <integer>189</integer>
    <key>http_result</key>
        <map>
        <key>error_body</key>
            <map>
            <key>error_code</key>
                <integer>9</integer>
            <key>error_description</key>
                <string>Item does not exist or was already deleted.</string>
            <key>error_filename</key>
                <string>item.py</string>
            <key>error_function</key>
                <string>Item.delete</string>
            <key>error_line_number</key>
                <integer>189</integer>
            <key>item_id</key>
                <uuid>a228321c-7538-3d85-9b9d-9f7b4dfc19f0</uuid>
            </map>
        <key>headers</key>
            <map />
        <key>message</key>
            <string />
        <key>status</key>
            <integer>1</integer>
        <key>success</key>
            <boolean>0</boolean>
        <key>type</key>
            <integer>410</integer>
        <key>url</key>
            <string>https://simhost-0ba370ab45708c33c.agni.secondlife.io:12043/cap/74584457-8294-26ed-7a1a-1a9572ae4453/item/a228321c-7538-3d85-9b9d-9f7b4dfc19f0</string>
        </map>
    <key>item_id</key>
        <uuid>a228321c-7538-3d85-9b9d-9f7b4dfc19f0</uuid>
    </map>
</llsd>

 

[Kadah Coba]

Kinda surprised the new JWT functions were in this roll, had expected they were gonna be in beta for like a week first before hitting an Aditi RC. Unless they were already and I just heard about them late.  

[Kaleb Nightingale]

With the rollback that just happened, is it possible that items that had been newly rezzed, would then disappear? I rezzed some items yesterday, but they seem to have disappeared, when I logged in this morning. They are copy, so no problem there, but I am just trying to work out if it is just my viewer can't see them or the rollback actually caused them to disappear.

Thanks for any comments.

 

[Alwin Alcott]

1 hour ago, Hani Morningstar said:

With the rollback that just happened, is it possible that items that had been newly rezzed, would then disappear? I rezzed some items yesterday, but they seem to have disappeared, when I logged in this morning. They are copy, so no problem there, but I am just trying to work out if it is just my viewer can't see them or the rollback actually caused them to disappear.

Thanks for any comments.

 

Yes that is possible,
this is part of the reason there always is a warning/note, nót to rezz things when restarts or rollbacks are happening,
With these bad issues there isn't a lot to blaim. Nobody plans rollbacks while new code gets installed.
 

Edited February 2, 2023 by Alwin Alcott

[Kadah Coba]

Since the deploy was causing damage to existing objects, I believe the rollback also reverted the sim state and not just reverting the simulator version.

If you manage to loose no-copy items due to having rezed them during the short window before the rollback, may want to contact support and see if they can do anything.

[Kaleb Nightingale]

3 hours ago, Kadah Coba said:

Since the deploy was causing damage to existing objects, I believe the rollback also reverted the sim state and not just reverting the simulator version.

If you manage to loose no-copy items due to having rezed them during the short window before the rollback, may want to contact support and see if they can do anything.

Thank you Kadah, but fortunately, they were copy items, so following the answer from @Alwin Alcott I just rezzed them again.

I knew there were updates going on, but, stupidly, I assumed I was living on a main region and not an RC region. There is no logic to my assumption, apart from thinking that residential regions are all main and not RC. Lesson learnt!

Edited February 3, 2023 by Hani Morningstar

[Signal Linden]

If anyone wants to give the new llHMAC , llVerifyRSA, llSignRSA and expanded llGetSimStats LSL functions a shake on Aditi, the next release candidate build has been provisioned to the following regions:

- Cloud Sandbox 2
- Cloud Sandbox 3
- Cloud Sandbox 4

Release notes are the same as 2023-01-27.577942, but there is a different build ID, replete with fixes for the bug that caused Wednesday's RC rollback.

[Lucia Nightfire]

3 hours ago, Signal Linden said:

If anyone wants to give the new llHMAC , llVerifyRSA, llSignRSA and expanded llGetSimStats LSL functions a shake on Aditi, the next release candidate build has been provisioned to the following regions:

- Cloud Sandbox 2
- Cloud Sandbox 3
- Cloud Sandbox 4

Release notes are the same as 2023-01-27.577942, but there is a different build ID, replete with fixes for the bug that caused Wednesday's RC rollback.

Is there a region on aditi that is NOT running version 2023-02-02.578100?

I want to test a specific vector that causes scripts to be set to not running, that was NOT mentioned in the bug reports.

[EDIT] NVM, found one. Said vector "seems" to be fixed as well.

Edited February 3, 2023 by Lucia Nightfire