Deploy Plan for the week of 2020-10-19

[Bugsly Linden]

Second Life Server:

No Roll (but see below)

Second Life RC:

No Roll (but see below)

Second Life Uplift:

This week marks the beginning of medium-scale migration of production regions to the cloud (AWS). We will be transferring a few hundred regions from all existing channels of Second Life based on the Engineering team's need for additional data and performance metrics. Bulk region migration will take place this week between 6:00 AM and 12:00 PM (Noon) Pacific Daylight Time. If you are interested in having a region you own (or are the alternate payer for) migrated to the cloud, please submit a Support Ticket. Regions that are migrated to the cloud may experience degraded performance or behave incorrectly; if you are in a cloud based region (you can check by clicking Help->About, and if you see the URL on line 3 end in "amazonaws.com" you're in the cloud) and observe behavior you believe is newly incorrect, please file a BUG at jira.secondlife.com. Due to the rapid speed of our Uplift efforts, we are unable to guarantee that regions in the cloud that are behaving incorrectly will be moved immediately back to our existing hosts.

On Region Restarts

Regions will be restarted if they have been running for more than 10 days on Tuesday (Main Channel) and Wednesday (RCs) regardless of whether or not new code is being deployed, for the general health and well being of the Simulators. Nothing beats turning it off and then on again ... once in a while.

[Lucia Nightfire]

Please do NOT put my region on AWS until the KVP servers have been migrated. I don't want my app's menu building and db cleanup operations to take days intead of seconds/minutes to complete, let alone my customer's rapid interaction with content and saved data recall to be lagged. Reads and writes currently take just over 1 server frame to complete, minimum. On AWS region servers they are 3x to 6x that or more.

Edited October 19, 2020 by Lucia Nightfire

[bigmoe Whitfield]

Just a reminder from what I've read,  do not depend on the ip of aws regions or aws naming conventions,  believe it was oz that said that.

[Kyle Linden]

1 hour ago, Lucia Nightfire said:

Please do NOT put my region on AWS until the KVP servers have been migrated. I don't want my app's menu building and db cleanup operations to take days instead of seconds/minutes to complete, let alone my customer's rapid interaction with content and saved data recall to be lagged. Reads and writes currently take just over 1 server frame to complete, minimum. On AWS region servers they are 3x to 6x that or more.

Hi Lucia,

There are a lot of moving parts right now and they're moving very quickly. We certainly do not want to cause any complications to your products by moving your region to AWS at an inopportune time, however, we cannot facilitate individual requests like this.  It must be opt-in or opt-out of moving your region to AWS. If you opt-out to avoid KVP timing issues we can always include your region in the next batch. 

Thanks! 

[Lucia Nightfire]

1 hour ago, Kyle Linden said:

It must be opt-in or opt-out of moving your region to AWS. If you opt-out to avoid KVP timing issues we can always include your region in the next batch.

Please explain this process in detail. When and how do I opt-out and whom do I contact about it?

If it just entails contacting support about moving my region off of an AWS server, have they already been instructed to oblige such requests for such reasons?

I ask because I have been denied in the past with claims that my region was chosen specifically as a test channel and cannot be moved. This was many years ago and I forget what new feature/code was being tested at the time.

Edited October 19, 2020 by Lucia Nightfire

[steeljane42]

Same question as Lucia's above.

I'm glad about uplifting starting on the main grid and all, but I'm not ready to deal with *degraded performance or behave incorrectly* part for who knows how long at my (private) region that I pay for. There should be enough of mainland to experiment with for the time being, too. So... how do I opt-out of (possible) early AWS migration?

[Aishagain]

@Lucia Nightfireand @steeljane42: if I read Bugsly's OP correctly you will need to raise a support ticket specifically opting OUT of this week's uplift.  It is not very clear but that is the only way that I can see of getting your wishes through to the operations team via Support!

[Kyle Linden]

15 minutes ago, Lucia Nightfire said:

Please explain this process in detail. When and how do I opt-out and whom do I contact about it?

If it just entails contacting support about moving my region off of an AWS server, have they already been instructed to oblige such requests for such reasons?

I ask because I have been denied in the past with claims that my region was chosen specifically as a test channel and cannot be moved. This was many years ago and I forget what new feature/code was being tested at the time.

Just to clarify, I made a false assumption that Lucia had previously requested to be one of the first included in the AWS region moves given their active involvement in so many aspects of Second Life. I realize now that was not the case. 

Lucia's region is not currently selected to be moved to AWS this week, as of 2020-10-19 17:20. If at some point their region (or anyone's region) is moved to AWS, it is best to follow the standard processes of filing bug reports for new issues.  Sorry for any confusion, it's been a long Monday.

[Lucia Nightfire]

1 hour ago, Kyle Linden said:

Just to clarify, I made a false assumption that Lucia had previously requested to be one of the first included in the AWS region moves given their active involvement in so many aspects of Second Life. I realize now that was not the case. 

Lucia's region is not currently selected to be moved to AWS this week, as of 2020-10-19 17:20. If at some point their region (or anyone's region) is moved to AWS, it is best to follow the standard processes of filing bug reports for new issues.  Sorry for any confusion, it's been a long Monday.

So there is no protocol to get a region off of an AWS server that support is aware of and will acknowledge? I would like to be in the last mandatory migration wave if possible, unless the KVP migration happens before then, then I would be interested in going to AWS as script memory handling is on average 2.5x better/faster. It's just the dominant service I offer to the public currently relies on heavy KVP usage which, ATM, is hindered on AWS regions.

[NeoBokrug Elytis]

@Kyle LindenI too also heavily use the KVP features of Experiences.  Please make sure my entire estate "The Wastelands" is opted-out.  If I need to file a support ticket please let me know.

[MBeatrix]

9 hours ago, steeljane42 said:

Same question as Lucia's above.

I'm glad about uplifting starting on the main grid and all, but I'm not ready to deal with *degraded performance or behave incorrectly* part for who knows how long at my (private) region that I pay for. There should be enough of mainland to experiment with for the time being, too. So... how do I opt-out of (possible) early AWS migration?

We pay in mainland too, you know?

[steeljane42]

2 hours ago, MBeatrix said:

We pay in mainland too, you know?

I'm aware. But there's plenty of regions that pretty much no one pays for. Empty water regions that are popular for sailing (blake sea was it?), some LL lands/parks/portals and so on. They get enough traffic so LL can collect some data while not disturbing anyone's homes. That's what I meant. Then again living on the mainland you pretty much agree with less control over your own land/region compared to private estates (less tools and control over everything. can't even restart it without LL support), and LL "owns" some parts of those (some of them at least) regions too; water, roads and so on, so it makes sense for them to start from those.

[Qie Niangao]

Some of Mainland moved already, a few weeks ago. I'd rather they just get on with this, so there's time to revert for a while if there are load scaling bugs.

[MBeatrix]

47 minutes ago, steeljane42 said:

I'm aware. But there's plenty of regions that pretty much no one pays for. Empty water regions that are popular for sailing (blake sea was it?), some LL lands/parks/portals and so on. They get enough traffic so LL can collect some data while not disturbing anyone's homes. That's what I meant. [...]

I can agree with that. Still, it's not just about traffic, it's also about rezzing stuff, creating scripts and all other things you do in your own land.
No one likes the idea of possible degraded performance but this has to be done, and I'm with Qie — the sooner, the better.

[arabellajones]

I'm guessing, but "a few hundred regions" and www.gridsurvey.com reports 24979 total regions, so around 2% of regions transferred to the Cloud today.

None of the places I usually visit are on the list.

The Grid Drives, a weekly event, usually cover around 150 regions,and so might see 3 regions on the Cloud.

I am more likely to see problems from the DST change in Europe, which takes lace on Sunday..

 

 

[VenKellie]

5 hours ago, Qie Niangao said:

Some of Mainland moved already, a few weeks ago. I'd rather they just get on with this, so there's time to revert for a while if there are load scaling bugs.

that was a sim crossing update, they switched how data is sent from the exiting region to the entering region. Mainland was still on the old servers till today

[Grumpity Linden]

Hi All, 

We understand the hesitation to move to the cloud - the warnings of possible performance issues are no joke! The unquestionable benefit of doing so early is that it allows you to raise specific issues that your regions are experiencing to our team and help us fix them before all of the grid moves to the cloud.  Our goal is to move as much of a representative cohort as possible so we can see realistic performance. Just today, because we migrated a large(ish) number of regions, we've identified and are in the process of fixing a bottleneck.

However, if your regions have been moved, we will accept requests to move them back to the datacenter for a while longer, and we will do our best to take note of the regions you would like to NOT be moved to AWS at this time. 

We're excited about the progress in uplift and grateful for your help as we move.  And for your patience too. 

 

[Whirly Fizzle]

Some good information from Inara about the uplift here: https://modemworld.me/2020/10/20/2020-sug-meeting-week-43-further-uplift-update/

[Whirly Fizzle]

For those like me who are too lazy to check Help -> About & are curious to know if they are on an uplifted region as they are TPing about, Fourmilab have a free halo attachment that will alert you when you are on an uplifted cloud region:  https://marketplace.secondlife.com/p/Fourmilab-Cloud-Halo/20555301

(Thanks to Inara again  )

[Aishagain]

tsk! that darn Grumpity, he's after another pizza!

[Qie Niangao]

4 hours ago, VenKellie said:

that was a sim crossing update, they switched how data is sent from the exiting region to the entering region. Mainland was still on the old servers till today

Not all of Mainland; at least some of Linden Village was on AWS a couple weeks ago, and last Friday, Oz's blog post included:

Quote

... over the last couple of weeks we've uplifted around a hundred regions on the main grid. 

I think those hundred regions included some Mainland beyond Linden Village, and in her blog post today, Inara Pey cited that post from Friday, noting:

Quote

At the time of that update, it was indicated that about 100 regions on Agni, the main grid, had been transitioned to running on AWS, comprising a mix of Linden-held, Mainland and private regions.

 

[VenKellie]

15 hours ago, Qie Niangao said:

Not all of Mainland; at least some of Linden Village was on AWS a couple weeks ago, and last Friday, Oz's blog post included:

 

i dont see it where in that blog that says the linden village was already moved to aws.

[Qie Niangao]

14 minutes ago, VenKellie said:

i dont see it where in that blog that says the linden village was already moved to aws.

It doesn't. I do, at least for Denby where I attended a meeting where it came up.

[Oz Linden]

If you have reason to believe that uplift has adversely affected something on your region, you should file a Support request to ask that it be moved back to the datacenter (but keep reading to find out how to do so in a way that will work).

You can tell whether or not a region has been uplifted by going there and looking at the simulator hostname in the Help > About dialog: if it ends in ".lindenlab.com" it's in the datacenter, if it ends in ".amazon.com" or ".secondlife.io" it's uplifted.

Be aware of a couple of things before you make a request to be moved back:

1. I have asked Support to collect specific information about exactly what has failed and exactly how it has failed for any such request. If we don't get this information, we can't fix whatever it is and we're likely to leave the region uplifted until we can get the data we need.
2. We are trying to move all of the regions as soon as we can (consistent with carefully listening, and watching all our internal metrics, for problems after each move). This process is going to move as quickly as we can make it happen, so your region(s) will be uplifted pretty soon. We encourage you to actively participate in this process rather than hide from it and hope someone else solves your problem before it affects you. If your region is among the last to move, the time available to address your problem may be very very short; it will be better to have detected it early. Lucia has been testing on the beta grid and other early-uplift regions, and we're very much aware of and working on solutions for the particular problem they have reported; follow Lucias example.

If you report a problem and we can observe it and we think it likely that moving back will fix it (there is some chance that the problem is not a result of the simulator uplift), then we'll move it back while we work on a fix. We have the vast majority of our development resources focused on this migration, but if we don't know about a problem we won't fix it. If we don't fix it, there will come a time in the near future when moving it back will no longer be an option.

Overall, this is going very well; as April said in her post last night - please be patient with us. Really, this is going to be better very soon. 

PS. That note about possible performance problems is true... it's possible ... but mostly what we've seen in the regions uplifted so far is that it's better.

[Ardy Lay]

How about those of us using Linden Lab's own viewer where that portion of the URL display was recently removed?  Perhaps I need a different version?

 

 

Edited October 22, 2020 by Ardy Lay