Jump to content

Deploy Plan for the week of 2020-10-19


You are about to reply to a thread that has been inactive for 183 days.

Please take a moment to consider if this thread is worth bumping.

Recommended Posts

Second Life Server:

No Roll (but see below)

Second Life RC:

No Roll (but see below)

Second Life Uplift:

This week marks the beginning of medium-scale migration of production regions to the cloud (AWS). We will be transferring a few hundred regions from all existing channels of Second Life based on the Engineering team's need for additional data and performance metrics. Bulk region migration will take place this week between 6:00 AM and 12:00 PM (Noon) Pacific Daylight Time. If you are interested in having a region you own (or are the alternate payer for) migrated to the cloud, please submit a Support Ticket. Regions that are migrated to the cloud may experience degraded performance or behave incorrectly; if you are in a cloud based region (you can check by clicking Help->About, and if you see the URL on line 3 end in "amazonaws.com" you're in the cloud) and observe behavior you believe is newly incorrect, please file a BUG at jira.secondlife.com. Due to the rapid speed of our Uplift efforts, we are unable to guarantee that regions in the cloud that are behaving incorrectly will be moved immediately back to our existing hosts.

On Region Restarts

Regions will be restarted if they have been running for more than 10 days on Tuesday (Main Channel) and Wednesday (RCs) regardless of whether or not new code is being deployed, for the general health and well being of the Simulators. Nothing beats turning it off and then on again ... once in a while.

  • Like 2
  • Thanks 10
  • Haha 1
Link to post
Share on other sites

Please do NOT put my region on AWS until the KVP servers have been migrated. I don't want my app's menu building and db cleanup operations to take days intead of seconds/minutes to complete, let alone my customer's rapid interaction with content and saved data recall to be lagged. Reads and writes currently take just over 1 server frame to complete, minimum. On AWS region servers they are 3x to 6x that or more.

Edited by Lucia Nightfire
Link to post
Share on other sites
1 hour ago, Lucia Nightfire said:

Please do NOT put my region on AWS until the KVP servers have been migrated. I don't want my app's menu building and db cleanup operations to take days instead of seconds/minutes to complete, let alone my customer's rapid interaction with content and saved data recall to be lagged. Reads and writes currently take just over 1 server frame to complete, minimum. On AWS region servers they are 3x to 6x that or more.

Hi Lucia,

There are a lot of moving parts right now and they're moving very quickly. We certainly do not want to cause any complications to your products by moving your region to AWS at an inopportune time, however, we cannot facilitate individual requests like this.  It must be opt-in or opt-out of moving your region to AWS. If you opt-out to avoid KVP timing issues we can always include your region in the next batch. 

Thanks! 

  • Thanks 2
Link to post
Share on other sites
1 hour ago, Kyle Linden said:

It must be opt-in or opt-out of moving your region to AWS. If you opt-out to avoid KVP timing issues we can always include your region in the next batch.

Please explain this process in detail. When and how do I opt-out and whom do I contact about it?

If it just entails contacting support about moving my region off of an AWS server, have they already been instructed to oblige such requests for such reasons?

I ask because I have been denied in the past with claims that my region was chosen specifically as a test channel and cannot be moved. This was many years ago and I forget what new feature/code was being tested at the time.

Edited by Lucia Nightfire
  • Like 1
Link to post
Share on other sites

Same question as Lucia's above.

I'm glad about uplifting starting on the main grid and all, but I'm not ready to deal with *degraded performance or behave incorrectly* part for who knows how long at my (private) region that I pay for. There should be enough of mainland to experiment with for the time being, too. So... how do I opt-out of (possible) early AWS migration?

Link to post
Share on other sites
15 minutes ago, Lucia Nightfire said:

Please explain this process in detail. When and how do I opt-out and whom do I contact about it?

If it just entails contacting support about moving my region off of an AWS server, have they already been instructed to oblige such requests for such reasons?

I ask because I have been denied in the past with claims that my region was chosen specifically as a test channel and cannot be moved. This was many years ago and I forget what new feature/code was being tested at the time.

Just to clarify, I made a false assumption that Lucia had previously requested to be one of the first included in the AWS region moves given their active involvement in so many aspects of Second Life. I realize now that was not the case. 

Lucia's region is not currently selected to be moved to AWS this week, as of 2020-10-19 17:20. If at some point their region (or anyone's region) is moved to AWS, it is best to follow the standard processes of filing bug reports for new issues.  Sorry for any confusion, it's been a long Monday.

  • Thanks 3
Link to post
Share on other sites
1 hour ago, Kyle Linden said:

Just to clarify, I made a false assumption that Lucia had previously requested to be one of the first included in the AWS region moves given their active involvement in so many aspects of Second Life. I realize now that was not the case. 

Lucia's region is not currently selected to be moved to AWS this week, as of 2020-10-19 17:20. If at some point their region (or anyone's region) is moved to AWS, it is best to follow the standard processes of filing bug reports for new issues.  Sorry for any confusion, it's been a long Monday.

So there is no protocol to get a region off of an AWS server that support is aware of and will acknowledge? I would like to be in the last mandatory migration wave if possible, unless the KVP migration happens before then, then I would be interested in going to AWS as script memory handling is on average 2.5x better/faster. It's just the dominant service I offer to the public currently relies on heavy KVP usage which, ATM, is hindered on AWS regions.

Link to post
Share on other sites
9 hours ago, steeljane42 said:

Same question as Lucia's above.

I'm glad about uplifting starting on the main grid and all, but I'm not ready to deal with *degraded performance or behave incorrectly* part for who knows how long at my (private) region that I pay for. There should be enough of mainland to experiment with for the time being, too. So... how do I opt-out of (possible) early AWS migration?

We pay in mainland too, you know?

  • Like 2
Link to post
Share on other sites
2 hours ago, MBeatrix said:

We pay in mainland too, you know?

I'm aware. But there's plenty of regions that pretty much no one pays for. Empty water regions that are popular for sailing (blake sea was it?), some LL lands/parks/portals and so on. They get enough traffic so LL can collect some data while not disturbing anyone's homes. That's what I meant. Then again living on the mainland you pretty much agree with less control over your own land/region compared to private estates (less tools and control over everything. can't even restart it without LL support), and LL "owns" some parts of those (some of them at least) regions too; water, roads and so on, so it makes sense for them to start from those.

Link to post
Share on other sites
47 minutes ago, steeljane42 said:

I'm aware. But there's plenty of regions that pretty much no one pays for. Empty water regions that are popular for sailing (blake sea was it?), some LL lands/parks/portals and so on. They get enough traffic so LL can collect some data while not disturbing anyone's homes. That's what I meant. [...]

I can agree with that. Still, it's not just about traffic, it's also about rezzing stuff, creating scripts and all other things you do in your own land.
No one likes the idea of possible degraded performance but this has to be done, and I'm with Qie — the sooner, the better.

  • Like 1
Link to post
Share on other sites

I'm guessing, but "a few hundred regions" and www.gridsurvey.com reports 24979 total regions, so around 2% of regions transferred to the Cloud today.

None of the places I usually visit are on the list.

The Grid Drives, a weekly event, usually cover around 150 regions,and so might see 3 regions on the Cloud.

I am more likely to see problems from the DST change in Europe, which takes lace on Sunday..

 

 

Link to post
Share on other sites
5 hours ago, Qie Niangao said:

Some of Mainland moved already, a few weeks ago. I'd rather they just get on with this, so there's time to revert for a while if there are load scaling bugs.

that was a sim crossing update, they switched how data is sent from the exiting region to the entering region. Mainland was still on the old servers till today

Link to post
Share on other sites

Hi All, 

We understand the hesitation to move to the cloud - the warnings of possible performance issues are no joke! The unquestionable benefit of doing so early is that it allows you to raise specific issues that your regions are experiencing to our team and help us fix them before all of the grid moves to the cloud.  Our goal is to move as much of a representative cohort as possible so we can see realistic performance. Just today, because we migrated a large(ish) number of regions, we've identified and are in the process of fixing a bottleneck.

However, if your regions have been moved, we will accept requests to move them back to the datacenter for a while longer, and we will do our best to take note of the regions you would like to NOT be moved to AWS at this time. 

We're excited about the progress in uplift and grateful for your help as we move.  And for your patience too. 

 

  • Like 4
  • Thanks 3
Link to post
Share on other sites

For those like me who are too lazy to check Help -> About & are curious to know if they are on an uplifted region as they are TPing about, Fourmilab have a free halo attachment that will alert you when you are on an uplifted cloud region:  https://marketplace.secondlife.com/p/Fourmilab-Cloud-Halo/20555301

(Thanks to Inara again  :D )

  • Like 2
  • Thanks 6
  • Haha 1
Link to post
Share on other sites
4 hours ago, VenKellie said:

that was a sim crossing update, they switched how data is sent from the exiting region to the entering region. Mainland was still on the old servers till today

Not all of Mainland; at least some of Linden Village was on AWS a couple weeks ago, and last Friday, Oz's blog post included:

Quote

... over the last couple of weeks we've uplifted around a hundred regions on the main grid. 

I think those hundred regions included some Mainland beyond Linden Village, and in her blog post today, Inara Pey cited that post from Friday, noting:

Quote

At the time of that update, it was indicated that about 100 regions on Agni, the main grid, had been transitioned to running on AWS, comprising a mix of Linden-held, Mainland and private regions.

 

Link to post
Share on other sites

If you have reason to believe that uplift has adversely affected something on your region, you should file a Support request to ask that it be moved back to the datacenter (but keep reading to find out how to do so in a way that will work).

You can tell whether or not a region has been uplifted by going there and looking at the simulator hostname in the Help > About dialog: if it ends in ".lindenlab.com" it's in the datacenter, if it ends in ".amazon.com" or ".secondlife.io" it's uplifted.

Be aware of a couple of things before you make a request to be moved back:

  1. I have asked Support to collect specific information about exactly what has failed and exactly how it has failed for any such request. If we don't get this information, we can't fix whatever it is and we're likely to leave the region uplifted until we can get the data we need.
  2. We are trying to move all of the regions as soon as we can (consistent with carefully listening, and watching all our internal metrics, for problems after each move). This process is going to move as quickly as we can make it happen, so your region(s) will be uplifted pretty soon. We encourage you to actively participate in this process rather than hide from it and hope someone else solves your problem before it affects you. If your region is among the last to move, the time available to address your problem may be very very short; it will be better to have detected it early. Lucia has been testing on the beta grid and other early-uplift regions, and we're very much aware of and working on solutions for the particular problem they have reported; follow Lucias example.

If you report a problem and we can observe it and we think it likely that moving back will fix it (there is some chance that the problem is not a result of the simulator uplift), then we'll move it back while we work on a fix. We have the vast majority of our development resources focused on this migration, but if we don't know about a problem we won't fix it. If we don't fix it, there will come a time in the near future when moving it back will no longer be an option.

Overall, this is going very well; as April said in her post last night - please be patient with us. Really, this is going to be better very soon. 

PS. That note about possible performance problems is true... it's possible ... but mostly what we've seen in the regions uplifted so far is that it's better.

  • Like 2
  • Thanks 6
Link to post
Share on other sites
You are about to reply to a thread that has been inactive for 183 days.

Please take a moment to consider if this thread is worth bumping.

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...