Jump to content
You are about to reply to a thread that has been inactive for 1489 days.

Please take a moment to consider if this thread is worth bumping.

Recommended Posts

  • Lindens
Posted

Second Life Server:

No Roll (but see below)

Second Life RC:

No Roll (but see below)

Second Life Uplift:

This week marks the beginning of medium-scale migration of production regions to the cloud (AWS). We will be transferring a few hundred regions from all existing channels of Second Life based on the Engineering team's need for additional data and performance metrics. Bulk region migration will take place this week between 6:00 AM and 12:00 PM (Noon) Pacific Daylight Time. If you are interested in having a region you own (or are the alternate payer for) migrated to the cloud, please submit a Support Ticket. Regions that are migrated to the cloud may experience degraded performance or behave incorrectly; if you are in a cloud based region (you can check by clicking Help->About, and if you see the URL on line 3 end in "amazonaws.com" you're in the cloud) and observe behavior you believe is newly incorrect, please file a BUG at jira.secondlife.com. Due to the rapid speed of our Uplift efforts, we are unable to guarantee that regions in the cloud that are behaving incorrectly will be moved immediately back to our existing hosts.

On Region Restarts

Regions will be restarted if they have been running for more than 10 days on Tuesday (Main Channel) and Wednesday (RCs) regardless of whether or not new code is being deployed, for the general health and well being of the Simulators. Nothing beats turning it off and then on again ... once in a while.

  • Like 2
  • Thanks 10
  • Haha 1
Posted (edited)

Please do NOT put my region on AWS until the KVP servers have been migrated. I don't want my app's menu building and db cleanup operations to take days intead of seconds/minutes to complete, let alone my customer's rapid interaction with content and saved data recall to be lagged. Reads and writes currently take just over 1 server frame to complete, minimum. On AWS region servers they are 3x to 6x that or more.

Edited by Lucia Nightfire
  • Lindens
Posted
1 hour ago, Lucia Nightfire said:

Please do NOT put my region on AWS until the KVP servers have been migrated. I don't want my app's menu building and db cleanup operations to take days instead of seconds/minutes to complete, let alone my customer's rapid interaction with content and saved data recall to be lagged. Reads and writes currently take just over 1 server frame to complete, minimum. On AWS region servers they are 3x to 6x that or more.

Hi Lucia,

There are a lot of moving parts right now and they're moving very quickly. We certainly do not want to cause any complications to your products by moving your region to AWS at an inopportune time, however, we cannot facilitate individual requests like this.  It must be opt-in or opt-out of moving your region to AWS. If you opt-out to avoid KVP timing issues we can always include your region in the next batch. 

Thanks! 

  • Thanks 2
Posted (edited)
1 hour ago, Kyle Linden said:

It must be opt-in or opt-out of moving your region to AWS. If you opt-out to avoid KVP timing issues we can always include your region in the next batch.

Please explain this process in detail. When and how do I opt-out and whom do I contact about it?

If it just entails contacting support about moving my region off of an AWS server, have they already been instructed to oblige such requests for such reasons?

I ask because I have been denied in the past with claims that my region was chosen specifically as a test channel and cannot be moved. This was many years ago and I forget what new feature/code was being tested at the time.

Edited by Lucia Nightfire
  • Like 1
Posted

Same question as Lucia's above.

I'm glad about uplifting starting on the main grid and all, but I'm not ready to deal with *degraded performance or behave incorrectly* part for who knows how long at my (private) region that I pay for. There should be enough of mainland to experiment with for the time being, too. So... how do I opt-out of (possible) early AWS migration?

Posted

@Lucia Nightfireand @steeljane42: if I read Bugsly's OP correctly you will need to raise a support ticket specifically opting OUT of this week's uplift.  It is not very clear but that is the only way that I can see of getting your wishes through to the operations team via Support!

  • Lindens
Posted
15 minutes ago, Lucia Nightfire said:

Please explain this process in detail. When and how do I opt-out and whom do I contact about it?

If it just entails contacting support about moving my region off of an AWS server, have they already been instructed to oblige such requests for such reasons?

I ask because I have been denied in the past with claims that my region was chosen specifically as a test channel and cannot be moved. This was many years ago and I forget what new feature/code was being tested at the time.

Just to clarify, I made a false assumption that Lucia had previously requested to be one of the first included in the AWS region moves given their active involvement in so many aspects of Second Life. I realize now that was not the case. 

Lucia's region is not currently selected to be moved to AWS this week, as of 2020-10-19 17:20. If at some point their region (or anyone's region) is moved to AWS, it is best to follow the standard processes of filing bug reports for new issues.  Sorry for any confusion, it's been a long Monday.

  • Thanks 3
Posted
1 hour ago, Kyle Linden said:

Just to clarify, I made a false assumption that Lucia had previously requested to be one of the first included in the AWS region moves given their active involvement in so many aspects of Second Life. I realize now that was not the case. 

Lucia's region is not currently selected to be moved to AWS this week, as of 2020-10-19 17:20. If at some point their region (or anyone's region) is moved to AWS, it is best to follow the standard processes of filing bug reports for new issues.  Sorry for any confusion, it's been a long Monday.

So there is no protocol to get a region off of an AWS server that support is aware of and will acknowledge? I would like to be in the last mandatory migration wave if possible, unless the KVP migration happens before then, then I would be interested in going to AWS as script memory handling is on average 2.5x better/faster. It's just the dominant service I offer to the public currently relies on heavy KVP usage which, ATM, is hindered on AWS regions.

Posted
9 hours ago, steeljane42 said:

Same question as Lucia's above.

I'm glad about uplifting starting on the main grid and all, but I'm not ready to deal with *degraded performance or behave incorrectly* part for who knows how long at my (private) region that I pay for. There should be enough of mainland to experiment with for the time being, too. So... how do I opt-out of (possible) early AWS migration?

We pay in mainland too, you know?

  • Like 2
Posted
2 hours ago, MBeatrix said:

We pay in mainland too, you know?

I'm aware. But there's plenty of regions that pretty much no one pays for. Empty water regions that are popular for sailing (blake sea was it?), some LL lands/parks/portals and so on. They get enough traffic so LL can collect some data while not disturbing anyone's homes. That's what I meant. Then again living on the mainland you pretty much agree with less control over your own land/region compared to private estates (less tools and control over everything. can't even restart it without LL support), and LL "owns" some parts of those (some of them at least) regions too; water, roads and so on, so it makes sense for them to start from those.

Posted
47 minutes ago, steeljane42 said:

I'm aware. But there's plenty of regions that pretty much no one pays for. Empty water regions that are popular for sailing (blake sea was it?), some LL lands/parks/portals and so on. They get enough traffic so LL can collect some data while not disturbing anyone's homes. That's what I meant. [...]

I can agree with that. Still, it's not just about traffic, it's also about rezzing stuff, creating scripts and all other things you do in your own land.
No one likes the idea of possible degraded performance but this has to be done, and I'm with Qie — the sooner, the better.

  • Like 1
Posted

I'm guessing, but "a few hundred regions" and www.gridsurvey.com reports 24979 total regions, so around 2% of regions transferred to the Cloud today.

None of the places I usually visit are on the list.

The Grid Drives, a weekly event, usually cover around 150 regions,and so might see 3 regions on the Cloud.

I am more likely to see problems from the DST change in Europe, which takes lace on Sunday..

 

 

Posted
5 hours ago, Qie Niangao said:

Some of Mainland moved already, a few weeks ago. I'd rather they just get on with this, so there's time to revert for a while if there are load scaling bugs.

that was a sim crossing update, they switched how data is sent from the exiting region to the entering region. Mainland was still on the old servers till today

  • Lindens
Posted

Hi All, 

We understand the hesitation to move to the cloud - the warnings of possible performance issues are no joke! The unquestionable benefit of doing so early is that it allows you to raise specific issues that your regions are experiencing to our team and help us fix them before all of the grid moves to the cloud.  Our goal is to move as much of a representative cohort as possible so we can see realistic performance. Just today, because we migrated a large(ish) number of regions, we've identified and are in the process of fixing a bottleneck.

However, if your regions have been moved, we will accept requests to move them back to the datacenter for a while longer, and we will do our best to take note of the regions you would like to NOT be moved to AWS at this time. 

We're excited about the progress in uplift and grateful for your help as we move.  And for your patience too. 

 

  • Like 4
  • Thanks 3
Posted
4 hours ago, VenKellie said:

that was a sim crossing update, they switched how data is sent from the exiting region to the entering region. Mainland was still on the old servers till today

Not all of Mainland; at least some of Linden Village was on AWS a couple weeks ago, and last Friday, Oz's blog post included:

Quote

... over the last couple of weeks we've uplifted around a hundred regions on the main grid. 

I think those hundred regions included some Mainland beyond Linden Village, and in her blog post today, Inara Pey cited that post from Friday, noting:

Quote

At the time of that update, it was indicated that about 100 regions on Agni, the main grid, had been transitioned to running on AWS, comprising a mix of Linden-held, Mainland and private regions.

 

Posted
15 hours ago, Qie Niangao said:

Not all of Mainland; at least some of Linden Village was on AWS a couple weeks ago, and last Friday, Oz's blog post included:

 

i dont see it where in that blog that says the linden village was already moved to aws.

Posted
14 minutes ago, VenKellie said:

i dont see it where in that blog that says the linden village was already moved to aws.

It doesn't. I do, at least for Denby where I attended a meeting where it came up.

Posted

If you have reason to believe that uplift has adversely affected something on your region, you should file a Support request to ask that it be moved back to the datacenter (but keep reading to find out how to do so in a way that will work).

You can tell whether or not a region has been uplifted by going there and looking at the simulator hostname in the Help > About dialog: if it ends in ".lindenlab.com" it's in the datacenter, if it ends in ".amazon.com" or ".secondlife.io" it's uplifted.

Be aware of a couple of things before you make a request to be moved back:

  1. I have asked Support to collect specific information about exactly what has failed and exactly how it has failed for any such request. If we don't get this information, we can't fix whatever it is and we're likely to leave the region uplifted until we can get the data we need.
  2. We are trying to move all of the regions as soon as we can (consistent with carefully listening, and watching all our internal metrics, for problems after each move). This process is going to move as quickly as we can make it happen, so your region(s) will be uplifted pretty soon. We encourage you to actively participate in this process rather than hide from it and hope someone else solves your problem before it affects you. If your region is among the last to move, the time available to address your problem may be very very short; it will be better to have detected it early. Lucia has been testing on the beta grid and other early-uplift regions, and we're very much aware of and working on solutions for the particular problem they have reported; follow Lucias example.

If you report a problem and we can observe it and we think it likely that moving back will fix it (there is some chance that the problem is not a result of the simulator uplift), then we'll move it back while we work on a fix. We have the vast majority of our development resources focused on this migration, but if we don't know about a problem we won't fix it. If we don't fix it, there will come a time in the near future when moving it back will no longer be an option.

Overall, this is going very well; as April said in her post last night - please be patient with us. Really, this is going to be better very soon. 

PS. That note about possible performance problems is true... it's possible ... but mostly what we've seen in the regions uplifted so far is that it's better.

  • Like 2
  • Thanks 6
Posted (edited)

How about those of us using Linden Lab's own viewer where that portion of the URL display was recently removed?  Perhaps I need a different version?

 

 

image.png

Edited by Ardy Lay
You are about to reply to a thread that has been inactive for 1489 days.

Please take a moment to consider if this thread is worth bumping.

Please sign in to comment

You will be able to leave a comment after signing in



Sign In Now
×
×
  • Create New...