Can we please have the Deploy Plans for that Week so we know if what will happen?

[Miller Thor]

To know if we maybe get an Update or get migrated to the Cloud is important for all Region owner.

[arabellajones]

It looks as though there is a steady trickle of migration after the few hundred of last Tuesday. It would be worth knowing how many there have been.

My guess is six weeks to get the migration finished, because of possible server fixes and Christmas/New-Year/Thanksgiving/Election, an average of about 4000 per week. Every working day would be about twice what we've had so far.

It gives me the same feeling as current British politics, a rather indigestible lump of change and chaos descending on us through a failure of planning and leadership.

[Miller Thor]

I see less lack of planlessness or the intention to cause chaos, as is the case in the UK at the moment, but more the unwillingness to communicate with the paying customer. Unfortunately, this has always been a major weakness at Linden Labs. Just as it is obviously still not understood that the European customer paying for his regions also wants to be informed in a timely manner, and I understand by that in the morning, after European time. This allows you to easily plan your work in SL and take into account the restart time in the scheduling. Even if you just give out the information that nothing is happening, it is very valuable for people who are working on their regions, or who are planning to do so.

[Sayrah Parx]

I've asked a few times in the past for the maintenance restart warning time to at least be doubled, from 5 minutes to 10 minutes.  I don't understand why it would be a problem for the rolls to end 5 minutes later, or to start the rolls 5 minutes earlier if they really want it to end at the same time.  It is about having respect for the customers who are paying for the regions, and the people who manage the regions for the people paying for them.  I'm not really sure that the employees understand that people are paying for this, and that it's not just their personal sandbox that they get paid to mess around with however they like.

[Aishagain]

I suspect that there will be few or maybe no further Deploy announcements from LL until the region uplift is either completed or abandoned temporarily.  There are likely to be sporadic restarts to groups of regions at any time, though it cannot be predicted quite which ones or when.

Sayrah's request for more warning is the most likely request to be acceded to but I hold out little hope of that.

Miller...you, like me, are irritated by the widespread inability of Americans to realise that a world exists outside of the USA's geographic boundaries and timezones.  I see no sign of that changing any time soon.

 

[arabellajones]

There's a message out in status. I read it to mean that more regions are being uplifted today, from the position of the word "today".

"We will be performing rolling restarts for regions that have been Uplifted today beginning at 11:45 AM PDT. Please refrain from rezzing no copy objects and remember to save all builds. Please check this blog for updates."

 

[Ardy Lay]

Gosh.  That is ambiguous.

[Qie Niangao]

This is from the Simulator User Group, happening now:

Quote

[12:04] The Spraycan (mazidox.linden): We're rolling all uplifted regions right now to put out a build with several fixes and better logging than I've ever seen in my tenure at Linden Lab.
[12:04] Torric Rodas: elise did my body make it with the rest of me?
[12:04] The Spraycan (mazidox.linden): (I am very happy about better logging, if you can't tell.)
[12:04] Beq Janus: better logging or more logging?
[12:04] Rider Linden: So... there are no RC or SLS rolls this week. There is a new Uplift roll imminent however.
[12:05] The Spraycan (mazidox.linden): *Better* logging
[12:05] Joe Magarac (animats): Nice. On AWS there are more logging capabilties.
[12:05] The Spraycan (mazidox.linden): Actually Rider, that's not true anymore (surprise!)
[12:05] Nal (nalates.urriah): How much of the region server grid is in the cloud?
[12:05] Lucy daughter of the Devil (lucia.nightfire): communication!
[12:05] Beq Janus: That's why Rider comes to these meetings, to find out what's going on
[12:05] Lucy daughter of the Devil (lucia.nightfire): 4%?
[12:05] Rider Linden: I should wait to get my up dates at 5 till the hour from now on then.
[12:06] The Spraycan (mazidox.linden): It was true the last time Rider and I spoke about it
[12:06] Rider Linden turns over the floor to Maz
[12:06] The Spraycan (mazidox.linden): And then we realized we needed more data on whether we fixed capabilities not being granted to viewers, so we're putting the current build out further than we initially planned.
[12:06] Torric Rodas: are you hoping this roll will fix "failed to rez" issues, or is that still ongoing?
[12:07] Lucy daughter of the Devil (lucia.nightfire): I hope it fixes the rando tp timeout crashes...
[12:07] The Spraycan (mazidox.linden): I won't promise it Torric, but I expect it will.
[12:07] Torric Rodas: ooh that would be good... and also failed to add inventory to scripted objects?
[12:07] The Spraycan (mazidox.linden): Those aren't uplift specific unfortunately Lucia.
[12:07] The Spraycan (mazidox.linden): Same issue, yeah Torric.
[12:07] Lucy daughter of the Devil (lucia.nightfire): they've only happened to me on uplift regions, how lucky am I
[12:07] The Spraycan (mazidox.linden): Now they do happen more often on uplifted regions, and that we think we have licked (at Lucia again)
[12:08] Torric Rodas: Mazi, I've been talking with a lot of you guys about TP Fail, and it is far far worse in the cloud. We're guessing latency. I know its grid wide, but we've been testing it and the fails are about double the rate of non-AWS
[12:08] The Spraycan (mazidox.linden): And, for the first time ever, we have uplifted simulator release notes
[12:08] Simon Linden: is that AWS-to-AWS, or between new and old?
[12:09] The Spraycan (mazidox.linden): https://releasenotes.secondlife.com/simulator/2020-10-26.551155.html
[12:09] Beq Janus: Torric, is that cloud to cloud or hybrid?
[12:09] Beq Janus: lol Simon is faster than me
[12:09] Second Life: Aelva is offline
[12:09] Lucy daughter of the Devil (lucia.nightfire): I tp timeout crashed on Cloud Sandbox 3 3x in 2 hours yesterday going from AWS to AWS
[12:09] The Spraycan (mazidox.linden): This is literally just the tip of the iceberg of the insane number of changes we made to get simulators running in the cloud.
[12:10] Torric Rodas: I don't know Simon, but we've just set a new portal park up ourselves and we're seeing about 30% fail rate to the 100+ destinations we have in there on a timer that rotate
[12:10] Torric Rodas: all destinations are verifed

Maybe more coming, but thought this might be of immediate interest to those engaged with this thread.

[ETA] More:

Quote

[12:21] The Spraycan (mazidox.linden): So as of... I don't know, last Tuesday night after everyone had gone to bed, I think? We had about 1,000 regions uplifted, and they've run with only minor hiccups on our end all week.
[12:23] The Spraycan (mazidox.linden): We anticipate tomorrow increasing that by around *checks notes* my notes say over 2000, but let's be ambitious and say 2500?
[12:23] Lou Netizen: wow
[12:23] Joe Magarac (animats): Randomly across the grid, or contiguous?
[12:23] Oz Linden: Keep Your Fingers Crossed 🙂
[12:24] The Spraycan (mazidox.linden): It will look random to you, but it's not random (because when is anything ever random?)

 

Edited October 27, 2020 by Qie Niangao

[arabellajones]

When is anything ever random? Oh, come on guys, it's quantum all the way down.

 

[AlettaMondragon]

Okay, so if I interpret things correctly from what Qie posted, the intention is to only focus on the Uplift and rolls for the uplifted regions to work on the issues at the same time of moving more and more sims to AWS hosts, so we don't have to bother with rolling restarts for datacenter hosted regions anymore, right? (As they'll slowly roll out anyway if things go well.)

(Edit: I realize this fails a bit at "which regions will be uplifted in the next round?", it's still important to let people know when there is a chance their regions will be uplifted, instead of suddenly realizing when it happens, without the chance to expect it.)

Edited October 27, 2020 by AlettaMondragon

[Qie Niangao]

45 minutes ago, AlettaMondragon said:

... we don't have to bother with rolling restarts for datacenter hosted regions anymore, right?

I think that was the original plan for the week, but I understood the following to mean that some non-Uplifted regions may get an update this week, too:

Quote

[12:06] The Spraycan (mazidox.linden): And then we realized we needed more data on whether we fixed capabilities not being granted to viewers, so we're putting the current build out further than we initially planned.

... but that interpretation doesn't quite align with Inara Pey's blog -- and frankly she has a lot better record of getting things right!

Edited October 27, 2020 by Qie Niangao

[arabellajones]

There are two announcements about a possible rollout next week, RC channels only. One for Tuesday, one for Wednesday. I suspect a typo in the first to be posted.

https://status.secondlifegrid.net/incidents/p36bn1s1p6p6

https://status.secondlifegrid.net/incidents/grpp7zhm3t1v

I shall be keeping an eye open for restarts both days, though I hear there's some sort of election happening next week.

 

[Logan Elf]

I think those are semi-automated, so maybe they meant uplifted regions for Tuesday. Or perhaps planning more uplifts.

[Aishagain]

10 hours ago, arabellajones said:

I suspect a typo in the first to be posted.

I suspect that also...what makes me think otherwise is that it has not (as yet) been either Ninja edited or otherwise ammended.  Sure it might be automated.  Am I then to assume that no one at LL has noticed the error??  I don't like to think about what that says about the Lindens on shift.

[Lucia Nightfire]

We're pretty much at the phase where channel variance is primarily going to be AWS or Non-AWS until all regions are on AWS.

Then you will start to see channel variance at the coding/feature/fix level again.

[arabellajones]

14 hours ago, Aishagain said:

I suspect that also...what makes me think otherwise is that it has not (as yet) been either Ninja edited or otherwise ammended.  Sure it might be automated.  Am I then to assume that no one at LL has noticed the error??  I don't like to think about what that says about the Lindens on shift.

It's a weekend. I can imagine some limits on who can edit a Status announcement. It doesn't look good, but it could be one of those edge cases.

But there should be a better routine process, even if there is an automatic weekly post here that reminds people of the normal timing and provides a hook to hang more precise info on. Something like:

"We wish to remind Residents that sim servers are regularly restarted in two blocks on Tuesdays and Wednesdays, for a variety of reasons. More complete details of what to expect in the coming week will be posted to this thread later."

And then they will still ignore the paying customers.

[Willow Wilder]

On 10/31/2020 at 8:58 AM, LoganPryor said:

I think those are semi-automated, so maybe they meant uplifted regions for Tuesday. Or perhaps planning more uplifts.

It was said at the TPVM on Friday that regions to be uplifted would increase significantly, so it didn't seem like a mistake when I read it. See 2020 TPVD MEETING WEEK #44: SUMMARY WITH UPLIFT NEWS

 

[Aishagain]

Reading ALL the above and applying a little common sense, the announcements are to me simply sloppy.  Sure, we do not (or at least ! do not) expect the uplift to suddenly occur in one fell swoop but referring to uplifted and datacentre regions as RC channel seems plain lazy (unless of course, it is only RC channels that are being uplifted still, in which case it is a little less inaccurate).

Edited November 1, 2020 by Aishagain

[Sayrah Parx]

Sandbox Island is a main grid AWS region that looks good for testing.  Anyone know any others?

[Logan Elf]

4 hours ago, Sayrah Parx said:

Sandbox Island is a main grid AWS region that looks good for testing.  Anyone know any others?

Some require premium membership, but these sandboxes are all AWS regions:

• Sandbox Cordova
• Sandbox Goguen
• Sandbox Wanderton
• BlueSteel Sandbox A
• LeTigre Sandbox A

 

[animats]

From the last server user group meeting, the plan was to "uplift" about 2000 regions each night. So it should be done within two weeks.

[arabellajones]

4 hours ago, animats said:

From the last server user group meeting, the plan was to "uplift" about 2000 regions each night. So it should be done within two weeks.

That is going to make things interesting for folk in some parts of the world.

As I understand it, the uplift process can't be invisible. Maybe not, strictly, a restart, but there's some risk of losing changes. We haven't even been told we should be careful about making changes at some particular time of day, other that these ambiguous restart warnings.

2 days per week at 2000 per day, doen't feel silly as an Uplift rate, call it 5 weeks, but it's too much like mushroom farming: kept in the dark and fed on bull*****.

[animats]

6 hours ago, arabellajones said:

That is going to make things interesting for folk in some parts of the world.

As I understand it, the uplift process can't be invisible. Maybe not, strictly, a restart, but there's some risk of losing changes. We haven't even been told we should be careful about making changes at some particular time of day, other that these ambiguous restart warnings.

2 days per week at 2000 per day, doen't feel silly as an Uplift rate, call it 5 weeks, but it's too much like mushroom farming: kept in the dark and fed on bull*****.

It is invisible to most users. Right now, if you drive around on mainland roads, you're crossing from sims hosted in the Linden Lab data center in Arizona to sims hosted in the northern California Amazon Web Services data center. And back. You probably won't notice.

Those of us who develop within SL were testing on the beta grid months ago. Bugs were reported and fixed long before main-grid user-visible sims were moved to AWS. There were some big problems at first, JIRAs were filed, people reported trouble at Server User Group meetings, and the problems were fixed. There are still some obscure timing problems, but some of those will go away when everything is on AWS.

I'm often critical of the dev team, but this migration is going well.

[arabellajones]

35 minutes ago, animats said:

It is invisible to most users. Right now, if you drive around on mainland roads, you're crossing from sims hosted in the Linden Lab data center in Arizona to sims hosted in the northern California Amazon Web Services data center. And back. You probably won't notice.

 

I am not talking about a region crossing. Are you changing any part of the vehicle during the region crossing? Currently, during region crossing, events may be sometimes happening out of order. Sometimes it involves a Cloud region, sometimes not.

Are you saying a Cloud uplift works perfectly every time? Are you willing to take that chance if you're building something?

[Mazidox Linden]

On 10/31/2020 at 12:45 AM, arabellajones said:

There are two announcements about a possible rollout next week, RC channels only. One for Tuesday, one for Wednesday. I suspect a typo in the first to be posted.

https://status.secondlifegrid.net/incidents/p36bn1s1p6p6

https://status.secondlifegrid.net/incidents/grpp7zhm3t1v

I shall be keeping an eye open for restarts both days, though I hear there's some sort of election happening next week.

 

Nope! We're doing two sets of changes this coming week, one on Tuesday to put more regions in the cloud running the code we deployed last week, and another on Wednesday to update the code running some regions already in the cloud with fixes to make startup and shutdown a little bit simpler and not rely on as many things still in our datacenter. And maybe, if we're really lucky, some bug fixes we're testing today as well; will let you all know with a Server Deploy Plan once we have it nailed down.