rolling restarts What is with the crazy restarts on Terror Tuesdays? How do we have events?

[Caleb Kit]

We in the Raglan Shire sims have had a building event every Tuesday for well over 10 years from 10:30AM SLT until 12NoonPM SLT.  We now call Tuesdays, "Terror Tuesdays".  In recent months as the server people are unable to carry out quick Rolling Restarts of the Main Channel sims which start at 6AM SLTand go until late in the afternoon.   We have had to build 2 backup sites in our sims to hold the event instead of on our main sim of Raglan Shire. Land Mangers(Officers) have to be present to monitor the sim status units we have placed on all of the sims to tell us when they have restarted.  And we have to watch like hawks and then decide at the last moment where we will hold our event.  Even so we have had to cancel the event for the first time in over 10 years once and been thrown out of the sim twice just in the last 3 months.  So, what should we do?  We are now frustrated with orange clouds, having to relog all the time to view things.  Getting gaming computers doesn't seem to help.  How much more money should we pump into this platform?

Caleb Kit----Land Manager ---Raglan Shire Activites and Events Group--- Premium Member-- Over 15 years Second Life Resident

[Aishagain]

@Caleb KitWell after 15 years of SL I am astonished that you try to hold events in the middle of Main Channel Restart day!  I have been in SL since 2008 and I can remember whole days of SL being inacessible due to Rolling Restarts. The fact is that the restart procedure, which was relatively straightforward when SecondLife sim-servers were in an LL-owned part of a data centre, is far less so now that SL is hosted on AWS' cloud servers. That has been true for 18 months now (I think) and throughout that period there have been issues of one sort or another with the restart schedule.

Having a building event on a Tuesday strikes me as being perverse in the extreme anyway, if your region is on SL's Main Server Channel.  In my opinion being caught out twice in the last three months suggests that your experience is no worse than many of us and having to cancel an event once in ten years strikes me as very reasonable indeed.

No one enjoys these protracted rolls, least of all the technicians who need to work extended shifts to keep tabs on the servers, but I would rather they WERE protracted than have borked rolls and roll-backs.

Nowhere in LL's ToS do they promise an uninterrupted service and they would be fools to do so.  As to rezzing issues and the like. most of those are caused by glitching internet connections and poorly set-up PCs.  I've not been immune to some of those myself...it's frustrating but that's life, first OR second.

Edited May 31, 2022 by Aishagain
syntax

[Rowan Amore]

1 hour ago, Caleb Kit said:

So, what should we do? 

Schedule on any day besides Tues?

[Linn Darkwatch]

And yet we have had that event on the same day/time for over a decade without these issues until relatively recently.

Aishagain wrote
>... I would rather they WERE protracted than have borked rolls and roll-backs.

As we had borked physics last week after the roll, with people starting to fall through a previously solid platform, I'm not sure the extra time is always helping.

Edited May 31, 2022 by Linn Darkwatch

[Kathlen Onyx]

3 hours ago, Caleb Kit said:

Even so we have had to cancel the event for the first time in over 10 years once and been thrown out of the sim twice just in the last 3 months.  So, what should we do? 

Consider yourself very lucky? Doesn't sound overly problematic to me.

[Aishagain]

@Linn Darkwatchdid I say that it was either/or?  No; some of the work done recently has NOT been of a high standard and I, like many others have complained.  There is much that they "could do better".  Linden Lab's user facing approach still is FAR from satisfactory, the repeated litany of "attend Server User Group meetings" is frequently trotted out.  I cannot, for very good RL reasons.

One thing that I fervently wish LL would address is the frequent silence on the Server Thread regarding the upcoming rolls.  Some weeks it is more important than others but it is still necessary, in my opinion.

 

3 hours ago, Linn Darkwatch said:

And yet we have had that event on the same day/time for over a decade without these issues until relatively recently.

In that case it is my opinion that you have been VERY fortunate.

Edited June 1, 2022 by Aishagain

[Profaitchikenz Haiku]

14 hours ago, Caleb Kit said:

So, what should we do?  We are now frustrated with orange clouds, 

Get yourselves orange cloud avatars and just carry on regardless?

Edited June 1, 2022 by Profaitchikenz Haiku

[Caleb Kit]

Thank you for all of your responses.  Many of you are really good trolls.   We have a community here. And some of the very best builders in SL.  It has always been that SL "Is a world created by it's residents".  We have community events every week. And no, rolling restarts didn't always happen on Tuesdays.  And until the last few months they were over with well before our regular event of over 15 years.   And yes I remember the old days where every few weeks there would be a day that you tried to log in and you got  the gorillas with the big bones and the message of "Sorry for the inconviencnce but we need to bang around on things".  I also remember a lot of worse things we put up with to build our communities.  Meanwhile putting up with high tier prices and more and higher fees. Less support from the Lindens.  More gentrification in world. And so on.  Actually we are getting this issue resolved as other private sim communities have contacted us and we are now working together.   So have a nice Second Life.

[Love Zhaoying]

12 minutes ago, Caleb Kit said:

Actually we are getting this issue resolved as other private sim communities have contacted us and we are now working together.  

Please let us know the outcome! For instance, if you are able to convince the Linden Labs team to schedule your restarts for a specific day of the week, and time of the day.

Thanks,

Love

[Kathlen Onyx]

2 hours ago, Caleb Kit said:

Thank you for all of your responses.  Many of you are really good trolls.   We have a community here. And some of the very best builders in SL.  It has always been that SL "Is a world created by it's residents".  We have community events every week. And no, rolling restarts didn't always happen on Tuesdays.  And until the last few months they were over with well before our regular event of over 15 years.   And yes I remember the old days where every few weeks there would be a day that you tried to log in and you got  the gorillas with the big bones and the message of "Sorry for the inconviencnce but we need to bang around on things".  I also remember a lot of worse things we put up with to build our communities.  Meanwhile putting up with high tier prices and more and higher fees. Less support from the Lindens.  More gentrification in world. And so on.  Actually we are getting this issue resolved as other private sim communities have contacted us and we are now working together.   So have a nice Second Life.

Information that you don't want to hear = Troll

Ok then.  I guess you are special because your event on the day and time you want to hold it must be more important that anyone else. What a clusterF if everyone wanted to choose their own time for restarts and how would they determine who gets priority over others? Food for thought.

[Aishagain]

3 hours ago, Caleb Kit said:

More gentrification in world.

I am fascinated by this phrase.  What on earth do you mean by it?

Edited June 7, 2022 by Aishagain

[Rowan Amore]

3 hours ago, Caleb Kit said:

Many of you are really good trolls

No one was trolling.  You asked a question...How do we have events?  You got answers.  Answers you don't like are just that and NOT trolling.

I'm sure if it were an easy fix to borked restarts, LL would have implemented it already but best of luck with your efforts.

[Alwin Alcott]

11 hours ago, Caleb Kit said:

 I remember the old days where every few weeks there would be a day that you tried to log in and you got  the gorillas with the big bones and the message of "Sorry for the inconviencnce but we need to bang around on things".  

it wasn't every few weeks.. it was every week, announced and known, nothing surprising, and always on "prime time" for Europe.
Of course we seen those screens also when the grid went down for unscheduled maintenance.( sometimes also very frequently ...)

[Gabriele Graves]

Caleb and The Raglan Shire folks are generally a good bunch who just want to enjoy their community events.

I don't think any of the responses were trolls or bad but I can understand Caleb's frustration and think it must have reached a tipping point to have come here to complain in the hopes that someone could help and when you feel like that you are probably looking to vent and for a bit of sympathy as well.  It's easy to interpret the reality as of the situation as uncaring when it isn't in fact.

Trying to find alternate times when a large group can attend can be really difficult.
Caleb, I hope you manage to find a solution that works out.

[Prokofy Neva]

On 6/7/2022 at 1:44 PM, Caleb Kit said:

Thank you for all of your responses.  Many of you are really good trolls.   We have a community here. And some of the very best builders in SL.  It has always been that SL "Is a world created by it's residents".  We have community events every week. And no, rolling restarts didn't always happen on Tuesdays.  And until the last few months they were over with well before our regular event of over 15 years.   And yes I remember the old days where every few weeks there would be a day that you tried to log in and you got  the gorillas with the big bones and the message of "Sorry for the inconviencnce but we need to bang around on things".  I also remember a lot of worse things we put up with to build our communities.  Meanwhile putting up with high tier prices and more and higher fees. Less support from the Lindens.  More gentrification in world. And so on.  Actually we are getting this issue resolved as other private sim communities have contacted us and we are now working together.   So have a nice Second Life.

I think what's different about it now as compared to recent years is that the sims are down for so long. It's not uncommon for a sim to go 30, 45, 60 minutes down -- or to go down repeatedly. This plays havoc not only on events but rentals where people can't go home for long periods. 

Why are they taking so long? 

[Love Zhaoying]

1 hour ago, Prokofy Neva said:

Why are they taking so long? 

3 guesses:

1) AWS updates may take longer

2) Updates could be more major

3) There could be a new, more thorough verification process in place, which occurs before the simulator servers are "brought back up"

 

[Aishagain]

10 hours ago, Prokofy Neva said:

Why are they taking so long? 

This puzzles me.  When Second Life was hosted on Linden Lab's own servers inside a terrestrial data centre, I can recall that a rolling restart instance where new software was being rolled out would take upwards of 20 minutes, sometimes an hour.  Over the years that time did reduce somewhat but I did not start to worry unless my home region was down for more than 45 minutes.

Following our elevation (cough) to the cloud such restarts could take over 90 minutes since the new server hardware appeared less happy to accept new software and Oz Linden and others  noted that the process was "not as smooth as we would wish"

Recently Mazidox Linden told us that rolls were being done more slowly out of  "an abundance of caution", and some servers appeared reluctant to restart promptly.

Perhaps I have been fortunate but my home has only once in recent months showed a reluctance to restart, and even so the downtime was less than two hours.

As Love Zhaoying notes above: There could be a new, more thorough verification process in place, which occurs before the simulator servers are "brought back up". This is, as I understand it, so.  Perhaps we should be glad that LL are proceding with "an abundance of caution" now, since the alternative would likely entail a good deal of data (asset) loss.

I have an analogy.  You build a tower of cards:  now, as you add new and better cards to this tower, you need to remove some of the older redundant cards from lower in the stack.  If this is not done carefully, the entire pile comes crashing to the ground.  Not a good scenario. Therefore all additions are done with great care.

SecondLife is a very high, somewhat unstable tower of cards.  I think we would all prefer that it did not come crashing down.

Edited June 9, 2022 by Aishagain
Additional content

[Prokofy Neva]

On 6/9/2022 at 12:58 PM, Aishagain said:

This puzzles me.  When Second Life was hosted on Linden Lab's own servers inside a terrestrial data centre, I can recall that a rolling restart instance where new software was being rolled out would take upwards of 20 minutes, sometimes an hour.  Over the years that time did reduce somewhat but I did not start to worry unless my home region was down for more than 45 minutes.

Following our elevation (cough) to the cloud such restarts could take over 90 minutes since the new server hardware appeared less happy to accept new software and Oz Linden and others  noted that the process was "not as smooth as we would wish"

Recently Mazidox Linden told us that rolls were being done more slowly out of  "an abundance of caution", and some servers appeared reluctant to restart promptly.

Perhaps I have been fortunate but my home has only once in recent months showed a reluctance to restart, and even so the downtime was less than two hours.

As Love Zhaoying notes above: There could be a new, more thorough verification process in place, which occurs before the simulator servers are "brought back up". This is, as I understand it, so.  Perhaps we should be glad that LL are proceding with "an abundance of caution" now, since the alternative would likely entail a good deal of data (asset) loss.

I have an analogy.  You build a tower of cards:  now, as you add new and better cards to this tower, you need to remove some of the older redundant cards from lower in the stack.  If this is not done carefully, the entire pile comes crashing to the ground.  Not a good scenario. Therefore all additions are done with great care.

SecondLife is a very high, somewhat unstable tower of cards.  I think we would all prefer that it did not come crashing down.

What is the checking and abundance of caution *about*?

[Aishagain]

@Prokofy NevaI was quoting Mazidox Linden's words, quite what he meant I would not like to say but I would imagine that the technical folk were watching the boot-up process for each sim-server closely to see if they could identify any rell-tale signs of incomplete loading of their software.

[Prokofy Neva]

On 6/10/2022 at 3:55 PM, Aishagain said:

@Prokofy NevaI was quoting Mazidox Linden's words, quite what he meant I would not like to say but I would imagine that the technical folk were watching the boot-up process for each sim-server closely to see if they could identify any rell-tale signs of incomplete loading of their software.

Something doesn't add up here. Before AWS, the Lindens rented servers from a server farm that "wasn't in the cloud," i.e. "wasn't other people's servers" in the aggregate, which is what the cloud essentially is. So now they presumably have more firepower, although it is more expensive, and more robust anti-hacking/anti-griefing mesasures from AWS engineers. Do they have complete control over their own servers? I should think they have the same panel that anyone has to check their servers. But maybe the delay is because they have to make requests that then have to be answered, and not automatically? How could that be in this modern world? Why do they have to do more checking and take longer than they used to with the terrestrial server farm? 

Of course, they are not required to answer these questions as they are a private company and not publicly traded and may decide this is proprietary information. Seeing sims go up and down again on a Sunday morning and taking a long time to reappear, I do wonder what it is all about. 
 

Given what is on some SL servers, I can totally understand if they become reluctant to start up again.

Edited June 12, 2022 by Prokofy Neva

[Qie Niangao]

My recollection is that the slower restarts only became common in the past few months, with earlier AWS-based restarts about as fast as longer ago, on Lab's hardware. I understand they've made some changes in AWS configuration over time, but I don't know if that has anything to do with the slower restarts or if it's some system change unrelated to hosting.

Apparently the OP has written us off as "good trolls" (finally, a promotion from Evil Troll) so he's on his own, but one suggestion might be to ask the Lab to move the region hosting the Tuesday events to one of the RC channels so the regular restarts would come on Wednesday instead.