Deploys for the week of 2013-06-24

[Maestro Linden]

Second Life Server (main channel):

The main channel is getting the server maintenance project that was on all three RC channels last week.  This project fixes some crash modes, addresses an issue with neighboring region visibility, and adds some new scripting features.

https://wiki.secondlife.com/wiki/Release_Notes/Second_Life_Server/13#13.06.18.277494

Scheduled Tuesday 2013-06-25 05:00-12:00 PDT

 

Second Life RC BlueSteel, Second Life RC LeTigre, and Second Life RC Magnum:

All three RC channels are getting a new server maintenance project.  This project fixes some crash modes and fixes some LSL-related bugs.

https://wiki.secondlife.com/wiki/Release_Notes/Second_Life_RC_BlueSteel/13#13.06.21.277682

https://wiki.secondlife.com/wiki/Release_Notes/Second_Life_RC_LeTigre/13#13.06.21.277682

https://wiki.secondlife.com/wiki/Release_Notes/Second_Life_RC_Magnum/13#13.06.21.277682

Scheduled Wednesday 2013-06-26 07:00-11:00 PDT

 

We will be monitoring this thread as the code gets released, so feel free to note any observations you have about the server updates.  If you have a specific bug you'd like to report, please file a Jira. 

[Imagin Illyar]

It would be great if someone could update the Grid Status Report website when these rolling restarts start and finish.  I've had tenants lose no-copy items during these and would appreciate knowing when they start and finish.

[Levio Serenity]

I would have them file a JIRA on this. No objects should be lost during a planned restart, only from unexpected crashes. However, this happens with an alarming frequency. I don't think LL realizes this is happening all the time.

 

[Imagin Illyar]

When they do update the Grid Status Report they warn not to rez no-copy items or make linden transactions, which is a pretty serious warning that depends on someone actually remembering to check the GSR page in the first place.  It's pretty bad when they don't bother to update the page though.  I check it daily on behalf of my tenants and send out a group notice so people are aware, I can't do that if they don't update the GSR page.  And yes, losing no-copy items during rolling restarts happens quite a lot.  

[Maestro Linden]

Hi Imagin, I would recommend checking the Calendar at http://status.secondlifegrid.net/ to see the rolling restart schedule in advance.  This calendar should be updated a few days in advance to show any non-emergency rolls.

[Imagin Illyar]

I do, and that's the only way I knew about today's RR.  However, the restarts rarely start at 5am as scheduled and the Calendar doesn't show when they are finished.  I would recommend you have someone update the actual Grid Status Report more promptly regarding rolling restarts as they do cause people to lose inventory.  

[Ron Khondji]

You could follow @SLGridStatus on twitter.
I do not know if that´s more reliable than the website though.

[Ayesha Askham]

@Ron

It is no more reliable in its content than the Grid Status page since the information is from the same source.  The issue with the slow, inaccurate or sometimes just plain wrong information on the GSP has been a pain for as long as I can remember.

Why Linden Lab cannot get this small issue sorted reliably I do not know, it is an internal matter that LL personnel really need to address.

@Maestro

Since the update to Main Server I have noticed that scripted HUDs are randomly detaching, ignoring RLV constraints, upon TP between restarted Main Server sims.  Also some scripted functions are apparently being borked.  If I get a repro on this I will post a JIRA.

[Maestro Linden]

Okay, keep us posted, Ayesha.

One possibility is that the scripts are executing 'llRequestPermissions(agent,0)', and are expecting a run_time_permissions() event to trigger.  This behavior has been 'broken' for since 2007 (see SVC-1006), but with the 13.06.18.277494 update, it is now 'broken' for attachments and sat-upon objects in addition to other types of objects.

[Perrie Juran]

---

Maestro Linden wrote:

Hi Imagin, I would recommend checking the Calendar at

http://status.secondlifegrid.net/

 

to see the rolling restart schedule in advance.  This calendar should be updated a few days in advance to show any non-emergency rolls.

---

Does the ability to send a grid wide notification In World not exist?  One that people would see when they log in or are logged in?

I know it can be difficult to Child Proof everything, but especially when unscheduled problems / maintenance occurs, many of us have been caught with our pants down.

This sure would be the cat's meow if it could be done.

[Amber Murfin]

@maestro.

 

Can i just say, SVC-8130 (and my BUG-2951 with more information) Appears to be fixed as well. I need to do a bit more testing to fully confirm this, but when in Following Seas, i can see all the sims i was previously having trouble with.

 

Thankyou

[Cincia Singh]

I think what is at the heart of the issue here is the same thing that's been discussed in the past; people being distracted and not noticing the warnings that begin 5 minutes before a region undergoes a scheudled restart on Tuesday or Wednesday each week. Since the rolls take a couple hours minimum I don't see the value of a grid-wide message unless it's spammed repeatedly to catch people as they log in.

[Ayesha Askham]

@Cincia

Your point is valid, those warnings are hard to ignore, and those that choose to suppress them have only themselves to blame if they are "caught out" by a restart.

However, it remains true that if you TP into a region that is within the 5 minute warning window you do not get subsequent warnings and the disconnect is unannounced and therefore a surprise (at the least).  This issue is one that needs to be addressed ASAP.

I maintain my point made in my earlier post that the accuracy of the GSP posts is totally inadequate and while I appreciate that Main Channel Restarts begin well before normal working hours in the Pacific Time Zone, the 24/7 nature of SecondLife means that the issuing of information via the GSP and feeds must also be reliable 24/7.

[Qie Niangao]

---

However, it remains true that if you TP into a region that is within the 5 minute warning window you do not get subsequent warnings and the disconnect is unannounced and therefore a surprise (at the least).  This issue is one that needs to be addressed ASAP.

---

Because my median "dwell" in a sim is a less than a minute, I get hit by this often if I'm in-world while the rolling restarts are happening. It's annoying to have to relog (often into an infohub), but for me it's not a top priority.

What I would consider a priority, however, is content loss during restarts. This seems to happen way too often to be a fluke, and I think it warrants some attention to the process. It would be good to know how often the sim state is not fully persisted before shutdown. I'd suppose that Operations would want to watch that metric for every week's deployments anyway, and if ever more than one sim fails to persist on the same week, it should be fixed before the next round of restarts.

[Imagin Illyar]

It's not just during the 5 minute warning frame that items are lost.  I myself have notice work undone up to an hour before a sim is restarted, landscaping lost on a lot I'm working on and also changes that I've made at my tier boxes reverting back to as they were before I made the changes.  Several times I've had tenants pay tier only to have the tier box lose the payment, I can confirm their payment in my transaction history and see the payment was made well before the sim was restarted.  It's as if there was a mini roll back along with the restart.  It is most frustrating and I've come to rely on the GSR to tell me when the rolling restarts begin and end, so that I can avoid doing anything until they are over, but this requires reasonable updating of the page.

[Norton Burns]

As your region may be backed-up up to an hour before the roll, as I understand it, it makes only sense to know that every Tuesday or Wednesaday, depending on your region's channel, if you want to build after 5am on that day - go to a sandbox on another channel.

Otherwise you could just lose a new something every week & never get any wiser, I guess.

[Qie Niangao]

Except that's not how it's supposed to work, at least not as I understand it. What's supposed to happen before a scheduled restart is that the sim disconnects all agents and neighboring sims and then, before shutting down to restart, persists all state that's changed since the last full backup (which indeed may have been an hour or even more before the restart process began). It's that incremental saving of state that doesn't seem to complete as reliably as one would expect.

[Norton Burns]

Not heard of any incremental happening - which in no way means it isn't true - but there must be a reason that the standard "Please refrain from rezzing no copy objects, making inworld L$ transactions and remember to save all builds." warning goes out every time.

Taking heed of that means I have never lost an item since I first discovered that whilst editing an attachment, periodically removing & re-attaching it was the way to save your work, in case you get kicked whilst afk.

[Perrie Juran]

---

Qie Niangao wrote:

Except that's not how it's supposed to work, at least not as I understand it. What's supposed to happen before a scheduled restart is that the sim disconnects all agents and neighboring sims and then, before shutting down to restart, persists all state that's changed since the last full backup (which indeed may have been an hour or even more before the restart process began). It's that incremental saving of state that doesn't seem to complete as reliably as one would expect.

---

Having been a victim of losing no copy items myself I am going to echo this.

Knowing LL Support's answer to this, "sorry, you're out of luck," I didn't even bother reporting.

It could be that

A.  LL is aware of the continued problem with this and is just being mum or

B.  Isn't aware or does not think it is happening frequently enough to warrant attention.

I don't think there are enough of us who it happens to or that it happens frequently enough that we could provide Maestro with time stamps, etc for him to investigate and fix this.

What I do do now is heed the warning to not rez any no copy items until the 'all clear' is sounded for the Grid.

[WolfBaginski Bearsfoot]

This week, the dates on Status have been wrong.

I can sort of see why the roll-out process starts before an announcement is made, something to do with the permissions needed to post a notice, so that the pre-announcement gives a start time (5am for Main Channel) and the actual in-progress comes out around 7am. It looks careless.

It's reliable in the sense that the pattern is predictable.

But there's no reason why, when you start up the Viewer, the log-in screen cannot show the latest status message. But if there's a couple of hours delay on the restarts-in-progress announcement, doing that isn't going to work well. There are ways of getting the information in front of people, but it should be better information that we get now.

(It seems a little strange that Linden Labs doesn't appear to trust the guy in charge of executing the roll-out to post a prepared message to Status)

 

[Checks: Firestorm reports the three most recent Status posts, currently all tagged as Resolved]

[Ayesha Askham]

The issue of the state-save immediately prior to shutdown of a simserver and its capricious failure is something that Linden Lab's Restart Team have known for over 12 months now.

That nothing has apparently been done about it suggests more a failure of internal Linden communication more than anything else.  If you are fortunate (or unfortunate, depending on your viewpoint) to hear an ex-Linden on this topic you will know that internal communication at the Lab is not what it ought to be.  I've heard the term "herding cats" used.

That having been said and at the risk of repeating myself, the Grid Status Page is an atrocious advertisement for the dysfunctional nature of Linden Lab.  That the RC Roll being resolved information is now appearing before the Main Server Roll information is just plain sloppy editing. According to that the RC rolls were resolved before Main Server! Get your house in order Linden Lab.

[Lucia Nightfire]

I emphasized the problem in https://jira.secondlife.com/browse/SVC-7959 last year. The region restart queue process needs some work. Judging from the data I see with restarts over the last year, it does seem that sim state saves do play some factor, but if there is a considerable amount of time left before the next state save this is ignored and the region is scheduled for the queue without any early/forced save. This is probably to prevent a possible surge of regions entering the queue since state saves are not all synchronized to any global timeframe. Still, I think there could be some tolerance of time left for next state save where an early state save could occur given a query for timing from the queue scheduler, anywhere from 30 minutes to an hour. This range of time should be a enough range to populate a proper queue without worry of any surge and in-turn reduce downtime for each region as well. It just isn't happening, lol.

[Darien Caldwell]

I can only relate my own experience. A couple weeks ago, I logged into my sim on a Wednesday, got a "one minute to restart" message, and high tailed it to another sim.

So to me that indicates logging in to an ongoing count does get you included into the countdown. Why it doesn't work for some, I have no idea.

[Maestro Linden]

The main channel rolls really do start at 5AM; it just takes about an hour for the first regions to shut down.

[Imagin Illyar]

Maestro - so are the regions really "rolled back" an hour?  Are they somehow saved and then an hour later restarted?  This would account for some of the work that I've had undone in the past.