RC Release Thursday 2/18

[Caleb Linden]

Just wanted to give you a heads up that we are likely pushing out all three RCs today on Thursday 2/18 containing a minor update on the simhosts. Nothing is changing Second Life functionality wise. Please, refer to the blog post for more information.

 

Second Life RC BlueSteel:

https://wiki.secondlife.com/wiki/Release_Notes/Second_Life_RC_BlueSteel/16#16.02.17.311120

Scheduled Thursday 2016-02-18 07:00-10:30 PST

 

Second Life RC LeTigre:

LeTigre is getting the same server maintenance project as BlueSteel

https://wiki.secondlife.com/wiki/Release_Notes/Second_Life_RC_LeTigre/16#16.02.17.311120

Scheduled Thursday 2016-02-18 07:00-10:30 PST

 

Second Life RC Magnum:

Magnum is getting the same server maintenance project as BlueSteel

https://wiki.secondlife.com/wiki/Release_Notes/Second_Life_RC_Magnum/16#16.02.17.311120

Scheduled Thursday 2016-02-18 07:00-10:30 PST

 

 

We will be monitoring this thread as the code gets released, so feel free to note any observations you have about the server updates.  If you have a specific bug you'd like to report, please file a Jira.

[Ayesha Askham]

Wow

Nice timing guys, gives us plenty of time to re-arrange scheduled events and the like..not!

Last minute decision, much?

Oh and the copy/paste additions to the general notice are amateurish to say the least.

[MBeatrix]

And today it's the main channel being restarted — was suprised I couldn't login at home. Again, nice timing... NOT.

[Qie Niangao]

I doubt we'll ever learn the reason, but this sure smells like an OS- or middleware patch for some security vulnerability. If so, Support would be rightly reluctant to say more, at least before all the sim hosts are updated.

[Whirly Fizzle]

Sounds like we will get a post-mortem once all regions have rolled: http://modemworld.me/2016/02/19/sl-project-updates-16-72-unscheduled-server-deployments-sl-viewer/

Speaking at the Server Beta User Group meeting following the deployment, Steven Linden had this to say:

We had an unscheduled RC deploy earlier today. It’s for a security vulnerability that was released, and we discovered that Second Life regions were vulnerable. A full public post-mortem will be coming after we deploy to the rest of the main grid. I can’t say until it goes out to the rest of Agni; I can say that it was related to region availability only…. I honestly can’t say a great deal, other than we have a fix, and that it’s coming very soon to the rest of Agni.

[Qie Niangao]

Oh, thanks Whirly! (If I'd read Inara's blog yesterday, as usual, I wouldn't have needed to speculate. Duh!)

[MBeatrix]

What some people fail to understand is that us, normal users, aren't in SL to go read blogs about it or know any technical details, we just want it working. I don't care about post-mortem technical info I probably wouldn't understand. What I want is to be informed in advance it's going to happen, I don't need to know why — as simple as that.

[Qie Niangao]

Oh, and here I'd been trying to delight those normal users with song, dance, and witty repartee.

Now I'll just exit the stage, weeping.

[Ayesha Askham]

Oh My

So it's most likely a security exploit issue.  That makes me feel so much better...NOT. 

These "panic updates", and that is what they are, are a real game-breaker for some and they make a mockery of any rational attempt at event scheduling.

Add to that if I was like many users who never bother to read blogs or access the Grid Status page, I'd be wondering what the **** was going on and why I couldn't log on.

I know this sort of thing is important but surely LL could have given us a little "heads up" warning at a reasonable time?

By the time the "advance warning" had been posted I was tucked up in bed and many other European users would have been also.  Secondlife is a Global system so a bit of consideration for those of us NOT in the USA would have been nice.

I usually keep track of SL via my mobile but I had not done so on this occasion, (I DO have a RL) and frankly I would not have had time to warn any Europeans likely to be affected of an impending issue.

ETA at 9:15am PST: Oh, thanks for the restart (s) Linden Lab not merely one but two!

[MBeatrix]

Exactly. All that.

I try following the grid status too, despite knowing it can't be trusted. This forum was my more or less reliable info source about rolling restarts — not anymore. Would it be asking too much that the main channel restarts would have been included in this thread?

Is it too much asking for proper warning in advance from those to whose salary I (we) contribute?

[ObviousAltIsObvious]

there is one new public vulnerability that could have made a real mess of the grid with crashes and so on.

if that is what they patched, a roll on short notice was the (much much) lesser of evils.

(ah, they have it on the blog now, it was that one.)

 

 

[Whirly Fizzle]

The reason for the emergency roll: https://community.secondlife.com/t5/Tools-and-Technology/Why-the-Friday-Grid-Roll/ba-p/3006809

[MBeatrix]

Thanks, Whirly. I never doubt the Lab does what is best for the grid — it's their bread on the table —, and that's why I don't care much for technical info (again, which I probably won't understand.) I just wish they would deal with info to the users in a professional, caring way.

[EDIT] The Lab didn't have to tell us why they were going to roll regions, what they could have done was providing the info in advance. I know the timing was short, but surely the main channels restarts could have been included in this very same thread. Anyway, I'm glad that whatever the issue was is now fixed.

[Ayesha Askham]

I whole-heartedly agree with MBeatrtix, particularly the Edit.

[ChantillyTex]

I appreciate the seriousness that is given to security issues, and the speed that the fix was applied (especially given the huge numbers of servers involved).  

[WolfBaginski Bearsfoot]

Essentially, it's the timing and nature of the MC rollout announcements. We're accustomed to a much longer test-run period, and doing a roll-out on a Friday is, in the wider world of system management, rather frowned upon.

The first mention of the MC rollout was 5:47 PM PST, 18 February, which is the early hours in Europe

The announced start of the rollout was "approximately 3:00 AM PST". That's probably near enough. It fits with my experience.

The first confirmation that the roll-out was happening was, as usual, later then that, at 6:50 AM PST, 19 February.

And the completion was announced at 8:00 AM PST, 02 February!

I'm not so bothered that somebody got the date wrong, though you'd think there was some sort of script that would mark an entry as resolved and get the date correct. The bit is the square-brackets is so standard. But the time is consistent with the timestamp for the complete text.

If the notice in this thread had just said that there was a possibility that the change would be rolled out on Friday, I would feel a lot better about how SL managed to commuicate with customers. As it is, they persist in presenting an image of believeing that the world beyong the USA' s 12-mile limit is a story intended to frighten children.

 

[Ayesha Askham]

Wolf

You put your finger (um, you DO have fingers I assume?) on something that gets in the crop of most non-Americans.

The USA is such a big place that the folk in the middle and to the West really do not think much about a place that is far, far away to them, namely Yurrup.

Those that do know are vastly outweighed by those who either don't or don't care either way.  It is just the way it is. 

Most of us Yurrpians don't know much about the States, if we are honest.  The distances are mind-boggling to those of us that live in crowded countries.

[Qie Niangao]

At the risk of admitting to reading a blog and showing an interest (both evidently forbidden "normal users"), Inara Pey summarizes an Operations blog post that explains secrecy around the timing of these rolls. She also mentions two other recent technical posts I found interesting, one about a Premium billing snafu, and the other highlighting the steps in diagnosing a network problem ultimately traced to a very unusual failure mode of some ISP hardware.

I love that stuff, and hope we'll see more of it.

[Ayesha Askham]

Qie

I have to admit that while not understanding a good half of what Chris Linden wrote, it is very heartening to see such pieces.  Not only do those who DO understand it all feel more informed and involved, it make us dunces out here realise that the Linden Lab staff not only do a compllicated job, they are prepared to tell us about it too.  That makes me feel a whole lot more connected to SL. :smileyhappy:

[bigmoe Whitfield]

\o/  LL good job!   thanks for fixing that, I just rolled/updated most of mine the other night.   that's a nasty little bugger. 

 

Also guys,  I know you find it inconvincing but,  these are physical operating systems they use, that had ot be patched and restarted.  they HAVE to do that,  that means it takes down the grid when they do.   and who ever came up with the "you cant do maintaince on fridays" is out of their minds,  weekend  (fri thru sun) are the best time to do anything.   most people are actually not using resources.   "but what if it breaks" it's why we have people on call.  even in my industry.