The Lindens Explain What Happened the Other Day

[Prokofy Neva]

I think it's important to point out (because the post only has about 1,000 views) that the Lindens did explain what happened the other day when the grid was down -- by my count -- for eight hours.

And I think it's important to note that they did provide this explanation, and that the outage wasn't due to "my land" or "your computer" and not even "their software" but "their vendors".

I find that puzzling, as just a norm not versed in Internet network technology, but someone can fill in the blanks for me.

Their server farm had problems? They have several of them in different states and countries.

Their phone service had problems? Do they have Comcast? Techies in California always complain bitterly about Comcast, which I find exaggerated, but then I have Verizon, which is complained about as much or more. The Internet people hate telecoms, and here we all are.

A "vendor" is a person who sells a service. So it was either servers, or telephone or...hmm...cloud providers? But I still haven't gotten a very clear read-out as to whether SL in fact "moved to the cloud" which everyone thinks is a wonderful thing, but as I always point out in both work meetings and public meetings on this issue: the cloud is other people's computers.

So, rely on them if you like -- it's cheaper -- but you know, back up your files. And keep those cell phones and flashlights at the ready.

This outage was NOT caused by the forest fire.

So heroically, if I understood this message correctly (and someone can translate it into norm-speak for me), the Lindens didn't get a good and robust response from their vendor, so they "routed around" which is what "the Internet does". They changed numbers on a config window or took out wrenches and unscrewed plates with bolts in them or did whatever they did.

But this thing -- this "re-routing" seems to me "still not to work so great".

SL, on many sims, is still having these problems:

o can't delete objects

o can't rez from inventory

o can't TP into the sim

o KittyCats can't be born and can't eat properly!

It's this last thing that I hope will really reach and touch the hearts of Lindens, the thought of all those hungry kitties, and kitties waiting to be born who can't be! Sob!

Hey, let me tell you, not being able to delete objects becomes a real problem when you have features on your installations that involve temp-rez prims.

Yes, asking the Lindens to re-set individual sims helps but you know, they can't do that with all 28,000 of them.

 

[bigmoe Whitfield]

the one thing I can say is they did announce years back everything was being moved to Arizona, so they only have the one data center,  some of their services now live in aws and they use a cdn now too, so somethings are spread out.

[Bree Giffen]

From what little I am aware of, the problem was because LL bought "SL" from a vendor and it came in a box. LL knows how to go to a sandbox, open up the box and take out "SL". It turns out that whatever was in the box was broken. The programmer of "SL" says that when they put it in the box "SL" was ok. So the problem is with the vendor! Thanks for explaining that LL. 😘

[Erwin Solo]

Blame it on the cloud. 

 

[Prokofy Neva]

1 hour ago, bigmoe Whitfield said:

the one thing I can say is they did announce years back everything was being moved to Arizona, so they only have the one data center,  some of their services now live in aws and they use a cdn now too, so somethings are spread out.

I didn't realize they moved "everything" to Arizona. I thought some were in Texas, some in the Netherlands, or whatever.

I did a geolocation on some of their agni servers and yes, it seems they all came up "Arizona". But I don't really have a representative sample.

I could see where they would not want to reveal this information casually so that competitors, disgruntled ex-employees, or angry customers couldn't try to DDoS them.

I did tracert and it was odd to see that agni is reached from my provider in one hop. Can that be? I seem to recall years ago when we were always researching this to try to find out why some servers lagged so badly that there's be dozens of hops.

 

[animats]

This looks like it was a fairly routine network outage at a data center. Big data centers are usually tied to several of the backbone Internet providers, and can switch if they have to, but apparently SL does not do this automatically. Routing changes can take a while to propagate as all the players come into agreement on the new routing. For a web site, that just means a few retries and a slow page load. LL has to be more cautious than web site operators -  30 seconds of connectivity loss and everyone is kicked out of the world.

4 hours ago, Bree Giffen said:

From what little I am aware of, the problem was because LL bought "SL" from a vendor and it came in a box.

No, Linden Lab people coded and maintain Second Life. If you go to Server User Group, Viewer User Group, or Creator User Group, you can talk to the people who maintain it.

[Kimmi Zehetbauer]

So the servers didn't come from Wal-mart!

[Bradford Mint]

What's the big issue? There was an outage, move on.

Sometimes the reason is technically tiny but the impact is massive, for example the fairly recent global outage of the entire O2 mobile phone service. Due to an expired certificate!

SL isn't part of a nations critical national infrastructure.

Something happened, they did stuff, resolved it. *Shrugs*

[Jackson Redstar]

it was the Russians!!

[Alyona Su]

13 hours ago, Prokofy Neva said:

Their server farm had problems? They have several of them in different states and countries.

They explained that it wasn't their system at all. It was their "telephone provider's" fault. Prok, sometimes you simply tend to over-analyze things. It's okay to plonk me for saying that as you usually do, I still respect you and your purview on things. Even when they can seem outlandish.

Edited November 2, 2019 by Alyona Su

[ItHadToComeToThis]

6 hours ago, Bradford Mint said:

What's the big issue? There was an outage, move on.

Sometimes the reason is technically tiny but the impact is massive, for example the fairly recent global outage of the entire O2 mobile phone service. Due to an expired certificate!

SL isn't part of a nations critical national infrastructure.

Something happened, they did stuff, resolved it. *Shrugs*

You are forgetting that we are dealing with the die hard players in this forum. 10 seconds of outage is scream, shouts, rages and a crater where their RL house used to be. Let alone the world ending war if the outage happens to be hours. My best advice is to just step back, slowly, don’t startle them and come over and join me in the “meh...it happened” corner. We have Mountain Dew and Cheetos.

Edited November 2, 2019 by ItHadToComeToThis

[Profaitchikenz Haiku]

I vote we set up a "Preppers" version of SL using packet radio and Radegast. Then it will be immune to sudden loss of DNS.

[Drake1 Nightfire]

4 hours ago, ItHadToComeToThis said:

You are forgetting that we are dealing with the die hard players in this forum. 10 seconds of outage is scream, shouts, rages and a crater where their RL house used to be. Let alone the world ending war if the outage happens to be hours. My best advice is to just step back, slowly, don’t startle them and come over and join me in the “meh...it happened” corner. We have Mountain Dew and Cheetos.

Voltage or Code Red? And i prefer Cheese balls tyvm.. 

[Pixie Kobichenko]

Having worked in customer service on and off for twenty years- all I can tell anyone who is worried about why the grid goes down, is there really isn't an answer that will appease the masses.  Customers will become irate and do not behave with a shred of patience when they feel they are inconvenienced, regardless of the reason. I had people call in and threaten to hurt my techs if they didn't work faster on an issue.  Other times, the problem would be out of my company's hands entirely, as they actually piggybacked their services from a larger provider (basically rented the lines).  But that wasn't an acceptable explanation.  I can only equate it to tin-foil hatters who already have a reason why in their head and just get some sort of enjoyment from pontificating "why" there is an issue. No matter what they're told.   

[Lindal Kidd]

2 hours ago, Profaitchikenz Haiku said:

I vote we set up a "Preppers" version of SL using packet radio and Radegast. Then it will be immune to sudden loss of DNS.

That's laaaaame!  REAL preppers would do it with letters.  Written longhand.  In pencil.  Delivered by itinerant tinkers.

[Tari Landar]

8 minutes ago, Lindal Kidd said:

That's laaaaame!  REAL preppers would do it with letters.  Written longhand.  In pencil.  Delivered by itinerant tinkers.

You forgot it has to be in code, on special paper, that only a special detector can detect and a special cipher can decipher. The detector and cipher are in pieces, in different locations, spread all around where even if someone sees it, they won't know what it is. Both can only be put together one way, you can't get the instructions for them without knowing the code when you knock on the special door and ask for Phrank. If Phrank is dead, his brother Phred will know the code. You have 48 hours to find all the pieces, get the code, put the detector together to display the code, put the cipher together to sort the letter and send a response. If you don't, it'll be assumed you're dead, and the next will take your place.

At the end of the journey, it's likely the message simply says "hi"

[Beth Macbain]

It's nearly the season anyway so... 

Beware of the decoder rings. 

 

[Tari Landar]

55 minutes ago, Beth Macbain said:

It's nearly the season anyway so... 

Beware of the decoder rings. 

 

My favorite Christmas movie  (coming in at a close second would be Elf, lol).  I watched it I don't know how many times as a lil tot. But I also watched it the first time it ever aired on TnT for its first "24 hours of a Christmas Story" in 1997. As in, we turned it on, Christmas eve, on the tv, and kept it playing all night, even while we slept (yes I know, bad us, bad, bad us) and also kept it playing all day Christmas day. That became a family tradition, and it carries on in my family, even when it moved to TBS.  I taped it, yes, taped, using a vcr, on the tv one year (2000) commercials and all until I ran out of tape. We still have that tape, and still watch it, even the commercials (which a lot are for basketball games, lmao). We don't have cable and haven't for a very long time, so until we got the dvd version, we played the tape, over and over and over for the full 24 hours  Now I play the dvd version for the full 24 hours While we're gone on Christmas day visiting family, the movie is playing for the furry members of our household. 

Yes I AM that person.

I've also been to the actual house dozens and dozens of times.

[Profaitchikenz Haiku]

2 hours ago, Lindal Kidd said:

That's laaaaame!  REAL preppers would do it with letters.  Written longhand.  In pencil.  Delivered by itinerant tinkers.

I beg to differ, they'd wrap the letter round a crossbow bolt and fire as far towards the target as they could.. That station pulls the bolt out of what it hit, checks the address, and fires it as far towards the target as they can...

[Garnet Psaltery]

I'd forgotten all about the outage until I read this.

[Tari Landar]

I like watching people freak out over outages and down time. It's one of my guilty pleasures on the forums. I laugh way too much at watching people seemingly go insane. There have been times int he past when it was down for days on end, as in, actually down for ALL, not just some, but all (this last outage didn't actually take the entire grid down at all, it took most of it down, but not all of it, there were still dozens of thousands online the whole time, that doesn't constitute an actual outage, but I digress). People freak out more and more as each outage or "downtime" happens. I can't help but find it hilarious (more so than it probably should be). Back in the day, it was annoying and we all still talked or complained about it now and then, but the freak out factor was way less. Now, someone gets logged out (for whatever reason) and the sky is falling -half the time it isn't even really an outage, lol. 

Usually I post this to threads when people start freaking out over any kind of outage...

We're all headed for a disaster of biblical proportions. Fire and brimstone coming down from the skies! Rivers and seas boiling! Forty years of darkness! Earthquakes, volcanoes...The dead rising from the grave! Human sacrifice, dogs and cats living together... mass hysteria!

It annoys people, and that makes me laugh even more. I really like laughing, it's my favorite

[Profaitchikenz Haiku]

13 minutes ago, Tari Landar said:

We're all headed for a disaster of biblical proportions. Fire and brimstone coming down from the skies! Rivers and seas boiling! Forty years of darkness! Earthquakes, volcanoes...The dead rising from the grave! Human sacrifice, dogs and cats living together... mass hysteria!

flashback to Rowan Atkinson "Will this wind be so mighty as to lay low the fundaments of the uuuuuuuurth?"

[Pixie Kobichenko]

Just now, Profaitchikenz Haiku said:

flashback to Rowan Atkinson "Will this wind be so mighty as to lay low the fundaments of the uuuuuuuurth?"

No that was Bill Murray in Ghostbusters

[Tari Landar]

16 minutes ago, Pixie Kobichenko said:

No that was Bill Murray in Ghostbusters

Yes , but it wasn't just him, it was the other ghostbusters too

We like those movies.

[SpiritSparrow Skydancer]

On 11/1/2019 at 6:49 PM, Prokofy Neva said:

I think it's important to point out (because the post only has about 1,000 views) that the Lindens did explain what happened the other day when the grid was down -- by my count -- for eight hours.

And I think it's important to note that they did provide this explanation, and that the outage wasn't due to "my land" or "your computer" and not even "their software" but "their vendors".

I find that puzzling, as just a norm not versed in Internet network technology, but someone can fill in the blanks for me.

Their server farm had problems? They have several of them in different states and countries.

Their phone service had problems? Do they have Comcast? Techies in California always complain bitterly about Comcast, which I find exaggerated, but then I have Verizon, which is complained about as much or more. The Internet people hate telecoms, and here we all are.

A "vendor" is a person who sells a service. So it was either servers, or telephone or...hmm...cloud providers? But I still haven't gotten a very clear read-out as to whether SL in fact "moved to the cloud" which everyone thinks is a wonderful thing, but as I always point out in both work meetings and public meetings on this issue: the cloud is other people's computers.

So, rely on them if you like -- it's cheaper -- but you know, back up your files. And keep those cell phones and flashlights at the ready.

This outage was NOT caused by the forest fire.

So heroically, if I understood this message correctly (and someone can translate it into norm-speak for me), the Lindens didn't get a good and robust response from their vendor, so they "routed around" which is what "the Internet does". They changed numbers on a config window or took out wrenches and unscrewed plates with bolts in them or did whatever they did.

But this thing -- this "re-routing" seems to me "still not to work so great".

SL, on many sims, is still having these problems:

o can't delete objects

o can't rez from inventory

o can't TP into the sim

o KittyCats can't be born and can't eat properly!

It's this last thing that I hope will really reach and touch the hearts of Lindens, the thought of all those hungry kitties, and kitties waiting to be born who can't be! Sob!

Hey, let me tell you, not being able to delete objects becomes a real problem when you have features on your installations that involve temp-rez prims.

Yes, asking the Lindens to re-set individual sims helps but you know, they can't do that with all 28,000 of them.

 

Lindens had nothing to do with the Kittycats issue, that was a server on their side.  just sayin'