The Lindens Explain What Happened the Other Day

[ChinRey]

5 hours ago, Parhelion Palou said:

I don't see how they'd be able to guarantee that on something like AWS unless LL could arrange a similar setup there.

There's no technical reason why they can't. The only real difference between the big cloud hosts and traditiional web hosts is that the cloud hosts have their servers spread across several server farms all over the world. It shold be easy eough for them to assign a client permanently to a specific location. But the cloud hosts are mainly competing on price, squeezing the smaller hosts out of the market by undercutting them. There are three ways they can do that. One is of course by being the biggest fishes and that's not a problem for their customers. But they also save money by locating as much as possible in low cost countries that may not have the best internet infrastrucutres in the world and by shifting the load between locations as and when there are spare capacity available. For those reasons they may not be too keen on offering a customer permanent hosting at a top location.

 

5 hours ago, Parhelion Palou said:

 LL has said they won't tell us that they're moving simulators to the cloud until after it's done. One clue might be region crossings getting a lot worse.

Hmmm... since you mention it. One new and very curious glitch I noticed in the months before last Sunday's breakdown was neighbor regions completely faiing to load. All I could see was the ground with the old default brown ground texture. Only when I crossed over into the sim did the items and the actual ground textures load. Did anybody else see the same? And haven't there been unusually many complaints about sim crossings recently? these are of course exactly the kind of problems we should expect with cloud hosted sim servers.

Edited November 6, 2019 by ChinRey

[bigmoe Whitfield]

Assets are on AWS and the CDN's.   other services are being moved, but they are not saying  what or when.  all they have given us is the asset server lives on amazons cloud now.

[Prokofy Neva]

On 11/4/2019 at 1:47 PM, Theresa Tennyson said:

From the blog post:

Now that we had it in a bad state we started to troubleshoot and figured out really fast that it wasn’t our equipment. Our stuff was (and still is) working just fine, but we were getting intermittent errors and delays on traffic that was routed through one of our providers. We quickly opened a ticket with the network provider and started engaging with them. That’s never a fun thing to do because these are times when we’re waiting on hold on the phone with a vendor while Second Life isn’t running as well as it usually does.

After several hours trying to troubleshoot with the vendor, we decided to swing a bigger hammer and adjust our Internet routing. It took a few attempts, but we finally got it, and we were able to route around the problematic network. We’re still trying to troubleshoot with the vendor, but Second Life is back to normal again.

Note that they identified the "vendor" as a "network provider" before they started calling it a "vendor." 

No, and no again. You're making this assumption as someone in the tech field. I realize you want to hammer away at this to try to prove poor reading comprehension, bad faith, or whatever, but when someone who is reasonably educated reads some jargony thing like this, it isn't always clear, and needs to be, and it is MORE THAN FINE to call that out.

First, she said "traffic that was routed through one of our providers". This isn't clear whether it is a telephone service or a server farm or what.

Next, she says she opened a ticket with "the network provider". But wait. She had just said "routed through one of our providers" but didn't qualify it as a NETWORK provider; when she next talks about NETWORK provider you can't be sure if she is talking about something ELSE.

After that, she talks in a generic sort of way "these are times when we're waiting on hold on the phone with a vendor" -- note there is again an "a" or indefinite article and not a definite article like "the". Do they still teach the parts of speech in school and do techies learn them? I don't think so.

She then speaks yet later about being able to "route around the problematic network". But it's still not clear if the thing always in question and broken was "the network" or "a vendor" that reaches the network, or what it is, i.e. a telephone service reaching a network of computers, separate from the telephone service.

So, for example, if I wrote:

"We opened a ticket with the maintenance office and started engaging with them. That's never a fun things to do...when we're waiting on hold on the phone with a repair man...

After several hours trying to troubleshoot with the plumber..."

In each case, we are discussing levels of things -- the overarching concept of "the maintenance office"; then "a handy man"; then "the plumber" -- perhaps dealing with three different people.

 

[Prokofy Neva]

On 11/4/2019 at 1:07 PM, Bradford Mint said:

I haven't said you wouldn't get to know (you or anyone else).  What I said was that your service is with LL, not the cloud provider.  LL cannot provide an SLA greater than that of their provider (well they could but would be dumb to try) and their SLA is absolutely zilch so they're hot to trot!   Cloud provision isn't vital to the functioning of any network, only to those who choose to consume the compute, service and storage resources that may be offered over any network.  This seems moot at this time however.  I'm not sure that there's an assumption that things get better with moving to cloud.  What does happen is that a local datacentre can be downsized or removed, no need for specialist staff to host "whatever service", the replacement services are consumed as a service and contractually against an SLA for a specific contract term and crucially for some, moves the cost model from Capex to Opex which has accounting implications as well as usually definable benefits in terms of service provision and upgrade. Those are some of the key points of outsourcing whether it's people or service, i.e. cloud.  Cloud in this context is just synonymous with service.  Nothing magical about cloud to see here.

Phone them up and ask, i'm sure they'll answer.  However, as i've already pointed out, how they deliver the service isn't really that relevant to the end user, random player or business owner.  The service as provided by LL remains the same with the same SLA and TOS. *shrugs*

All of this is just specious edge-casing, literalizing, and assorted other BS to obfuscate the basic lack of full communication from the Lindens on their situation and plans.

If Verizon suddenly stops working -- not an infrequent occurrence -- and I see from my cell phone working on the wireless of another company that a client is contacting me urgently because I'm missing a deadline, I don't say "Your service agreement is with me, not Verizon, but there's nothing I can do about your job when Verizon is down." 

What I have to do in those situations is run over to FedEx which hopefully isn't on Verizon or is in a part of town not affected (during Hurricane Sandy, everything below 49th was dark; everything above 49th was working due to Con Ed's lines or something) and get that job done. Which is what I had to do during Hurricane Sandy for three weeks if I didn't want to lose my clients. I could have dispensed with electricity entirely and typed it on an old-fashioned typewriter; I could have run up to the FedEx, put the text on a USB, and mailed it to any USPS station still working, etc.

Which, to continue the analogy, is what Linden did. They routed around, they did some things, some other things, and got it working.

If I shrugged and told my client, well, Verizon, or well, Hurricane Sandy, let me snail-mail it to you he would not hire me again.

Perhaps you don't have to think about things like getting jobs done in RL so this is all highly theoretical to you, and you can keep hectoring me about my supposed excessive expectations about LL.

All of this nattering on about Capex and Opex is completely beside the point, as is silly efforts to "puncture" myths about alleged "magic" in clouds; ditto the demand to "call them".

There's another thing that can happen: they can write business communications clearly for the uninitiated, and explain about the whole cloud thing in general. The end.

17 hours ago, Scylla Rhiadra said:

LOL

Only in America would Obama be called a "socialist."

No, only in New York City in 1986, at the Democratic Socialist of America (DSA) annual conference which I attended, and which Obama attended as well, as he did regularly.

 

Edited November 6, 2019 by Prokofy Neva

[Pixieplumb Flanagan]

Bit rude, bit supercilious, bit of an undeserved superiority complex showing <sighs> still, the main thing is the Lindens will totally read all this, will totally change how they do stuff (doing stuff is, I imagine, pretty much their entire job description) and then everything will be peachy keen.  Or not.  Probably not.  How long ago was this outage anyway? 

[Prokofy Neva]

10 hours ago, Parhelion Palou said:

The Head of Hardware (I forgot the title) who spoke in a video a couple years ago said LL's data center is becoming obsolete. They decided it would be cheaper to put SL on the cloud completely than to build a new data center. I'm sure they thought it shouldn't be that hard; they'd done it with Sansar and could use the people who did that work. I would've loved to see the look of horror on the faces of those people when they saw the SL code base.

The data center was set up to get the fastest possible data transfer between the servers in order to reduce region crossing problems. I don't see how they'd be able to guarantee that on something like AWS unless LL could arrange a similar setup there. LL has said they won't tell us that they're moving simulators to the cloud until after it's done. One clue might be region crossings getting a lot worse.

Bingo.

Thank you.

This is exactly what I remembered, and why I posed the question again.

And what you said about Sansar.

And sim crossings.

I don't get this "we can't tell you until it's finished" stuff.

What, they imagine Russian GRU commandos will snipe at the crevices in their sim crossings WHILE they are going to the cloud if any of us has "loose lips" that "sink ships"? 

???

I don't know if you can assume that the sim seams thing is what gives out first when "moving to the cloud". It might be teleports. Or chat. Or saving stuff. Or all of the above.

 

[Prokofy Neva]

10 hours ago, ChinRey said:

There's no technical reason why they can't. The only real difference between the big cloud hosts and traditiional web hosts is that the cloud hosts have their servers spread across several server farms all over the world. It shold be easy eough for them to assign a client permanently to a specific location. But the cloud hosts are mainly competing on price, squeezing the smaller hosts out of the market by undercutting them. There are three ways they can do that. One is of course by being the biggest fishes and that's not a problem for their customers. But they also save money by locating as much as possible in low cost countries that may not have the best internet infrastrucutres in the world and by shifting the load between locations as and when there are spare capacity available. For those reasons they may not be too keen on offering a customer permanent hosting at a top location.

 

Hmmm... since you mention it. One new and very curious glitch I noticed in the months before last Sunday's breakdown was neighbor regions completely faiing to load. All I could see was the ground with the old default brown ground texture. Only when I crossed over into the sim did the items and the actual ground textures load. Did anybody else see the same? And haven't there been unusually many complaints about sim crossings recently? these are of course exactly the kind of problems we should expect with cloud hosted sim servers.

Rey, that "not being able to see the next sim over" happens all the time, has always happen, and happened even back 15 years ago when SL wasn't run on the Internet, but was run with electric wires and tin cans.

So that's not a thing.

Also, describe the measure of "big fish".

I'm thinking that 28,000 simulators (are they 4 sims to a server) isn't a big fish? Or?

Edited November 6, 2019 by Prokofy Neva

[Theresa Tennyson]

7 hours ago, Prokofy Neva said:

So, for example, if I wrote:

"We opened a ticket with the maintenance office and started engaging with them. That's never a fun things to do...when we're waiting on hold on the phone with a repair man...

After several hours trying to troubleshoot with the plumber..."

In each case, we are discussing levels of things -- the overarching concept of "the maintenance office"; then "a handy man"; then "the plumber" -- perhaps dealing with three different people.

 

But at least *I* would assume you were talking about one discrete incident. I wouldn't think that you called the maintenance office for a broken window, then waited on hold for an electrician, and then spent hours troubleshooting with the plumber while your problem with the window and the electrician that you mentioned seconds ago weren't mentioned again. (By the way, I'm not in tech, unless you count sewing machines and box trucks as "tech.")

Of course, I don't know if I should respond to your post at all, because you responded to a post by "Theresa Tennyson" but you referred to the person who you were addressing as "you", who, by your logic, may be someone completely different.

Edited November 6, 2019 by Theresa Tennyson

[Cindy Evanier]

21 minutes ago, Pixieplumb Flanagan said:

How long ago was this outage anyway? 

About 8 hours for some people.  I slept through a lot of it.  The rest of the time I hung out on our opensim region 

[ChinRey]

25 minutes ago, Prokofy Neva said:

Rey, that "not being able to see the next sim over" happens all the time, has always happen, and happened even back 15 years ago when SL wasn't run on the Internet, but was run with electric wires and tin cans.

Oh. I've never seen that until the last month or two.

 

30 minutes ago, Prokofy Neva said:

I'm thinking that 28,000 simulators (are they 4 sims to a server) isn't a big fish? Or?

Medium sized perhaps. 😉

But Bigmoe reminded me that AWS does offer fixed location hosting too. Both Akamai and Umbra - the two CDN providers LL uses for SL and Sansar - are hosted on AWS servers. It wouldn't be cloud hosting though, just traditional web hosting by a compeny that also happens to offer cloud. I doubt there's much money to save that way.

[Dakota Linden]

Greetings all!

Off topic posts have been removed. 

Since posts after the off topic thread hijacking attempt have remained on topic, we have opted to leave this thread open, for now.  However, if the comments and posts go off topic again, we will have no choice but to lock the thread. 

 

[Prokofy Neva]

On 11/4/2019 at 6:03 PM, Wulfie Reanimator said:

imbecilic*

What's ridiculous is the tone and the apparent amount of vitriol you have towards people whose primary opinion seems to be "I don't care, why do you?" which is about as neutral as you can get.

This is from the blog post you linked:

As far as I know, "network providers" don't provide pizza, and SL doesn't run on a cell service. That doesn't seem to satisfy your question because I know you're still very paranoid about "the cloud" as you have little understanding of it, which is fine as you are a non-technical person. But because you are a non-technical person, giving you the full answer might not satisfy you either, because you're hung on the scary "cloud" term.

Lots of different things are "the cloud." There is no one "cloud network" type, the term is ambiguous and useless without proper context. Not everything has to be on "the cloud(s)" for SL to work.

• If you can log into any computer at your workplace/school (especially in separate buildings) and instantly get all of your files on the desktop, that's "the cloud."
• If you remember SL Go, that was "the cloud." (Same goes for OnLive or any game-streaming service. Cloud gaming.)
• If your company has ANY software that runs on a remote server, not on the same computer, that is "the cloud."
  • This means that even if/when LL had its own server farms, they were "the cloud."
• Linden Lab has been using AWS for a good while now, that is "the cloud." (<-- This is the answer you want, even if unrelated to the actual problem.)
  • A presentation of this move was done back in 2017.
  • They've also made blog posts related to AWS even before that, but that post is a different "cloud."

• "The cloud" is just "a remote server that does something for you." The internet as a whole is "a cloud."

Let me simplify this for you.

The cloud is other people's computers.

I am not "scared" of this cloud thing, any more than I am "scared" of the Internet. I'm just utterly sanguine about it and don't have any unicorn fantasy realm notions about it, as so many techies in fact do, nor do I think it is so complicated that an ordinary person can't understand it -- car mechanics use the same kind of obfuscation on drivers, but you know, drivers change their own oil now and thanks to the Internet, they research and comparison shop way more and aren't as victimized as car mechanics.

Coders and programmers and "engineers" who don't even wield wrenches are merely car mechanics, so I don't genuflect.

One of my jobs relies on AWS and in fact moved to AWS to deal with constant Russian DDoSing. I've seen other jobs "move to the cloud" and other systems I'm familiar with, and this has not gone smoothly "like a pianola," although AWS appears to be the most robust.

If the Lindens gave a presentation in 2017, that's not 2019 and an update, especially if a "different" cloud.

It's just very, very basic. Did LL move everything to the cloud? Some things? Is that why we have problems? Will it get better? You know, basic stuff which -- as I will point out again, the Lindens seem far, far more willing to discuss, and even put up blog posts about than their avid fan base.

Sometimes it's better when the business side does the communications because they will make it more clear and less technical. But as most of us who have been in SL for 15 years can tell you, we have PhDs in physics now that we never knew we would have to earn. So truly, you can have the techs tell us in full detail and we may actually be able to follow along, imagine that. And the first rule of following tech talk is this: techies disagree among themselves about a lot of things, and they can be proven wrong not only by their peers but by society at large (see Facebook). They aren't a priesthood, and it's ok to challenge them. 

I became familiar with this set of issues when I first had to get my Masters in Physics merely to play The Sims -- and that was offline.

OK, I jest, but I once took my computer to be serviced because various hostile governments had savaged it. And this specialized techie who deals with those kinds of situations was going over everything with me, and he told me to ditch Skype, which was very full of security holes, and told me to ditch Second Life -- of course the cybersecurity professionals will want you to have a computer not routinely accessed remotely by things that you don't fully control. But I told him that Skype was the only way to call Afghanistan, and there was no way I was ditching Second Life.

He asked me to show him the program. So I showed him how it worked, and showed him my different sims, and showed him the customer service tickets, if you will, i.e. IMs from tenants. I showed him how on one homestead, there were objects on physics colliding and this lagged the sim. I showed him how you could find those with beams and determine whether to remove them from the sim. Those menus also have the high script times (wish we had that for the Mainland although you can get a reasonable facsimile with various commercial products or free products on the MP). Etc.

I showed him bunches of stuff -- and he he just goggled at me. "I can't believe you learned how to do all this, and you only have a degree in languages and literature". Well, "only" -- I'm a Russian translator by professor. Unless you're a native speaker, you don't know third person past imperfect and perfect participles and how to use them, nor would I expect you to. But you know, rote learning can take you very far.

 

 

[SpiritSparrow Skydancer]

On 11/4/2019 at 2:16 PM, Drake1 Nightfire said:

I say we go back to the grid beng down most of the day once a week for maintenance..  lets say... Wednesdays till 4pm SLT? Does that work for everyone? 

If you have this much of a freak out over a partial grid black out for half a day, you need to find a hobby.. What if there was a minor earthquake and it knocked out the grid, would you still flip out? How about wildfires taking out the power and internet lines. Or some drunk slams into a hub and blows the power for a large chunk of the grid. Seriously, it was a few hours for some people. I was online the whole time, not a single hiccup. If the grid is down, find something else to do.  

no, no that does not work for me... hm how about Tuesdays?

[SpiritSparrow Skydancer]

10 hours ago, Prokofy Neva said:

Let me simplify this for you.

The cloud is other people's computers.

I am not "scared" of this cloud thing, any more than I am "scared" of the Internet. I'm just utterly sanguine about it and don't have any unicorn fantasy realm notions about it, as so many techies in fact do, nor do I think it is so complicated that an ordinary person can't understand it -- car mechanics use the same kind of obfuscation on drivers, but you know, drivers change their own oil now and thanks to the Internet, they research and comparison shop way more and aren't as victimized as car mechanics.

Coders and programmers and "engineers" who don't even wield wrenches are merely car mechanics, so I don't genuflect.

One of my jobs relies on AWS and in fact moved to AWS to deal with constant Russian DDoSing. I've seen other jobs "move to the cloud" and other systems I'm familiar with, and this has not gone smoothly "like a pianola," although AWS appears to be the most robust.

If the Lindens gave a presentation in 2017, that's not 2019 and an update, especially if a "different" cloud.

It's just very, very basic. Did LL move everything to the cloud? Some things? Is that why we have problems? Will it get better? You know, basic stuff which -- as I will point out again, the Lindens seem far, far more willing to discuss, and even put up blog posts about than their avid fan base.

Sometimes it's better when the business side does the communications because they will make it more clear and less technical. But as most of us who have been in SL for 15 years can tell you, we have PhDs in physics now that we never knew we would have to earn. So truly, you can have the techs tell us in full detail and we may actually be able to follow along, imagine that. And the first rule of following tech talk is this: techies disagree among themselves about a lot of things, and they can be proven wrong not only by their peers but by society at large (see Facebook). They aren't a priesthood, and it's ok to challenge them. 

I became familiar with this set of issues when I first had to get my Masters in Physics merely to play The Sims -- and that was offline.

OK, I jest, but I once took my computer to be serviced because various hostile governments had savaged it. And this specialized techie who deals with those kinds of situations was going over everything with me, and he told me to ditch Skype, which was very full of security holes, and told me to ditch Second Life -- of course the cybersecurity professionals will want you to have a computer not routinely accessed remotely by things that you don't fully control. But I told him that Skype was the only way to call Afghanistan, and there was no way I was ditching Second Life.

He asked me to show him the program. So I showed him how it worked, and showed him my different sims, and showed him the customer service tickets, if you will, i.e. IMs from tenants. I showed him how on one homestead, there were objects on physics colliding and this lagged the sim. I showed him how you could find those with beams and determine whether to remove them from the sim. Those menus also have the high script times (wish we had that for the Mainland although you can get a reasonable facsimile with various commercial products or free products on the MP). Etc.

I showed him bunches of stuff -- and he he just goggled at me. "I can't believe you learned how to do all this, and you only have a degree in languages and literature". Well, "only" -- I'm a Russian translator by professor. Unless you're a native speaker, you don't know third person past imperfect and perfect participles and how to use them, nor would I expect you to. But you know, rote learning can take you very far.

 

 

 You call that simple?

 

[Pixieplumb Flanagan]

11 hours ago, Prokofy Neva said:

 

OK, I jest, but I once took my computer to be serviced because various hostile governments had savaged it.

 

 

well ok, and how did that make you feel?

[Alwin Alcott]

i'm a bit surprised nobody got roasted at a certain blog yet.. or did i overlook and miss it?

[Beth Macbain]

2 hours ago, Alwin Alcott said:

i'm a bit surprised nobody got roasted at a certain blog yet.. or did i overlook and miss it?

I clearly haven't been offensive enough. 😝

[Alyona Su]

I can help if you like. Being offensive and not getting suspended is a fine-line and takes lots and lots of practice. I have it almost down to a science. Because snowflakes are as snowflakes do. Bahaha!