The Lindens Explain What Happened the Other Day

[Prokofy Neva]

26 minutes ago, Bradford Mint said:

Just to pick up on this piece, why do you think you need to know?

Your service is provided by LL, the cloud provider (if relevant) provides service to LL.  Do you know what material the pipes are made of that transport your domestic water?  Which route they take? What redundancy they have in place?  What their operational plans are for a targeted terrorist attack?  My guess is no.  You pay your water company against an SLA and they in turn deliver you water, in accordance with that SLA (or not).

You're over thinking it and yes, you can complain, you can ask whatever questions but that doesn't mean they'll listen or respond.

The better question to ask is: why do you think I wouldn't get to know? I'm a user who runs a business on this platform. Hello? Cloud provision is vital to the functioning of any network but particularly a virtual world. There is an assumption that things get better if a company "moves to the cloud"; they often get worse. 

It's completely specious to raise issues like water transport or whether they have go-bags in the closet. That's good if they do. If Con Ed, whose manholes are always exploding around my house, wasn't in the cloud, and said they were moving to it, I'd expect hitches and concerns. Newspapers, assembly people would discuss it. Anything that affected their service would not only be a legitimate topic of debate; it would be a necessary one.

But when it comes to Linden Lab, the fanboyz get all shirty and protective and imagine that none of us can ask any questions about it. Ridiculous.

The Lindens have talked about "moving to the cloud" and I assumed it was done already; then I found out it wasn't. So I'd like an update on that. Perhaps Oz Linden could do it. Fortunately the Lindens are more amenable to public questions and being accountable than their fan base.

[Prokofy Neva]

23 minutes ago, Ardy Lay said:

Internet access provider is a vendor.  I see my packets take a different, shorter, less lossy route now.  Saying in public which "vendor" was causing the problem is bad business practice and is likely not permitted by Linden Lab's own rules for how we and they use this forum.

The request wasn't that they name a specific company -- their own rules for the forums prohibit naming specific companies with criticism.

The request was to ask vendor of WHAT KIND of service. Internet? Servers? Telephone? Pizza delivery? And once again, this is about WHETHER OR NOT SL moved to the cloud. Not which cloud provider.

[Ardy Lay]

The outage that I witnessed was between me and the simulator hosts.  I am GUESSING the vendor was one of the Internet access providers for the simulator hosts.  And no, I don't really know where the simulator hosts are located.

[Prokofy Neva]

On 11/3/2019 at 1:34 AM, Tarina Sewell said:

Lindens had nothing to do with the Kittycats issue, that was a server on their side.  just sayin'

I thought as much, but it's also true that if the Linden servers aren't working, scripts can't execute, etc. and it could affect them. 

Also, there's this as a possible factor.

[Bradford Mint]

4 minutes ago, Prokofy Neva said:

The better question to ask is: why do you think I wouldn't get to know? I'm a user who runs a business on this platform. Hello? Cloud provision is vital to the functioning of any network but particularly a virtual world. There is an assumption that things get better if a company "moves to the cloud"; they often get worse. 

I haven't said you wouldn't get to know (you or anyone else).  What I said was that your service is with LL, not the cloud provider.  LL cannot provide an SLA greater than that of their provider (well they could but would be dumb to try) and their SLA is absolutely zilch so they're hot to trot!   Cloud provision isn't vital to the functioning of any network, only to those who choose to consume the compute, service and storage resources that may be offered over any network.  This seems moot at this time however.  I'm not sure that there's an assumption that things get better with moving to cloud.  What does happen is that a local datacentre can be downsized or removed, no need for specialist staff to host "whatever service", the replacement services are consumed as a service and contractually against an SLA for a specific contract term and crucially for some, moves the cost model from Capex to Opex which has accounting implications as well as usually definable benefits in terms of service provision and upgrade. Those are some of the key points of outsourcing whether it's people or service, i.e. cloud.  Cloud in this context is just synonymous with service.  Nothing magical about cloud to see here.

4 minutes ago, Prokofy Neva said:

The Lindens have talked about "moving to the cloud" and I assumed it was done already; then I found out it wasn't. So I'd like an update on that. Perhaps Oz Linden could do it. Fortunately the Lindens are more amenable to public questions and being accountable than their fan base.

Phone them up and ask, i'm sure they'll answer.  However, as i've already pointed out, how they deliver the service isn't really that relevant to the end user, random player or business owner.  The service as provided by LL remains the same with the same SLA and TOS. *shrugs*

[Theresa Tennyson]

45 minutes ago, Prokofy Neva said:

The request wasn't that they name a specific company -- their own rules for the forums prohibit naming specific companies with criticism.

The request was to ask vendor of WHAT KIND of service. Internet? Servers? Telephone? Pizza delivery? And once again, this is about WHETHER OR NOT SL moved to the cloud. Not which cloud provider.

From the blog post:

Now that we had it in a bad state we started to troubleshoot and figured out really fast that it wasn’t our equipment. Our stuff was (and still is) working just fine, but we were getting intermittent errors and delays on traffic that was routed through one of our providers. We quickly opened a ticket with the network provider and started engaging with them. That’s never a fun thing to do because these are times when we’re waiting on hold on the phone with a vendor while Second Life isn’t running as well as it usually does.

After several hours trying to troubleshoot with the vendor, we decided to swing a bigger hammer and adjust our Internet routing. It took a few attempts, but we finally got it, and we were able to route around the problematic network. We’re still trying to troubleshoot with the vendor, but Second Life is back to normal again.

Note that they identified the "vendor" as a "network provider" before they started calling it a "vendor." 

[Drake1 Nightfire]

I say we go back to the grid beng down most of the day once a week for maintenance..  lets say... Wednesdays till 4pm SLT? Does that work for everyone? 

If you have this much of a freak out over a partial grid black out for half a day, you need to find a hobby.. What if there was a minor earthquake and it knocked out the grid, would you still flip out? How about wildfires taking out the power and internet lines. Or some drunk slams into a hub and blows the power for a large chunk of the grid. Seriously, it was a few hours for some people. I was online the whole time, not a single hiccup. If the grid is down, find something else to do.  

[Lancewae Barrowstone]

Theresa posted the LL language:  "We quickly opened a ticket with the network provider and started engaging with them. That’s never a fun thing to do because these are times when we’re waiting on hold on the phone with a vendor while Second Life isn’t running as well as it usually does."

Thank you, that is clear.  However I can see how it is curious an ISP is referred to as "provider in one breath and "vendor" thereafter.  It only suggests to me that the communication came from the business side rather than the technical.  In contracts and accounting an ISP can indeed be a "vendor."

But, I quite like this keen little comment, that went unnoticed:

3 hours ago, Prokofy Neva said:

BTW, a judge ruled that the TOS is a contract of adhesion, Mr. Lawyer.  Again, in real life, these aren't good practices.

Hah!  That was like Lecter saying, "Oh, agent Starling, do you really think you can dissect me with this blunt little tool?"  And it's such a keen point with all the ToS being thrown in OP's face.  Adhesion contracts are broad and one-sided and while reasonable, they have holes precisely because they are broad. Purchasing an airline ticket is agreeing to an adhesion contract and yet airlines are sued all the time.  There are many, many other reasonable expectations.

The weakness of adhesion contracts ( TOS, or "General Conditions") is that they are ultimately challengeable in court since they can be found unfair, especially since they are also the blunt-force bargaining chip leveraged against consumers wishing to partake, or already invested in, the provision of a service.

In a discussion amongst squabbling vendors on a development or a construction site you might be laughed at for bringing up the boilerplate general conditions when  it is the “special conditions” that holds relevance to most arguments of interest, such as reasonable expectations of service.

Nice point, hit like an uppercut nobody saw.  Just because something is in TOS does not mean one should not be able to ask questions.

[Rex Cronon]

the cloud is a computer network. a set of computers/servers that read and write to and from storage(hard drive, dvd, ssd...), that run programs, that communicate(generates requests and answers requests) with computers on the same network and on other networks. there are server farms(buildings filed with servers) located on different parts of states, countries or continents that are part of the same network(aka cloud).

if a file is uploaded to one of the server farms in in usa and if that cloud has another server farm in australia, when a user from australia requests that file, then a copy is sent from the server farm in the usa to the server farm in the australia(only if there is no local copy or if a new version is available) and the user from now on can access that file from the server farm in australia without having to wait to be downloaded from usa. requests from other australian users for the same file will also get the file stored in the australian server farm

in the case of sl the cloud provides storage of user data, like textures, sounds, objects, scripts(whatever is in your inventory). at this time not every type of inventory has been ported to the cloud. i think the cloud copies only requested files from sl servers to other remoter server farms based on what is requested. although is possible that ll now stores most of our inventories in the cloud exclusively.

therefore ll by moving to the cloud should allow users from any part of the earth to have access to objects(textures, linked objects, sounds...) just as fast as somebody located in california(as long as the cloud doesn't bork:) in sl case not everything might be able to be moved to the cloud. chat, speech and position updates for objects and avatars will still have to be sent from sl servers to users across the web in real time. that is until we switch to the quantum web:)

Edited November 4, 2019 by Rex Cronon
typos

[Wulfie Reanimator]

On 11/4/2019 at 8:51 AM, Prokofy Neva said:

... or is somehow immature, imbecile ...

imbecilic*

17 hours ago, Prokofy Neva said:

But when it comes to Linden Lab, the fanboyz get all shirty and protective and imagine that none of us can ask any questions about it. Ridiculous.

What's ridiculous is the tone and the apparent amount of vitriol you have towards people whose primary opinion seems to be "I don't care, why do you?" which is about as neutral as you can get.

17 hours ago, Prokofy Neva said:

The request was to ask vendor of WHAT KIND of service. Internet? Servers? Telephone? Pizza delivery? And once again, this is about WHETHER OR NOT SL moved to the cloud. Not which cloud provider.

This is from the blog post you linked:

Quote

We quickly opened a ticket with the network provider and started engaging with them.

As far as I know, "network providers" don't provide pizza, and SL doesn't run on a cell service. That doesn't seem to satisfy your question because I know you're still very paranoid about "the cloud" as you have little understanding of it, which is fine as you are a non-technical person. But because you are a non-technical person, giving you the full answer might not satisfy you either, because you're hung on the scary "cloud" term.

Lots of different things are "the cloud." There is no one "cloud network" type, the term is ambiguous and useless without proper context. Not everything has to be on "the cloud(s)" for SL to work.

• If you can log into any computer at your workplace/school (especially in separate buildings) and instantly get all of your files on the desktop, that's "the cloud."
• If you remember SL Go, that was "the cloud." (Same goes for OnLive or any game-streaming service. Cloud gaming.)
• If your company has ANY software that runs on a remote server, not on the same computer, that is "the cloud."
  • This means that even if/when LL had its own server farms, they were "the cloud."
• Linden Lab has been using AWS for a good while now, that is "the cloud." (<-- This is the answer you want, even if unrelated to the actual problem.)
  • A presentation of this move was done back in 2017.
  • They've also made blog posts related to AWS even before that, but that post is a different "cloud."

• "The cloud" is just "a remote server that does something for you." The internet as a whole is "a cloud."

Edited November 5, 2019 by Wulfie Reanimator

[ChinRey]

5 hours ago, Ardy Lay said:

Internet access provider is a vendor.  I see my packets take a different, shorter, less lossy route now.

I haven't checked the route but what I do notice is that SL loads much, much faster now. It's like night and day - I think I'm actually back to the performance I had before USA lost its net neutrality!

[ChinRey]

5 hours ago, Prokofy Neva said:

And once again, this is about WHETHER OR NOT SL moved to the cloud.

The answer is both yes and no. We have to remember that SL uses different servers with very different tasks. Some functions are better done with cloud hosting and some of those have been on the cloud since 2014. Other functions are not cloud hosted and hopefully they'll never be.

[Prokofy Neva]

11 hours ago, Theresa Tennyson said:

From the blog post:

Now that we had it in a bad state we started to troubleshoot and figured out really fast that it wasn’t our equipment. Our stuff was (and still is) working just fine, but we were getting intermittent errors and delays on traffic that was routed through one of our providers. We quickly opened a ticket with the network provider and started engaging with them. That’s never a fun thing to do because these are times when we’re waiting on hold on the phone with a vendor while Second Life isn’t running as well as it usually does.

After several hours trying to troubleshoot with the vendor, we decided to swing a bigger hammer and adjust our Internet routing. It took a few attempts, but we finally got it, and we were able to route around the problematic network. We’re still trying to troubleshoot with the vendor, but Second Life is back to normal again.

Note that they identified the "vendor" as a "network provider" before they started calling it a "vendor." 

Yeah, I can read Theresa. And I read it twice when it came out. But I didn't know that the network provider WAS the vendor; they are coterminous. I think when techies use this sort of jargon like "provider" and "vendor" -- which *is* jargon -- they don't realize that it can be opaque.

[Prokofy Neva]

11 hours ago, Bradford Mint said:

I haven't said you wouldn't get to know (you or anyone else).  What I said was that your service is with LL, not the cloud provider.  LL cannot provide an SLA greater than that of their provider (well they could but would be dumb to try) and their SLA is absolutely zilch so they're hot to trot!   Cloud provision isn't vital to the functioning of any network, only to those who choose to consume the compute, service and storage resources that may be offered over any network.  This seems moot at this time however.  I'm not sure that there's an assumption that things get better with moving to cloud.  What does happen is that a local datacentre can be downsized or removed, no need for specialist staff to host "whatever service", the replacement services are consumed as a service and contractually against an SLA for a specific contract term and crucially for some, moves the cost model from Capex to Opex which has accounting implications as well as usually definable benefits in terms of service provision and upgrade. Those are some of the key points of outsourcing whether it's people or service, i.e. cloud.  Cloud in this context is just synonymous with service.  Nothing magical about cloud to see here.

Phone them up and ask, i'm sure they'll answer.  However, as i've already pointed out, how they deliver the service isn't really that relevant to the end user, random player or business owner.  The service as provided by LL remains the same with the same SLA and TOS. *shrugs*

This is just Newspeak and isn't the point.

I realize you want to show off your knowledge and have us all know that you're the inside dope with the inside dope and you know grown-up terms like "Capex to Opex". But it's irrelevant to this very basic conversation and very basic question I've been asking.

Linden Lab no longer provides SLAs to my knowledge. That is not what a premium account is. They used to provide something like this to the IBM sort of companies with multiple islands, when they had the "Nebraska" program. But they've discontinued those.

I'll never forget when I happened to be standing next to Philip when a guy from some giant company came up to Philip and began yammering about what he wanted from Linden Lab, wjem they began ordering up lots of islands, and at some point said "I'll have our compliance department talk to your compliance department."

"We don't have a compliance department," Philip said sort of naively. And not surprisingly. LL didn't really start out as a company company. It started out as a, well, Lab. It was kind of thrown together. They got famous and bigger at a time when they didn't have the staff or skills to meet the difficulties.

 

[Prokofy Neva]

6 hours ago, ChinRey said:

The answer is both yes and no. We have to remember that SL uses different servers with very different tasks. Some functions are better done with cloud hosting and some of those have been on the cloud since 2014. Other functions are not cloud hosted and hopefully they'll never be.

Thank you. As always, you're the first/only person to answer my question. And you've provided actual factual information instead of trying to either show of your knowledge or humiliate those who aren't in this field. So there's a debate here among techies most likely, and they may disagree. LL may think something should be put in the cloud others are horrified at, but maybe these are esoteric questions that real companies in the real world have to do.

I still have to insist on getting a question to the basic, larger question of "moving to the cloud" which is "a thing" that is done that is on a mass scale and where things go wrong.

For example, when my hospital "moved to the cloud," they completely lost my file. When some place I worked for "moved to the cloud" the CMS ceased working. And so on. "Moving to the cloud" is like, oh, the Oregon Trail or something. Thomas "Broken Hand" Fitzpatrick and the Laramie Treaty. The Donner Party. Stuff like that. There are trails, there are wagons that move, there are guides, but then, they are attacked by Indians whose land they then steal when they break the treaties. Am I close?

[Prokofy Neva]

6 hours ago, ChinRey said:

I haven't checked the route but what I do notice is that SL loads much, much faster now. It's like night and day - I think I'm actually back to the performance I had before USA lost its net neutrality!

This idea that the utterly faux concept of "net neutrality" is something "that changed" is the most laughable thing I've heard in this discussion and I think you're smarter than that, Rey.

First of all, SL didn't change "for everybody" -- it's as bad or worse as it always has been. You find it better -- for who knows what reason. I have a new computer and new graphics card -- it was better for awhile; now it's terrible again.

"Net neutrality" is an ideology conceived by a professor in a university. Real life is far more complicated. Obama didn't succeed in somehow establishing socialism -- including the socialism of "net neutrality," which is why we have AOC and her posse. Trump didn't succeed in dismantling whatever state of affairs existed under Obama such as to say we "lost" something or that somehow it was "regained". 

Companies did not suddenly start charging higher fees to have data go faster here. The price of our phone bills is exactly the same as it was under Obama. AWS servers, which companies I work for rent in various large numbers do not charge more or work faster or do anything differently now than they did under Obama. All of this is sheer lunacy.

[Kimmi Zehetbauer]

11 hours ago, Lancewae Barrowstone said:

Theresa posted the LL language:  "We quickly opened a ticket with the network provider and started engaging with them. That’s never a fun thing to do because these are times when we’re waiting on hold on the phone with a vendor while Second Life isn’t running as well as it usually does.

Gotta love vendors. I deal with them all the time in my RL business.

[Mollymews]

2 hours ago, Prokofy Neva said:

"Net neutrality" is an ideology conceived by a professor in a university. Real life is far more complicated. Obama didn't succeed in somehow establishing socialism -- including the socialism of "net neutrality," which is why we have AOC and her posse. Trump didn't succeed in dismantling whatever state of affairs existed under Obama such as to say we "lost" something or that somehow it was "regained". 

Companies did not suddenly start charging higher fees to have data go faster here. The price of our phone bills is exactly the same as it was under Obama. AWS servers, which companies I work for rent in various large numbers do not charge more or work faster or do anything differently now than they did under Obama. All of this is sheer lunacy.

is interesting how different countries approach the question of supply. And what 'net neutrality' can mean, in any given practiced approach applied to a supply market

in New Zealand we have now approx. 140 ISPs who between them offer about 1360 different broadband plans to consumers

my 10/50 broadband plan costs $NZ65 a month. About $US42. Before the government-sponsored (UFB) nation-wide fibre network was connected down my street I used to pay $NZ105 a month for 1/12 ADSL2 plan, less than 2 years ago

and before the UFB network there were only about 6 ISPs in the country. And only 2 where I live from whom I could get ADSL2. Both who charged the same price. And one of the ISPs was also the line company for them both. Choose AB = BA

now I can get any of the about 140 ISPs that are available.  Most of these new ISPs were able to enter the unregulated ISP consumer market, because a) they run on the UFB fibre network and b) they are hosting on Amazon, Google and Microsoft cloud services

in New Zealand "net neutrality" means that the ISPs (content providers) are unregulated, as this best serves the consumer market in terms of consumer choices. When there are many choices of ISPs, pricing plans get competitive

the UFB fibre company is regulated as its a monopoly. NZ has the same model for the electricty, gas, and water/waste grids. The line companies for these are also regulated. Electricity and gas suppliers are unregulated. Which means that we as consumers can switch between electricity and gas suppliers as and when we choose. Which has made the 'content' (electricity/gas) cheaper and far more reliable in terms of continuity of supply, than was the previous case when the 'content' was also provided by the line company

regulated in the New Zealand sense means that the fibre line company sits down with its customers (the ISPs) and works out a fair and reasonable price to use the line into my house (which I pay a maintenance fee for, to the line company, same for gas and electricity and water/waste pipes into my house) should I buy 'content' from a ISP. When the ISPs and line company can't voluntarily agree a price then they are regulated into a mediation process

the line company by regulation can't refuse or throttle 'content'. The ISPs can refuse, throttle and not supply 'content' (services) as they choose. If as consumers our ISP is not providing what we want then we switch to another ISP to buy from. 140+ locally available. Or if we want we can subscribe to an overseas ISP as we like

in the New Zealand model, the "net neutrality" regulations have nothing to with consumer 'content'. It has everything to do with the line company, which in this model is required to be neutral toward its customers - the ISPs

[Pixieplumb Flanagan]

oh socialism, eh? uh huh.

[ChinRey]

4 hours ago, Prokofy Neva said:

LL may think something should be put in the cloud...

One crucial factor here is time. Data transfer between computers is fast but not instantanous and the further away from each other they are in the physical world, the longer it takes so ideally we want all servers to sit right next to each other so they can communicate faster with each other. But we also want them to sit right next to the users and that's impossible of course. The two main types of servers SL uses have different priorities here:

The content delivery servers are the ones that store all the big files for the content - textures, meshes etc. They don't communicate much with each other and it's easy enough to keep several copies of the commonly used files on several servers. This is what the cloud does best and SL uses a CDN (content delivery network) with 20 (I think) server farms all over the world. The one that delivers content to you is the one that is physically closest to where you live.

The sim servers are the ones that run and keep track of all the activities in a region. You can't run several copies of the same simulator at the same time because that would create a holy mess and since SL'ers are spread out all over the world, there is no ideal place to keep them close to the users. But they do communcate a lot with each other so you want them as close to each other as possible.

LL has been talking about moving the sim servers to the cloud too but I'm fairly sure they haven't and I hope they never will.

[Alyona Su]

Never-mind. Taking the high road here, LOL

Edited November 5, 2019 by Alyona Su

[Pixieplumb Flanagan]

11 hours ago, Prokofy Neva said:

This idea that the utterly faux concept of "net neutrality" is something "that changed" is the most laughable thing I've heard in this discussion and I think you're smarter than that, Rey.

First of all, SL didn't change "for everybody" -- it's as bad or worse as it always has been. You find it better -- for who knows what reason. I have a new computer and new graphics card -- it was better for awhile; now it's terrible again.

"Net neutrality" is an ideology conceived by a professor in a university. Real life is far more complicated. Obama didn't succeed in somehow establishing socialism -- including the socialism of "net neutrality," which is why we have AOC and her posse. Trump didn't succeed in dismantling whatever state of affairs existed under Obama such as to say we "lost" something or that somehow it was "regained". 

Companies did not suddenly start charging higher fees to have data go faster here. The price of our phone bills is exactly the same as it was under Obama. AWS servers, which companies I work for rent in various large numbers do not charge more or work faster or do anything differently now than they did under Obama. All of this is sheer lunacy.

you little tease, you

 

[KirinGale]

Lurking here... I was on board with being more transparent and specific with consumers, the whole spiel about not worshiping TOS, or using TOS as a crutch or point to bully other users who want more open communications akin to what is available in other franchises and products. All good stuff until... whatever is happening now.

Y'all have fun 😂

Edited November 5, 2019 by KirinGale

[Maitimo]

On 11/4/2019 at 4:23 PM, Luna Bliss said:

So how many people does it have to disturb, and to what degree, before anybody has the right to complain?  How do you determine when the 'complaint level' has been reached?

That's really down to the customers. The level is reached when enough people have cancelled their premium payments or given up their private estates, and deleted their SL accounts. Its up to each individual to decide how much disruption needs to happen before they do that.

And then its up to Linden Lab to decide how many deleted accounts is enough before they consider SL no longer viable, and pull the plug permanently.

[Parhelion Palou]

11 hours ago, ChinRey said:

LL has been talking about moving the sim servers to the cloud too but I'm fairly sure they haven't and I hope they never will.

The Head of Hardware (I forgot the title) who spoke in a video a couple years ago said LL's data center is becoming obsolete. They decided it would be cheaper to put SL on the cloud completely than to build a new data center. I'm sure they thought it shouldn't be that hard; they'd done it with Sansar and could use the people who did that work. I would've loved to see the look of horror on the faces of those people when they saw the SL code base.

The data center was set up to get the fastest possible data transfer between the servers in order to reduce region crossing problems. I don't see how they'd be able to guarantee that on something like AWS unless LL could arrange a similar setup there. LL has said they won't tell us that they're moving simulators to the cloud until after it's done. One clue might be region crossings getting a lot worse.