SL Moving to the Cloud

[Parhelion Palou]

The New World Notes blog yesterday covered a Linden Lab talk at Amazon's AWS re:invent 2017 conference, in which LL mostly talked about Sansar on the AWS cloud but also covered moving SL to the cloud. LL hopes to have the migration finished next year. I think we now know why LL moved some Sansar developers to SL. They're already familiar with the work needed to run things on AWS so now they get to try it with SL. LL wants to run regions only when someone is on them, though busy regions could be kept running to save start-up time. (per NWN and LL's talk)

I believe next year will happen just as much as I believed Ebbe when he said in the summer of 2014 that an alpha version of Sansar would be out by the end of 2014 or early 2015, with a beta in the summer of 2015. If LL has Sansar developers (who are likely new to SL) doing the work, they have no clue how SL works. They're used to experiences which don't interact with anything outside of themselves, so they're fairly easily started or stopped as people enter and leave them. With SL they have to decide how to handle the regions around the region someone is in. Does the person on the region see nothing, then suddenly the region next door appears when someone enters it? What happens with region crossings? I don't see any way that region crossings won't be worse than they are now. Teleports would be slower if you're teleporting to a region that isn't running at the time, and might be slower anyway. How fast can one simulator instance talk to another one in the AWS cloud?

The plus side is land could get cheaper since LL won't be paying as much to run it. Other than that, is there anything good for us in the move?

[Phil Deakins]

11 minutes ago, Parhelion Palou said:

The plus side is land could get cheaper since LL won't be paying as much to run it.

I think cheaper land is unlikely. I don't see LL voluntarily reducing their income, but I could be wrong.

Edited November 30, 2017 by Phil Deakins

[Rhonda Huntress]

Moving to cloud is over used buzz but you need  to remember that it also means going to virtual servers.  Basically a sim = a server right now.  That physical server is constantly running.  If it were not,m it would take 10 minutes more or less to boot up.  A virtual server just sits in RAM (possibly non-volatile memory at that) and when it is idle, it simply does not get any CPU cycles.  To go from idle to fully running is almost instant  They are not so much turned os as put into a frozen state.

Moving to virtualization does not need much intimacy with the server code as you may think.  The sim code will hardly change at all.  There will be a need for an additional layer for cross sim communications of course.  But the biggest change would be expanding power on demand for the asset database. 

 

Anyway, that's just the initial move; simply swapping physical for virtual.

[Parhelion Palou]

I'm not sure a sim can safely suspend then start again where it left off. If it has to be shut down then it'll have to do the region start-up process. The person doing the talk, senior systems engineer Tara Hernandez, said that they're still working out when to shut regions down. In the case of a very popular region, she said that if the region becomes empty but is likely to have someone show up in 5 minutes then they wouldn't want to shut the region down because they want to maintain high speed. If it was near instantaneous that wouldn't be a concern.

 

[Chic Aeon]

Just wanted to note that they have used this "on demand sim" in Kitely for years (likely forever and before I was there). Kitely is part of Opensim for those of you that do not know. I can't see how it would work on MAINLAND at all, but it could work for islands or groups of islands. In Kitely and other Opensim grids you can have up to 16 sims come online and go off line together. This TAKES TIME however -- especially if the sim is filled with less than great mesh. 

 

A single sim (with smart mesh) CAN work very well this way. I had mine up for a few months there before leaving OS.  So it seems like from what has been said that this will be the new model people can choose from. I am having a difficult time seeing how Mainland could be retrofitted this way.

And -- a question --  Do we KNOW that sims are still one sim per server?  I was under the impression -- especially with all the empty mainland -- that many sims were now stacked on a single server. I remember a time when mainland took a BIG hit in FPS (before it was so empty) and assumed that had something to do with it.  So any actual FACTUAL info on that would be good to know. 

 

 

[LittleMe Jewell]

Assuming that this wiki is at least somewhat current, then it is one sim per 'core'.  Servers these days can have many cores, so unless someone has more specifics on the types of servers that LL uses, I'm not sure we can figure out how many sims per server.  Additionally, with virtual setups and over-allocation of resources, you could theoretically have more sims that 'physical cores' - though it would take a lot of careful setup and monitoring to attempt that with SL sims.

http://wiki.secondlife.com/wiki/Land - "There is one full region per server host CPU core."

 

[Parhelion Palou]

15 minutes ago, Chic Aeon said:

Just wanted to note that they have used this "on demand sim" in Kitely for years (likely forever and before I was there). Kitely is part of Opensim for those of you that do not know. I can't see how it would work on MAINLAND at all, but it could work for islands or groups of islands. In Kitely and other Opensim grids you can have up to 16 sims come online and go off line together. This TAKES TIME however -- especially if the sim is filled with less than great mesh. 

 

A single sim (with smart mesh) CAN work very well this way. I had mine up for a few months there before leaving OS.  So it seems like from what has been said that this will be the new model people can choose from. I am having a difficult time seeing how Mainland could be retrofitted this way.

And -- a question --  Do we KNOW that sims are still one sim per server?  I was under the impression -- especially with all the empty mainland -- that many sims were now stacked on a single server. I remember a time when mainland took a BIG hit in FPS (before it was so empty) and assumed that had something to do with it.  So any actual FACTUAL info on that would be good to know.

SL has long been running multiple sims per server. A full region uses one CPU core. A homestead shares a CPU core with 3 other homesteads. In the old days a quad core server could handle 4 sims. I don't know what LL is using now, but they could be running 16 or more sims on a server.

I wonder how on demand sims would work in SL, which was designed for sims to run constantly. Only LL can shut a sim down; a region owner can only restart the sim. How much content is out there that expects the sim to be running constantly?

[Vanity Fair]

I thought that part of Tara's talk was very interesting. I think that Linden Lab is going to have a serious challenge on its hands to move all or part of their sim servers over to the AWS cloud. It's dated technology that will require a lot of rejigging in order to work. I'm actually surprised that LL is taking this step, when they could leave everything running on their server farm in Arizona. Then again, like Tara said in her talk, it's costly to run and upgrade a physical server farm. They're obviously hoping it will be cheaper using AWS, and that they can in turn pass some of those cost savings onto SL users. 

It will be interesting to watch.

 

[Parhelion Palou]

Tara said the data center is using a lot of old hardware. Updating the hardware practically requires a data center rebuild (new racks, new fiber, etc.).

Some interesting bits from the talk:

- SL is still getting somewhere between 0.75 to 1 million monthly average users. In 2016 the in-world economy generated around 1/2 billion US dollars.

- Some parts of SL's system are extremely sensitive to latency. They won't work if you put them in separate racks. So far they haven't had to deal with it, but they'll have to when moving to AWS.

- Third party viewers are a problem. This is one I don't get ... the third party viewers are mostly based on LL's viewer code. If the viewer has to change to work with AWS, I'd think that change would become part of the next releases of the third party viewers. One of the NWN comments indicated it could be a security problem with AWS. I thought Sansar's no 3rd party viewer policy was to maintain consistency, although experiences could provide plug-ins to change the look & functionality of the viewer. Perhaps it's something to do with AWS.

[Anna Nova]

As an SL sailor and pilot, I can't see those sim crossings getting any better.  They're almost unusable right now. (and da, I understand why)  If they do become impossible then 98% of the reason I pay tier and premium will go too.

Edited November 30, 2017 by anna2358

[NevaehHeartstrings]

One wonders what they will do regarding bots? people use them for traffic boosting in search not just to inflate the traffic numbers but people leave them out to make it look like place is busy. I've seen lots of clubs just log bots in and out all day long.

Will they finally ban bots or find a new way to restrict them, seems it could be abused hard by some, I count 40 a day on average just stalking me unwanted in my private region.I wonder if Telehubs will finally make a comeback after all these years? people actually having to stand around talking to each other before the experience is 'ready'

That other thread "Does LL want to fix SL'  well this is some idea they are trying to at the least do something.It would seem to me they are trying to turn a product neglected by past management into a great product, it will be hard to fault them at that considering 75% of today's current linden lab had anything to do with past management mistakes or decisions.. considering they were not employed at the time by the lab.

 

 

 

Edited November 30, 2017 by NevaehHeartstrings

[Penny Patton]

12 hours ago, Phil Deakins said:

I think cheaper land is unlikely. I don't see LL voluntarily reducing their income, but I could be wrong.

The idea is that by setting up servers in this way would drastically reduce LL's costs. This would allow them to reduce the cost of land while still maintaining higher profits even if the lower prices did not draw in new landowners. However, the whole point of reducing the cost of land would be to entice more people to buy land, thus raising LL's income. You see how that works? People might scoff at paying $25/mo for 4096sq.m., but it the same land cost $10/mo, a lot more people might be interested.

Think of it this way: McDonald's would make a heckuva lot more on every cheesburger sold if their cheeseburgers cost $10 instead of $1, but they'd sell a lot fewer cheeseburgers that way. Right now, Linden Lab is selling $10 cheeseburgers.

[Paul Hexem]

1 hour ago, Penny Patton said:

The idea is that by setting up servers in this way would drastically reduce LL's costs. This would allow them to reduce the cost of land while still maintaining higher profits even if the lower prices did not draw in new landowners. However, the whole point of reducing the cost of land would be to entice more people to buy land, thus raising LL's income. You see how that works? People might scoff at paying $25/mo for 4096sq.m., but it the same land cost $10/mo, a lot more people might be interested.

Think of it this way: McDonald's would make a heckuva lot more on every cheesburger sold if their cheeseburgers cost $10 instead of $1, but they'd sell a lot fewer cheeseburgers that way. Right now, Linden Lab is selling $10 cheeseburgers.

I have a nagging feeling that anything they save in costs would go right into someone's pockets, and our prices wouldn't go down much, if at all.

[Penny Patton]

That's entirely possible depending on the management at LL. Based on what I've seen, it seems a safe bet that SL land tier is LL's number one source of income, and land ownership is dropping. According to information at GridSurvey if trends continue then in a few more years land ownership in SL will have declined to half of what it was 10 years prior. LL knows that they need to bolster their income if they want to stay in business and have shown a surprising amount of development activity in SL recently, so we'll just have to wait and see. Just remember, it was LL who first brought up the possibility that this move might allow them to be more flexible with pricing.

[Kurshie Muromachi]

Another similar earlier video from GDC17

 

[Phil Deakins]

6 hours ago, Penny Patton said:

The idea is that by setting up servers in this way would drastically reduce LL's costs. This would allow them to reduce the cost of land while still maintaining higher profits even if the lower prices did not draw in new landowners. However, the whole point of reducing the cost of land would be to entice more people to buy land, thus raising LL's income. You see how that works? People might scoff at paying $25/mo for 4096sq.m., but it the same land cost $10/mo, a lot more people might be interested.

Think of it this way: McDonald's would make a heckuva lot more on every cheesburger sold if their cheeseburgers cost $10 instead of $1, but they'd sell a lot fewer cheeseburgers that way. Right now, Linden Lab is selling $10 cheeseburgers.

Oh yes, More people almost certainly would buy land or more land, but there's a downside. Those who have land would pay less for what they already have. And it's not just mainland. If mainland tier comes down, then a whole mainland sim's tier comes down, which means that a private island's tier must also come down. I just can't see LL risking that the tier payment increases of new buying will outweigh the tier losses from existing landowners. As I said, I could be wrong about that.

It would be good if tier prices did come down, because owners of mainland might even be able to make an actual profit, simply through owning it, depending on the decrease. I did some sums a few days ago, and, done the right way, the second 512 that a premium member gets need only cost an extra US$1.43 per month instead of the normal US$5 per month. And it's the same all the way up land sizes, so that a whole sim need only cost a little less than half the normal US$195.

Edited December 1, 2017 by Phil Deakins

[Penny Patton]

I am not saying LL will definitely reduce land prices, I'm just saying that I believe it makes good business sense and trying to explain why. A lot of people want to own land but think it costs too much for what you get.

However, I would also pair that with showing people how to get more value from their land. Show them it's worth the money.  I've seen so tragically few people who've gotten even half the value from the land they're paying for, and you don't have to be super skilled at modding to do it. LL just needs to give people examples through public spaces like the welcome areas and hubs, and provide better tools.

[ChinRey]

16 hours ago, Parhelion Palou said:

- SL is still getting somewhere between 0.75 to 1 million monthly average users.

It's probably around 700,000 by now. The last figure we heard from LL was 800,000. That was in February this year and the number seems to be dropping by a fairly steady 10,000 a month.

And please don't use the world "users" here, that always leads to misunderstandings. It's the number of accounts logged on the last 30 days. Nobody knows how many people logs on to Second Life regularly but the best estimate I have been able to come up with is somewhere between 75,000 and 200,000, probably in the lower half of that range.

 

16 hours ago, Parhelion Palou said:

In 2016 the in-world economy generated around 1/2 billion US dollars.

I'm not sure where you get that figure from or what it means.

The total amount of money taken out of SL by entrepeneurs (merchants, land rentals, DJs, escorts, LindEx speculants, money gamers etc.) is somehwere between 45 milion and 60 million US dollars a year.

Linden Lab's gross income from Second Life is probably lower than that and certainly not higher.

[Phil Deakins]

52 minutes ago, Penny Patton said:

I am not saying LL will definitely reduce land prices, I'm just saying that I believe it makes good business sense and trying to explain why. A lot of people want to own land but think it costs too much for what you get.

I agree with that. Mainland ownership has to remain exclusive to premium members, or all manner of mayhem would break loose. But the sizes need to be increased, or the cost decreased, or both, to make it more desirable. Also, the tier level jumps between the larger sizes is just too great. Perhaps moving SL to a cloud might cause changes in land and tier to come about. It's just that I don't see LL taking the risk of reducing the income from tier significantly, in the hope that tier from increased land buying would cover it. I hope I'm wrong.

[Qie Niangao]

Although I'd love for tier to come down (or equivalently cover more land for the same tier), I suspect that folks have very unrealistic expectations about the effect of reducing hosting costs by moving to the cloud. Even if there were a 100% no-cost cloud, most of LL's total cost of operating SL would remain unchanged. Although hosting SL is a big deal, other costs (notably development and non-hosting operations) must represent a much bigger chunk of the expense budget than they do for other popular "platform" businesses.

So if tier came down by, say, 5% -- likely the most that could be justified by a 25% reduction in hosting cost -- would people be all excited by the cuts, or fuss and fume about getting screwed by the Lab?

[LittleMe Jewell]

2 hours ago, ChinRey said:

 

19 hours ago, Parhelion Palou said:

In 2016 the in-world economy generated around 1/2 billion US dollars.

I'm not sure where you get that figure from or what it means.

 

The 1/2 billion US dollars amount was stated in the GDC17 video above and it was referred to as the GDP for 2016.  I think it thus includes all L$ transactions in SL, not just the amount of money pulled out.

[LittleMe Jewell]

I still don't understand how we sometimes get double posts with only one click of the button.

Edited December 1, 2017 by LittleMe Jewell

[LittleMe Jewell]

9 minutes ago, Qie Niangao said:

So if tier came down by, say, 5% -- likely the most that could be justified by a 25% reduction in hosting cost -- would people be all excited by the cuts, or fuss and fume about getting screwed by the Lab?

Maybe a reduction in hosting costs would be enough to justify breaking down the current tier groups a bit more, rather than the doubling that is there now.  i.e. No change in the current costs, but the ability to go to 1536 without being charged for 2048, or to jump from 2048 to 3072 without being charged for an entire 4096, etc...  Even that would make a lot of people happier and I think it would encourage many to increase their existing land holdings a bit.

[ChinRey]

4 hours ago, LittleMe Jewell said:

The 1/2 billion US dollars amount was stated in the GDC17 video above and it was referred to as the GDP for 2016.  I think it thus includes all L$ transactions in SL, not just the amount of money pulled out.

Oh, that explains it.

So, you buy 100 Lindens on LindEx (never mind you can't buy such a small amount - this is an illustration), that's 100 Lindens added to SL's "GDP". You give those 100 Lindens as a tip to a DJ, that's another 100 Lindens added to SL's "GDP". The DJ buys a new shirt for the money - another 100 Lindens added. The shirt merchants uses the money to buy a new template and the template merchant finally sells it on LindEx for hard cash. Those 100 Lindens you bought added 500 Lindens to the economy. But it's not generated money, it's recycled money. There's a slight difference there.

As the old saying goes, there are three kinds of lies: lies, damned lies and statistics.

[Qie Niangao]

6 minutes ago, ChinRey said:

But it's not generated money, it's recycled money. There's a slight difference there.

Yeah, but painfully familiar from how sports franchises (for example) justify new facilities and other subsidies on the basis of supposed economic benefit to the community.

Edited December 1, 2017 by Qie Niangao