Amazon cloud servers and FPS improvement/less lag - nope

[Jackson Redstar]

I filmed a wedding the other nite on a sim that was in the Blake area and is now on Amazon servers - it is a marina with a airport above and the wedding had about 40 guests. I was getting between 6-12 FPS which is alot lower than I often get doing weddings on linden servers. Not of course any type of A/B comparison -  I had hoped there would be some improvement in lag reduction and/or FPS on the new cloud servers, and at least in this one sim - not a bit. I suppose there may be other benefits but maybe sot so much in performance...

[Qie Niangao]

For a decrease in viewer FPS, I wonder if the Texture Console showed that all textures had loaded. Lately -- and especially on regions with so many avatars -- I'm seeing substantial lag from slow asset loading, regardless of whether the sim is hosted in the datacenter or the cloud. It feels like the CDN is saturated, maybe throttled somehow.

Aside from all that, is there reason to expect viewer FPS to increase as a result of better server performance, which would seem the most we could hope from the cloud? I guess there would be if viewers were bottlenecked by delayed server updates, but I'm not sure when this happens.

[Fritigern Gothly]

2 hours ago, Jackson Redstar said:

I filmed a wedding the other nite on a sim that was in the Blake area and is now on Amazon servers - it is a marina with a airport above and the wedding had about 40 guests. I was getting between 6-12 FPS which is alot lower than I often get doing weddings on linden servers. Not of course any type of A/B comparison -  I had hoped there would be some improvement in lag reduction and/or FPS on the new cloud servers, and at least in this one sim - not a bit. I suppose there may be other benefits but maybe sot so much in performance...

Why do you expect your viewer to perform better when the improvements are only server-side?
You never mentioned if you had been doing recordings in that region with 40 people attending before, with a similar scene complexity. With a similar number of textures that needed to be loaded and kept in memory.
You should, however, see an improvement in how long it takes to cross from one region into another, and how fast and reliable teleports are. Scripts may run smoother too. All that takes place server side.
But rendering the textures, objects, avatars, etcetera, that happens on your computer and no matter how fast the server is, no matter how smooth it runs over there on the Amazon servers, none of that has any effect on how well your viewer performs.
What DOES have an effect is

• Rebooting your PC before such an important event to make sure nothing is running in the background which should not be there.
• Making sure you have plenty of RAM
• Making sure your graphics card is nice and modern.
• Make sure to only run SL and your screen recorder (OBS?)
• And for pete's sake, do NOT clear your cache every time you close SL, you will have to load all the textures as well as your inventory all over again, causing lots of delay.

In short: Neither the server's software, nor where the server is hosted has ANY influence on your FPS. Anyone telling you otherwise has little knowledge of how networking works. Yes, slow loading textures is a thing but again that is not the FPS.

[Jackson Redstar]

11 hours ago, Fritigern Gothly said:

Why do you expect your viewer to perform better when the improvements are only server-side?

so, in short, other than better region crossings the end user will see zero benefits of cloud based servers vs linden servers?

[nuzicx]

As a datacentre technican who works for one of the big 4 datacenter providers. who you host with and the technologies you employ does make a difference. Cloud hosting does make a difference as they support colocation accross many continents, hundreds of datacenters. Reducing lag by having colocated server closer to the customer, creating mesh networks of servers that communicate together and act as one big server will enable servers to share files (ie textures, sprites) and bring them closer to the customer, the humen reaction time is 250 milliseconds, would you rather content loaded cross country maby taking 200-400 milliseconds or localy in under 30 milliseconds.

The viewer has a bandwidth cap of 10 mbps (1.25MB), if you had a 100mbit connection, you would only be using 10% of that, this is the major problem, with the advancement of graphic cards we are able to load many things at once but we are being limited by bandwidth. There are many other factors that come into play also, its upstream capabilities and even the responce time of the servers storage, nvme is the future and what i would recomend but the standard is still only 100iops.

Rendering is not an issue and this game is playable on onbult graphics nowdays as long as settings are kept low, a $500 pc would run this game A.O.K. If you got the $$$ a 3080 RTX and Ryzen 5900 with 32GB ram would be the way to go

Servers are important as they send you the files before rendering takes place. They to need fast ram, multicored CPU's and fast storage

 

 

13 hours ago, Fritigern Gothly said:
In short: Neither the server's software, nor where the server is hosted has ANY influence on your FPS. Anyone telling you otherwise has little knowledge of how networking works. Yes, slow loading textures is a thing but again that is not the FPS.

Just to add to this there is a stuttering effect felt by the user when they wait for textures/sprites to be downloaded from the server, however a moden graphics card once files recived from the server loads these in nanoseconds

Edited November 13, 2020 by nuzicx

[Fritigern Gothly]

4 hours ago, Jackson Redstar said:

so, in short, other than better region crossings the end user will see zero benefits of cloud based servers vs linden servers?

Pretty much that, and teleports being faster and more reliable. There will be more in due time but an FPS increase will not be one of them, because that depends on your PC, not on the servers.

[Sharie Criss]

So far, Teleports are not faster or more reliable going between uplifted (AWS) regions. It actually seems to be worse right now. Hopefully they will sort that out along with pretty horrible performance from the content servers delivering assets / textures. Akamai is great for web content delivery, but it seems to be not so great dealing with game assets. It doesn't help that the viewer's caching is just completely broken, where zooming far away, letting that load, and then resetting the camera to close and finding everything gray again having to wait for the textures to get reloaded.... That's just inexcusable when you have a large cache on a high-end NVMe drive. As I mentioned in other threads, I had high hopes too that moving from the OLD datacenter servers to the modern AWS would result in much less server side lag caused by lots of avatars in the region and significant scripts, but I'm not seeing any performance benefit at any level. It's as if they brought everything over using 2007-era computing resources ignoring Moore's law.  With the size of the deployment, LL isn't paying retail prices here, but when I look at how much I pay for regions and what I can get for a VM at a hosting company like Digital Ocean (which has a simple pricing scheme unlike AWS) it leaves me scratching my head and frustrated that the performance is still just as crappy in this new supposedly "high end" environment.

Continuing performance problems is that elephant in the room that LL doesn't appear to want to acknowledge.

[Taiga Cloud]

Change your way of thinking and stop believing linden labs cloud will work wonders on lag regarding your pc. The majority of lags is caused by technically bad, but visually pleasing meshes made by "creators" who dont do proper topology on their meshes or lack the knowledge of optimizing their meshes they throw on the market. This wont end unless londen labs puts regulations on that. It cant be that a lunar top has 340k tris.

[anthonytorino]

I notice some big improvements, however FPS will have alot to do with the browser used, and graphical settings.

On a I9 system with draw at 512m LOD 4.0 advanced lights on, but shadows off I get around n120 FPS with about 16 avatars near. WIth shadows on and water reflection to trees etc I get 50 to 60 fps. IF I raise shadow detail manually up to very high and smooth it will plummet (firestorm), I get better performance with alchemy if sacrificing some features in interface isn't a big issue, but for video I raise the texture memory use higher which helps a lot (remember if multi viewers are used it will divide the total per viewer active on Nvidia cards). 

Item details, the way they are created, what people are wearing and loading up front, will cause your FPS to drop. The performance has increased on their end by a great margin, but that is not how SL works. 

On dual core processors that are older it will be low. 

 

Sim restarts:  Are much faster on Cloud servers. A full sim restarts in a few minutes and is ready to go. 

 

 

 

 

Edited February 9, 2021 by anthonytorino

[Nalates Urriah]

On 11/13/2020 at 4:21 AM, Jackson Redstar said:

so, in short, other than better region crossings the end user will see zero benefits of cloud based servers vs linden servers?

The servers carry data like the region crossings, communication - they are telling everyone which animation you are using and which direction your walking - press a key to turn and that info has to be updated for everyone that can see you, and scripts... - the servers run those and those have a big effect on vehicle performance. Plus all the physics happens on the server-side. Walk on a floor or bump a wall and the server is figuring out where the avatar can be. There is quite a bit of stuff the viewer depends on that happens server side.

However, that FPS thing for the viewer and render quality are not likely to change much, as people are pointing out. But things will likely happen faster. A scene rendering at 30 FPS and taking a 1 minutes on the old system is likely to render at 30 FPS in 30 seconds (made up numbers)... so 'faster' depends on where you look.

Scaling in the cloud is easier. A lagging region may be able to spin up another instance of it self and split the workload, at some point. I doubt that is implemented now. Just like idle regions may eventually spin down or semi-sleep rather than run at 100% as they do now.

The equipment is going to be newer and my understanding is it will generally stay newer. So, we are likely to have better and more CPUs and faster RAM server side. The Lab used to do massive hardware upgrades every so many years. They will no longer have to concern their selves with that. With Amazon's buying power hardware cost is likely surprisingly low. I suspect the cost reduction will likely show up at some point in our cost.

Users will get benefits from the server-side being in the cloud. Whether they can SEE IT is a matter of knowing where to look.

 

[Raven Huntsman]

Second life needs an engine rewrite to make it multi threaded. It's the primary bottle neck for FPS. You can get the best CPU and GPU combo on the market today, last year,  next year. You'll get the same crappy performance because SL viewer source code (and i think server logic?) is heavily CPU bound and to a single core. It's pretty depressing tbh and every other day I google for news that they may be working on future proofing SL but I don't find much

Edited March 10, 2021 by Raven Huntsman

[Fritigern Gothly]

10 hours ago, Raven Huntsman said:

Second life needs an engine rewrite to make it multi threaded. It's the primary bottle neck for FPS. You can get the best CPU and GPU combo on the market today, last year,  next year. You'll get the same crappy performance because SL viewer source code (and i think server logic?) is heavily CPU bound and to a single core. It's pretty depressing tbh and every other day I google for news that they may be working on future proofing SL but I don't find much

Go ahead, feel free to refactor the viewer code and add multithreading. It's all open source so what is keeping you from undertaking this project?

[jamiesizzle]

On 11/12/2020 at 7:25 PM, Fritigern Gothly said:

Why do you expect your viewer to perform better when the improvements are only server-side?
You never mentioned if you had been doing recordings in that region with 40 people attending before, with a similar scene complexity. With a similar number of textures that needed to be loaded and kept in memory.
You should, however, see an improvement in how long it takes to cross from one region into another, and how fast and reliable teleports are. Scripts may run smoother too. All that takes place server side.
But rendering the textures, objects, avatars, etcetera, that happens on your computer and no matter how fast the server is, no matter how smooth it runs over there on the Amazon servers, none of that has any effect on how well your viewer performs.
What DOES have an effect is

• Rebooting your PC before such an important event to make sure nothing is running in the background which should not be there.
• Making sure you have plenty of RAM
• Making sure your graphics card is nice and modern.
• Make sure to only run SL and your screen recorder (OBS?)
• And for pete's sake, do NOT clear your cache every time you close SL, you will have to load all the textures as well as your inventory all over again, causing lots of delay.

In short: Neither the server's software, nor where the server is hosted has ANY influence on your FPS. Anyone telling you otherwise has little knowledge of how networking works. Yes, slow loading textures is a thing but again that is not the FPS.

Sometimes people try to make things more complicated than they need to be. Basically the variable in his control has remained the same. His experience has become worse after supposed server side improvements there's no need to sidestep the observation. You may have a vested interest in being a dedicated fan but I'm just a paying consumer and the game is becoming a chore to try and enjoy. For the past year the entire grid resembles the busiest clubs I used to attend just 5 years ago with avi's that were 10 times heavier than the new batch of reengineered ruths we've been forced to use. As for your other suggestions I can record 100 plus players in a battle at a decent FPS of 15 and can't record anything with more than 20 avatars in SL with anything but 11 FPS. So please explain to me why there are different FPS on the same connection, same rig, different game and it has nothing to do with the server's software or hosting. It's like saying that the reason I lost the car race has noting to do with the fact that I was forced to drive on dirt roads and other people had paved roads but I might be able to keep up if I buy a Ferrari for the dirt roads.

[RunawayBunny]

5 hours ago, jamiesizzle said:

As for your other suggestions I can record 100 plus players in a battle at a decent FPS of 15 and can't record anything with more than 20 avatars in SL with anything but 11 FPS.

In SL every creator doing max quality 1024x1024 or 2048x2048 this is not a optimized way. This outcome normal for a non coordinated virtual platform. Inside computer games designer usually uses suitable textures for their model 256x256 512x512 1024x1024 etc... design based on visibility. Example: things player often see in high detail, things player rarely see or not worth noticing low detail.

Another thing.. games usually uses assets from local drive while SL using server assets and it takes some time to download all assets (called "rez" i think).

[Theresa Tennyson]

On 3/21/2021 at 1:42 AM, jamiesizzle said:

So please explain to me why there are different FPS on the same connection, same rig, different game and it has nothing to do with the server's software or hosting. It's like saying that the reason I lost the car race has noting to do with the fact that I was forced to drive on dirt roads and other people had paved roads but I might be able to keep up if I buy a Ferrari for the dirt roads.

You just explained it yourself.