SecondLife - Server/Network Performance et al post Jan 2022

[Suzanna Soyinka]

So I'm not one to complain often,(my post history for a 16 year old account is anemic) I usually try to resolve issues outside of the public forum, I tend to call Concierge or put in tickets or even at times file JIRA's but I've done all three and its getting no where at all so I want to kinda put this out in the public sphere and see what others think.

So let me first put my specs on the table. I am a US based player, on a gigabit cable fiber connection which I am hard wired to via CAT6. I am on a I7-9700 with 64GB of RAM and a Geforce 2080 RTX Super

I have played SL for 16 years now and have been an active developer/designer for most of that time.

And starting around January 2022, I started seeing huge latency spikes like the attached.

They are completely random, they last for 3 to 10 minutes and then they go away and I'm back to around 80-90ms latency and my frames clear up.

I've put this on the table with Concierge and support and the Lindens that have addressed the issue have basically done the same old thing they always do, try to blame my computer, or my internet.

But I had my ISP come out literally THE NEXT day and run line tests for me and there is nothing on our end, we even replaced my local router to eliminate that as a potential issue.

And the issues continue to occur. And whats more they don't just happen in one region, they happen almost anywhere. I can be in my empty building sim and see this happen, but the potential for it happening always seems to be more possible when I am around a lot of avatars/shifting assets so the occurrences are greater in busier sims than in less busy ones, but I was at the Zerkalo furniture store the other day and I was the only one there (nice stuff btw, very well made) and I had this occur.

But again the Lindens continue to maintain that this is MY problem only. Except its not, I get friends that are around me that confirm they are seeing higher latencies around the same time I am, though the total latency is always variable.

And whats more I do not have this problem with any other online application I use, in fact my latency to most online services in the United States is generally around 14ms, and in fact my latency to the Amazon CDN's themselves is also around that on average, as the latency test attached shows.

In my 16 years of SL I have never experienced lag like this and I have built 7 different computers since 2005 and always hoped this next video card upgrade would give me the magic bullet that would run this platform at a smooth 60 FPS and have been disappointed every time. I know, from great experience that SL is never going to be perfect, but this is beyond that quirky little understanding that this old streaming asset solution being rendered in real time is always going to have its fits and quirks.....this is extremely laggy and the lag then affects my frame rate, which then affects the entire experience...and if this is what I'm experiencing with my hardware and my capabilities, then I am incredibly concerned what people on lower end hardware are seeing.

SO with that in mind, I dropped 380 dollars on a CHUWI i5-6600 Minipc, with 16GB of RAM and Intel Iris 655 integrated rendering, and guess what? It's even worse, connected to the very same internet, its always 100ms higher than the other PC, regardless of the lag and textures barely even load past 50% LOD even on extremely spartan graphics settings. And Intel Iris is Intel's NEWEST integrated graphics solutions, its supercedes Intel HD and Intel UHD and in the correct situation SL looks almost as good on that as it does on my RTX PC but the rez times are insane and the specs on this minipc are actually better than most midrange laptops.

So in the end this isn't me sayin OMG SL SUCKS SO BAD, I love SL I've loved it for a very long time, but perhaps its time for us to get someone above support to actually start looking at these network level things, cause I feel like this is as laggy as I've ever experienced SL and I don't understand it especially since everything about my hardware and network situation is literally better than its ever been since 2005 and SL is the ONLY platform I use that performs like this, I never get sudden slow downs on large downloads on Steam, I never get video stutter or issues with playback when streaming TV or movies to my TVs, and what's more I'm the only person ON my network, there are two TV's, my two PC's, my phone and my tablet connected to it and usually when I'm on SL its just the ONE PC and none of the rest of them are even on....so I don't have a local bandwidth issue, and all my speed tests and latency tests to anywhere in the US are all within 15 to 30ms.

So why is SL like this? Is SL like this for anyone else out there? My friends say they see it too, but I wanna know if this is global or not.

 

Edited March 13, 2022 by Suzanna Soyinka

[bigmoe Whitfield]

Sounds like a routing issue between you and the aws center in Oregon.    I'm at 120ms in indiana,   in utah I was at 36ms everywhere, so routing def plays a part.

[Suzanna Soyinka]

Well yes, but generally my average is 80 to 90 MS, so the sudden 500ms spikes are not anywhere near regular, they're random and they seem more frequent when lots of assets are in play.

[Henri Beauchamp]

Your statistics floater shows a rate of 2 frames per second only, meaning the viewer executes its main thread (which is also the rendering thread) loop in 500ms: no surprise then that you see a 540ms ”ping time” in the statistics floater: this ”ping time” shall be taken with a grain of salt, really (and here a huge amount of salt), because since it is calculated once per frame, it can never be smaller than the frame rendering time (which actually adds to the real, network ping time); it's okey-ish when you render at 100fps (the error is then of only 10ms), but certainly not at 2fps !!!  If anything, subtract the frame time to the ping time in the statistics floater (here: 0.540 -1/2.1 = 540ms-476ms = 64ms).

Note that your frame rate also affects how fast textures and meshes are downloaded, because here again, the textures and meshes updates are performed, once per frame, in the main thread (even if they download in a child thread), meaning that the slowest the frame rate, the fewer textures and meshes are rezzed (get rendered) per second, and this also affects how fast their required LOD is updated (meaning that when your camera is moving, such as when your avatar is ridding a vehicle, the LODs lag big time behind the importance to the camera of the objects around you, and the viewer will be late to trigger a better texture or mesh LOD download to match the increased importance of the objects that got closer since last frame was rendered). This is also why you shall never, ever use the Vsync (or even adaptive sync) feature with a SL viewer: it would ruin your rezzing time and texture/mesh LOD updates speed; always prefer triple buffering to fight (and suppress) tearing.

As another note, iGPUs (Intel's) or even APUs (AMD) are notably too weak. Use a discrete graphics card instead: even an old GTX460 will perform way better than any existing iGPU/APU !

And finally, once you got a decent enough GPU, the SL viewers rendering engine will see its bottleneck moved to the single core performance of your CPU: the highest the single core performance, the best the frame rate (the frame rate increase becomes linearly proportional with the CPU single core performance increase, once the GPU taken out of the equation).

To summarize: you do not have a network issue, but a frame rate issue: it is because the frame rate ”clears up”, like you say, that your (Stats floater) ”ping time” sees its (inaccurate/flawed) figure going ”back to normal”, not the other way around (the frame rate is actually independent of the network speed) !

Edited March 13, 2022 by Henri Beauchamp

[Suzanna Soyinka]

The frame rate issue happens everywhere and my main PC is a I7-9700 with a 2080 RTX and 64GB of RAM there is no reason I should have a systemic frame issue.

The frame rate drops when the latency increases, and that is what happens every time. Every time I see a latency spike, the frame rate dives down to nothing.

I can get 120 FPS in SL in an empty sim or at high altitude with not much rezzed around me, but even the when the latency spikes up, my frame rate drops.

It is a network issue, its the only factor I do not have direct control of and its the only factor I haven't already been able to personally address.

Edited March 13, 2022 by Suzanna Soyinka

[Suzanna Soyinka]

42 minutes ago, Henri Beauchamp said:

Your statistics floater shows a rate of 2 frames per second only, meaning the viewer executes its main thread (which is also the rendering thread) loop in 500ms: no surprise then that you see a 540ms ”ping time” in the statistics floater: this ”ping time” shall be taken with a grain of salt, really (and here a huge amount of salt), because since it is calculated once per frame, it can never be smaller than the frame rendering time (which actually adds to the real, network ping time); it's okey-ish when you render at 100fps (the error is then of only 10ms), but certainly not at 2fps !!!  If anything, subtract the frame time to the ping time in the statistics floater (here: 0.540 -1/2.1 = 540ms-476ms = 64ms).

Note that your frame rate also affects how fast textures and meshes are downloaded, because here again, the textures and meshes updates are performed, once per frame, in the main thread (even if they download in a child thread), meaning that the slowest the frame rate, the fewer textures and meshes are rezzed (get rendered) per second, and this also affects how fast their required LOD is updated (meaning that when your camera is moving, such as when your avatar is ridding a vehicle, the LODs lag big time behind the importance to the camera of the objects around you, and the viewer will be late to trigger a better texture or mesh LOD download to match the increased importance of the objects that got closer since last frame was rendered). This is also why you shall never, ever use the Vsync (or even adaptive sync) feature with a SL viewer: it would ruin your rezzing time and texture/mesh LOD updates speed; always prefer triple buffering to fight (and suppress) tearing.

As another note, iGPUs (Intel's) or even APUs (AMD) are notably too weak. Use a discrete graphics card instead: even an old GTX460 will perform way better than any existing iGPU/APU !

And finally, once you got a decent enough GPU, the SL viewers rendering engine will see its bottleneck moved to the single core performance of your CPU: the highest the single core performance, the best the frame rate (the frame rate increase becomes linearly proportional with the CPU single core performance increase, once the GPU taken out of the equation).

To summarize: you do not have a network issue, but a frame rate issue: it is because the frame rate ”clears up”, like you say, that your (Stats floater) ”ping time” sees its (inaccurate/flawed) figure going ”back to normal”, not the other way around (the frame rate is actually independent of the network speed) !

This is my normal stats in my building homestead, I have two islands the main one I get around 20 to 60 FPS depending on where I'm at because its a very intense build.

The homestead there I am at about 2000m Z axis, but I am in a fully furnished skybox of mine that isn't made of prim plywood by any means and I run SL on ultra with only Ambient Occulusion and Antialiasing off because I override those settings in my Geforce settings for my viewer because my GPU does it better than SL does it and less expensively.

Also keep in mind I can see that latency spike at this homestead too, I have seen it, and of course then my frame rate dives down into the sub 10 range.

Edited March 13, 2022 by Suzanna Soyinka

[Solar Legion]

Any node issues will have an effect. Meaning the issue is likely between you and the severs.

Meaning it is on neither endpoint.

[Henri Beauchamp]

1 hour ago, Suzanna Soyinka said:

Also keep in mind I can see that latency spike at this homestead too, I have seen it, and of course then my frame rate dives down into the sub 10 range.

You do not seem to understand... The ”latency spikes” are not a network latency at all: the network latency CANNOT reduce your frame rate. If you are not convinced, turn off your network interface and watch the frame rate: it won't go down, and even might go up (because there is no network packet for the viewer to process during the frame render loop) !

The ”ping time” as shown in the Statistics floater is totally bogus and misleading, because it includes the frame rendering time. If you want to measure the true network ping, look at the sim host address in the About floater, and ping that host from a terminal.

You are confusing the cause with the effect !!!

Your problem is a frame rate issue (which, as a result, makes the Statistics floater ”ping” look bad).

As a side note: I highly discourage overriding the viewer renderer settings (AA, ambient occlusion or Vsync) with the drivers settings: this usually leads to conflicts (at best) and sometimes may cause hiccups and ”crashes” (watchdog timeouts).

Edited March 13, 2022 by Henri Beauchamp

[Suzanna Soyinka]

5 hours ago, Henri Beauchamp said:

You do not seem to understand... The ”latency spikes” are not a network latency at all: the network latency CANNOT reduce your frame rate. If you are not convinced, turn off your network interface and watch the frame rate: it won't go down, and even might go up (because there is no network packet for the viewer to process during the frame render loop) !

The ”ping time” as shown in the Statistics floater is totally bogus and misleading, because it includes the frame rendering time. If you want to measure the true network ping, look at the sim host address in the About floater, and ping that host from a terminal.

You are confusing the cause with the effect !!!

Your problem is a frame rate issue (which, as a result, makes the Statistics floater ”ping” look bad).

As a side note: I highly discourage overriding the viewer renderer settings (AA, ambient occlusion or Vsync) with the drivers settings: this usually leads to conflicts (at best) and sometimes may cause hiccups and ”crashes” (watchdog timeouts).

Your statement makes no sense because if I was having frame rate issues they'd be consistent. I only have frame rate issues when my ping also seems to be anywhere from 290 to 600MS and it happens in random cycles and are not triggered by literally anything I am doing in SecondLife at the time.

I know what frame rate issues are but you can't tell me you've looked at my hardware and you're telling me that 2.1 FPS RANDOMLY for no reason when I'm not even moving or manipulating my camera, that not only seems to affect my frame rate but also lags my inventory access/texture rez times and everything else is somehow affected as well including, even, at times, my text input.

And as stated it only seems to happen in the SecondLife client, when its happening I can easily tab over to my Blender or Photoshop tabs, my Chrome tab, whatever, theres no delay on any of those windows there's no lag in any of those apps while SL is performing in this manner. So I know it is not my hardware.

6 hours ago, Solar Legion said:

Any node issues will have an effect. Meaning the issue is likely between you and the severs.

Meaning it is on neither endpoint.

This is also entirely possible as I've been running http services out of SL to a server of my own and I've seen this but my server is not on a class A network service provider. My ISP is a class A network provider, the CDN services SL uses are a class A network provider and my overall latency to the Amazon CDN's is usually 4 times lower, in direct test, than my latency in SL is. I might suspect a routing issue if I was somewhere in like Kalamazoo or whatever, connected via a wireless provider or sat uplink, but this is direct glass fiber via a class A national cable service and my routing literally anywhere else seems fine.

[Henri Beauchamp]

6 minutes ago, Suzanna Soyinka said:

Your statement makes no sense because if I was having frame rate issues they'd be consistent.

I am afraid your conclusions make no sense. If you do not want to hear the truth, then why asking ?  Did you even try to disable your network interface like I suggested, to see by yourself that even with an infinite network lag, your frame rate won't be reduced ?...

Anyway, I won't loose any more my time to try and convince you... Good luck !

[Suzanna Soyinka]

1 minute ago, Henri Beauchamp said:

I am afraid your conclusions make no sense. If you do not want to hear the truth, then why asking ?  Did you even try to disable your network interface like I suggested, to see by yourself that even with an infinite network lag, your frame rate won't be reduced ?...

Anyway, I won't loose any more my time to try and convince you... Good luck !

 

Disabling my network interface would disconnect me from SL and given that the frame rate issues seem to be present when the network appears to be under the most stress obviously I would probably not have any issues because I'd be disconnected from SL entirely. I mean you might as well tell me to try to fix it with Alt-F4 or Ctrl+Q+Q at this point because it just seems like you are trolling.

Why would I just have random, non consistently triggerable frame rate issues, just in SL, when I run Cyberpunk 2077 at a consistent 60 to 90 FPS and never lag or stutter there if the network component wasn't an issue? What's more I can run SecondLife in the background while playing Cyberpunk 2077 and Cyberpunk 2077 doesn't lose enough frames for me to notice while I do.

My conclusions are based on empirical observable evidence. In literally any sim I go to in SL, most especially since around January of 2022, there can be occasional sudden high latency issues, which then affect the frame rate and the entire viewer experience and I had confirmation from Concierge support that network latency CAN affect frame draw rates because literally the entire simulator experience is streamed to your client in real time, so I don't understand your conclusions because your conclusions appear to be again, pointing at my hardware while ignoring all the data about said hardware.

[Solar Legion]

Point blank: What FPS you get in actual games is irrelevant.

Second Life is not Cyberpunk 2077. The assets used therein are pre-optimized and shipped with the game itself. No, moddable games are also different beasts to Second Life.

Do not attempt to compare Apples to Granite.

With as long as you've been around, one would think you'd know and understand this by now.

Edited March 13, 2022 by Solar Legion

[Henri Beauchamp]

10 minutes ago, Suzanna Soyinka said:

Disabling my network interface would disconnect me from SL

No !  There is a 2 minutes timeout before you get disconnected (and if you reenable your network interface in this delay, the viewer will catch up).

You could also run the ping to the sim host in a terminal, like I suggested, and you'd see that it does not vary when you get those lag spikes.

10 minutes ago, Suzanna Soyinka said:

My conclusions are based on empirical observable evidence.

Or rather on flawed reasoning due to the fact you do not know how a SL viewer actually works... I have been developing a TPV for the past 15 years. I think I do know what I am speaking about, unlike you !

'nuff said.

Edited March 13, 2022 by Henri Beauchamp

[Suzanna Soyinka]

52 minutes ago, Solar Legion said:

Point blank: What FPS you get in actual games is irrelevant.

Second Life is not Cyberpunk 2077. The assets used therein are pre-optimized and shipped with the game itself. No, moddable games are also different beasts to Second Life.

Do not attempt to compare Apples to Granite.

With as long as you've been around, one would think you'd know and understand this by now.

Oh I'm aware my point on this that my assets are optimized, I'm not building with freebie textures from a box, or uploading unoptimized meshes that I haven't gone through and simplified to SL polycount standards. I've been designing world spaces in SL since 2006, very successful ones and I am insanely OCD about keeping design standards within SL's best parameters while trying to maximize its overall visuals at the same time.

And largely, in most cases, in my own sims I get a pretty consistent frame rate in all areas of my sim except when these latency spikes occur.

49 minutes ago, Henri Beauchamp said:

No !  There is a 2 minutes timeout before you get disconnected (and if you reenable your network interface in this delay, the viewer will catch up).

You could also run the ping to the sim host in a terminal, like I suggested, and you'd see that it does not vary when you get those lag spikes.

Or rather on flawed reasoning due to the fact you do not know how a SL viewer actually works... I have been developing a TPV for the past 15 years. I think I do know what I am speaking about, unlike you !

'nuff said.

Uhhh I was materially involved in the development of Singularity for a good stretch of time with my development partner Melanie Milland, who ran the Avination opensim grid for around 6 years, I wouldn't call myself a TPV developer but I am familiar with TPVs and how they work. And you seem to continue to ignore the statements I make for what you appear to think you know....which appears erroneous in comparison to my long term observations, and yes there is a 2 minute time out but my point is if I disable the network interface, the latency cannot affect the frame rate and instead of just arguing with you about it the next time I see this latency spike I will do exactly that and I guarantee you my frame rate will then be unaffected at that point because the network will no longer be sending data.

You're not being helpful and just appear to be trolling so I'm going to ask you to go ahead and just stop replying entirely, I'm trying to get actual feedback from others who may be experiencing the same things so I can collect information for a JIRA ticket here.

Edited March 13, 2022 by Suzanna Soyinka

[Aishagain]

Reading all this, including some oddly short-sighted comments from those who know better, is disappointing. 

The spikes in "sim ping" (or as SL Stats would have it ping sim) are directly proportional to the FPS (I'm sure you realise this) since the render time component of the sim ping is inversely proportional to the frame rate the viewer achieves, as Henri points out.

I assume that being an experienced user both of PCs and of SL you have taken the obvious (to me) step of ensuring that your Antivirus programme is operating properly and all your viewer executables and folders and files are suitably whitelisted.

The artefacts you are seeing point to something periodically mopping up processor capacity on your PC and while you might have whitelisted stuff before some update, whether it be your OS, GPU or antivirus itself might have invalidated those entries.  It can't harm to simply whitelist all over again.

Edited March 14, 2022 by Aishagain
clarification

[Jackson Redstar]

one thing Im curious about your OS and recent updates. Im on Win 11 and I swear sometimes after a update I see big performance hits in SL and I don't know why. My 'seat of my pants" gage is my home in SL at 3000m up. Most of the time I am on with no shaders or shadows running and usually fps is 260-330 fps. But then on occasion and usualy after some Win updates, I will see my idle speed be like 120-160 fps. May mean nothing, buit I do wonder if something is going on under the hood in Windows

[Ardy Lay]

I blame ignorance, arrogance, and legislated power saving operating system changes.