Ardy Lay Posted July 19, 2021 Share Posted July 19, 2021 I was trying to help somebody troubleshoot their viewer performance when I noticed they were getting textures via UDP. I thought that was disabled some time ago. 1 1 Link to comment Share on other sites More sharing options...
animats Posted July 20, 2021 Share Posted July 20, 2021 That's worth bringing up at Server User Group on Tuesday at noon. 1 Link to comment Share on other sites More sharing options...
Ardy Lay Posted July 20, 2021 Author Share Posted July 20, 2021 5 minutes ago, animats said: That's worth bringing up at Server User Group on Tuesday at noon. I have a real job. Can't be there. Link to comment Share on other sites More sharing options...
Qie Niangao Posted July 20, 2021 Share Posted July 20, 2021 @animatsasked. Seems to have aroused some Linden interest: [extraneous chatter elided] Quote [2021/07/20 12:41] Joe Magarac (animats): Someone asked on the forums: they found a viewer supposedly fetching assets over UDP., Is that even possible?[...][2021/07/20 12:42] Qie Niangao: (That UDP textures question was post https://community.secondlife.com/forums/topic/474731-why-is-udp-texture-delivery-available-again/ ) [2021/07/20 12:42] URL Reader HUD: https://community.secondlife.com/forums/topic/474731-why-is-udp-texture-delivery-available-again/ = "<nolink>Why is UDP texture delivery available again? - Second Life Server - Second Life Community</nolink>"[2021/07/20 12:44] Rider Linden: I saw that post. It should not be possible and the post was rather light on details.[2021/07/20 12:44] Monty Linden: Hmm, texture delivery was to be blocked for UDP but not certain that happened. They won't enjoy it. Would like a capture of a shift-ctrl-3 screen.[2021/07/20 12:45] Monty Linden: texture asset delivery via simulator is one of our most pesimal systems[2021/07/20 12:46] Rider Linden: I'd certainly need more information on what they were seeing/doing when this happened.[2021/07/20 12:47] Joe Magarac (animats): I'm just mentioning that issue from forums, not reporting it.[2021/07/20 12:48] Rider Linden: Thanks Joe.[2021/07/20 12:48] Monty Linden: tnx - will check on the udp status - that's curious 1 Link to comment Share on other sites More sharing options...
Ardy Lay Posted July 21, 2021 Author Share Posted July 21, 2021 I was using Second Life Release 6.4.21.561414 (64bit). "... found a viewer ..." was not from me. I was using a recent viewer from Linden Lab. Did the same on a previous release too. Second Life Viewer certainly can still use UDP. I was researching this because a user in Europe stated they were using UDP because "UDP is faster than HTTP here", and, indeed, in their instance it is! Evil, I know. CDN is failing them miserably. They are on an access provider called Virgin Media, in UK. They said they had really good connectivity to the CDN node the SL viewer was using but the CDN node was delivering at a paltry rate so they switched off HTTP Textures in the viewer and found UDP to be functional and slow, but much better than HTTP for them. They and I were both receiving textures via UDP at about 2 Megabits per second. They were receiving textures via HTTP at about 128 Kilobits per second while I was receiving textures via HTTP at about 54 Megabits per second. I am in USA. I suspect my connection details are irrelevant. So we have two surprises. HTTP texture delivery rate via a CDN node used by some residents isn't always what we expect, and UDP texture delivery seems to be possible again/still. The first is obviously alarming but is the second even an issue? Maybe it causes no harm. In this instance, UDP texture delivery makes a person's use of Second Life possible. @Rider Linden @Monty Linden I am asking the UK resident what viewer they use. 1 1 Link to comment Share on other sites More sharing options...
Coffee Pancake Posted July 21, 2021 Share Posted July 21, 2021 57 minutes ago, Ardy Lay said: They are on an access provider called Virgin Media, in UK. Virgin are, and always have been, pretty p... poor, especially for bandwidth and domestic hardware. I'm going out on a limb here and say, it's not the CDN. Might as well use BT, they own most of the physical infrastructure one way or another and lease it out to other providers, and if you ever have problems (like rubbish speed) might as well be dealing with the (totally not a monopoly anymore, really, not even a little bit) company that (kinda sorta really does actually) own the wires and all the engineers. Especially if you're not living in a major city. (In the UK - it's almost impossible for a non Openreach (cough BT) provider to get there own service to your door, most of the local infrastructure is buried under the roads, not on poles like it is in the US) Before I brexited, did experiment with other ISPs and while it was a little cheaper, the moment there were any problems that required more than tech support telling me to switch it off and on again, it became a nightmare (because they didn't want to have to pay BT to send someone out to do line tests etc). The moment I did switch, I had a BT engineer at my house poking things and swapping my router out for a better one.. WOOSH ! Link to comment Share on other sites More sharing options...
Ardy Lay Posted July 21, 2021 Author Share Posted July 21, 2021 2 hours ago, Ardy Lay said: I am asking the UK resident what viewer they use. Catznip R12.3 on Win 10 20H2 I really doubt it's a viewer thing. Oh, and they had exactly the same problem on BT before they switched to Virgin. Link to comment Share on other sites More sharing options...
Profaitchikenz Haiku Posted July 21, 2021 Share Posted July 21, 2021 10 hours ago, Coffee Pancake said: the moment there were any problems that required more than tech support telling me to switch it off and on again, it became a nightmare This is progress. Back in the dial-up days the first thing they would ask you to do was "re-install Windows TCPIP"" "But, I'm on Linux" "We don't support Linux" 1 Link to comment Share on other sites More sharing options...
Profaitchikenz Haiku Posted July 21, 2021 Share Posted July 21, 2021 (edited) 11 hours ago, Ardy Lay said: HTTP texture delivery rate via a CDN node used by some residents isn't always what we expect, and UDP texture delivery seems to be possible again/still. My immediate thoughts here were the ISP was bandwidth throttling them, HTTP was being targetted because it is how almost all web-content is delivered, UDP was not being affected because it' not used for the same thing? Back when Oz was still attending the server user group meetings I queried something about UDP once, a few months after it had been turned off for things like inventory fetching and clothing (and the mobile client). When asked why UDP was still around after LL had supposedly killed it off he responded along the lines of "We're still using UDP, only now just for the things it's meant to be used for". Coffee mentioned something in another thread about UDP being how the region gets essential data to the client, I suspect this is what he was referring to. Edited July 21, 2021 by Profaitchikenz Haiku Link to comment Share on other sites More sharing options...
Aishagain Posted July 21, 2021 Share Posted July 21, 2021 This, while it ought to surprise me, does not. I accessed SL via Virgin Media for many years (2008 to 2016) and suffered a number of performance issues. Strangely texture delivery was not the worst. In 2016 I moved and was obliged to change ISPs, to BT in fact, since Virgin did not supply my new location. I was surprised to find that despite a substantial bandwidth loss, I had much better delivery and lower ping to SL. Once we became reliant on http texture delivery, I noticed a distinct slowing of texture loading after an initial rapid resolving of textures (presumably those already cached) which I attribute to poor CDN performance, since my modem is one of the best available performers on texture data delivery. All that I read here points to aspects of data delivery that Linden Lab and their partners are both unaware of and unable ( I hesitate to say incompetent) to control. 1 Link to comment Share on other sites More sharing options...
Coffee Pancake Posted July 21, 2021 Share Posted July 21, 2021 13 hours ago, Ardy Lay said: Catznip R12.3 on Win 10 20H2 I really doubt it's a viewer thing. We (and I presume other TPV's) avoid tinkering with code like that, all viewers will likely be using the exact same code as provided by LL. 13 hours ago, Ardy Lay said: Oh, and they had exactly the same problem on BT before they switched to Virgin. Did they get a new router when they switched? Are they using an ISP provided one (if so model number) ? It's either the router (old / cheap / junk) , they are being throttled (likely happening anyway) or there is a problem with the wires (good luck !) Link to comment Share on other sites More sharing options...
Henri Beauchamp Posted July 22, 2021 Share Posted July 22, 2021 Strange, really... Perhaps the UDP service got accidentally re-enabled in the sim servers during the AWS migration... As for viewers, the Cool VL Viewer is still able to fetch textures via UDP (as a fallback to HTTP), but, after LL shut down UDP texture serving (which happened quite some time ago) only on OpenSim grids (i.e. the UDP fetch retry code is disabled while logged in SL). However, if someone found a really old release of my viewer, they might be able to try and fetch textures with UDP (there was even a switch to disable HTTP fetching in even older releases, dating back from the time when HTTP fetching was still experimental). The same is true of other TPVs that do not have a kill-switch (no forced update & Co), so I suppose it is possible to still find a viewer nowadays that could do that (but at the cost of a sh*tload of missing features, of course). Link to comment Share on other sites More sharing options...
Profaitchikenz Haiku Posted July 22, 2021 Share Posted July 22, 2021 3 hours ago, Henri Beauchamp said: the Cool VL Viewer is still able to fetch textures via UDP (as a fallback to HTTP), Several TPV's offer a tickbox to enable/disable texture fetch via HTTP, and also inventory fetching. I haven't found a reason to turn off http (yet) but it sounds like I ought to go play... 3 hours ago, Henri Beauchamp said: I suppose it is possible to still find a viewer nowadays that could do that (but at the cost of a sh*tload of missing features, of course). If this translates to "I don't have to see the two dozen 1024-sized textures that woman's parading around in" then I for one will not weep. Link to comment Share on other sites More sharing options...
Whirly Fizzle Posted July 22, 2021 Share Posted July 22, 2021 (edited) I just tested on the LL Viewer Second Life Release 6.4.21.561414 (64bit) Setting the debug ImagePipelineUseHTTP to FALSE allows you to still fetch textures over UDP. Just for fun, I went to Lagland region (specifically set up for texture fetch tests) & tested clean cache fetching all textures over UDP & then relogging with clean cache & ImagePipelineUseHTTP set to TRUE to fetch over HTTP. Times taken to fetch all textures on the region:UDP: 23 minutes!HTTP with HttpPipelining disabled: 2 mins 37 seconds.HTTP with HttpPipelining enabled: 1 min 15 secs On my system, fetching textures over UDP is so freaking slow. It's also nice to see how much faster HTTP texture fetch is when pipelining is enabled too. Note: Setting ImagePipelineUseHTTP to FALSE also disables HTTP Pipelining & setting ImagePipelineUseHTTP to TRUE again & relogging does not automatically enable Pipelining. Make sure to set HttpPipelining to TRUE when testing fetching over HTTP Edit to add: Location: UK ISP: TalkTalk (possibly more *****e than Virgin!) Download speed at time of test: 41 Mbps Upload speed at time of test: 10 Mbps Edited July 22, 2021 by Whirly Fizzle 1 5 Link to comment Share on other sites More sharing options...
M Peccable Posted July 22, 2021 Share Posted July 22, 2021 26 minutes ago, Whirly Fizzle said: Times taken to fetch all textures on the region:UDP: 23 minutes! Wow! Brings back memories of the good old days... 1 Link to comment Share on other sites More sharing options...
Coffee Pancake Posted July 23, 2021 Share Posted July 23, 2021 4 hours ago, M Peccable said: Wow! Brings back memories of the good old days... I used to run a dedicated SL proxy cache server for everyone in my house that used SL in the period between the introduction of HTTP textures and the CDN. It was a pain to set up, stored hundreds of GB of textures and was only marginally better than the CDN, just wasn't worth the effort to keep running after that. Link to comment Share on other sites More sharing options...
Henri Beauchamp Posted July 23, 2021 Share Posted July 23, 2021 8 hours ago, Whirly Fizzle said: I just tested on the LL Viewer Second Life Release 6.4.21.561414 (64bit) Setting the debug ImagePipelineUseHTTP to FALSE allows you to still fetch textures over UDP. I find it rather funny (and quite silly) that LL did not even properly change the code of their own viewer to prevent UDP fetching of textures after they shut down the service... Got a ”mCanUseNET = !gIsInSecondLife && mUrl.empty();” line in my viewer to implement this trivial limitation (which would translate to ”mCanUseNET = false;” for LL's viewer, since it won't log in OpenSim grids)... Link to comment Share on other sites More sharing options...
Noelle Delaunay Posted July 23, 2021 Share Posted July 23, 2021 (edited) I've got a question regarding the texture console, I'm not too familiar with it. After setting ImagePipelineUseHTTP and HttpPipelining back to true in Firestorm and relogging I still see some occasional NET states in the texture console. Should that be the case? As far as I undertand the Wiki page says NET indicates UDP. In the screenshot you can see the NET state in the middle, and HTP states on the bottom. Edited July 23, 2021 by Noelle Delaunay Link to comment Share on other sites More sharing options...
Ardy Lay Posted July 23, 2021 Author Share Posted July 23, 2021 This has me wondering, is the logic to fall back to UDP per asset, or does it apply to all assets, once triggered? Link to comment Share on other sites More sharing options...
arabellajones Posted July 23, 2021 Share Posted July 23, 2021 10 hours ago, Coffee Pancake said: I used to run a dedicated SL proxy cache server for everyone in my house that used SL in the period between the introduction of HTTP textures and the CDN. It was a pain to set up, stored hundreds of GB of textures and was only marginally better than the CDN, just wasn't worth the effort to keep running after that. That's mildly interesting. I am trying to remember how the local, "in viewer" caching has changed. I've been running mine on an SSD, and it is going to be a long while before my connection, running 100% on feeding that, clocks up enough writes to need a replacement. I know some of you have a lot faster connection speed than I do, but my rough figure is an hour to fill the cache. I remember descriptions of how to set up an extra local caching layer, Squid comes to mind, but I think that was long before CDN, and connections were much slower. I also saw talk of using a RAM-disk, in days when the viewer-cache was much smaller. I suspect a bigger problem is the creators using 1024-size textures for everything. A lot of the time, the on-screen image is never going to be big enough for all those pixels to make a difference. I know the complexity number is an imperfect measure, but I have good-looking all-mesh avatars and outfits that are still under 20k complexity. I have been known to rant a bit about people who have accused me of lagging a sim just because I am a furry. But enough of that. Maybe the viewer-level caching can be improved. And there is the way texture-caching is done on the GPU card (that's where those big textures can be the killer). But, when you look at cache sizes and connection speeds, where can you get an improvement? That's way above my pay grade. Just make it reliable (and that's where the UDP to HTTP change made a big difference). Link to comment Share on other sites More sharing options...
Coffee Pancake Posted July 23, 2021 Share Posted July 23, 2021 2 hours ago, arabellajones said: I suspect a bigger problem is the creators using 1024-size textures for everything. A lot of the time, the on-screen image is never going to be big enough for all those pixels to make a difference. I know the complexity number is an imperfect measure, but I have good-looking all-mesh avatars and outfits that are still under 20k complexity. I have been known to rant a bit about people who have accused me of lagging a sim just because I am a furry. But enough of that. Maybe the viewer-level caching can be improved. And there is the way texture-caching is done on the GPU card (that's where those big textures can be the killer). But, when you look at cache sizes and connection speeds, where can you get an improvement? That's way above my pay grade. Just make it reliable (and that's where the UDP to HTTP change made a big difference). Big textures aren't what eat up all your VRAM, that's mostly mesh. Make sure your cache folder is set as an exception for whatever AV software you're using and put it on the fastest drive you can .. IO performance has a huge impact. Cache on a ram drive makes SL rez very fast. To be honest, the entire fetch-cache-decode-gpu-render pipeline needs to be rethought as part of any upgrade to vulkan. Link to comment Share on other sites More sharing options...
Lindens Monty Linden Posted July 24, 2021 Lindens Share Posted July 24, 2021 Okay, missed the notifications here and stuff got busy. So some chatter... 1st. Yep, UDP RequestImage service is alive and well on simulators. Went looking for any attempt at scheduling a deprecation and couldn't find it. I'd be mad but someone is finding the fallback useful so I'll be quiet for now. Regardless, this is an *extremely* throttled data path and @Whirly Fizzle's numbers are what I'd expect. You don't want to use this. 2nd. Fallback is an asset-by-asset decision. The comments at the beginning of this code may be of interest: https://bitbucket.org/lindenlab/viewer/src/master/indra/newview/lltexturefetch.cpp 3rd. There are some indirect controls on UDP fallback but I'm not liking what I see in the above. My carefully extracted and maintained locking flags have not been maintained so I'm already distrustful of what I see. @Henri Beauchamp is not wrong. 6 Link to comment Share on other sites More sharing options...
Lindens Monty Linden Posted July 24, 2021 Lindens Share Posted July 24, 2021 Okay, the bigger problem of Sad-in-the-UK. 128kbps is pretty much insane. If UDP over a sea cable to a throttled simulator on the west coast of the US is faster than an unthrottled, local (though possibly unpopulated) HTTP cache something is fundamentally wrong. HTTP from Svalbard would be faster. User changed network provider and problem persisted. Well, if we trust that one provider is not simply reselling services from the other, this tends to point to a local problem. ISP change usually changes the CDN Point-of-Presence and routing to same. Still the same CDN supplier but a good piece of the final hops tends to change. User can do experiments in the browser downloading textures by visiting URLs of the form: http://asset-cdn.glb.agni.lindenlab.com/?texture_id=<uuid> Replace '<uuid>' with suitable texture UUIDs (I don't have a list of large ones at the moment). These won't display in the browser but they will download. They should download faster than 128kbps. There are issues with Pipelining and certain equipment, software, and ISPs. Keep HTTP but disable pipelining along the lines that @Whirly Fizzle showed. Given what has been described so far, I'm still inclined to believe the problem is between user and the ISP. CDN is provided by Akamai and they are the benchmark. (I've also wanted to do experiements with https:// retrieval of assets to defeat all but the most motivated of traffic twiddlers. This would be a good test case for it.) That said, I have a story from a recent adventure. CDN issues were reported involving Ukrainian users, VPNs, Akamai, and some other stuff. One of the things that showed up was a certain Ukrainian residential ISP was providing DNS and other services, as expected. However, their DNS was hijacking requests for certain Akamai DNS names and returning IPs for their own hosts. For whatever reason, they had set up their own CDN in front of Akamai. The performance of this Potemkin Village of a CDN was on the order of what the user is experiencing. *Many* seconds for certain requests to even start or just fail. Never trust your ISP. 6 Link to comment Share on other sites More sharing options...
Kathrine Jansma Posted July 24, 2021 Share Posted July 24, 2021 @Monty Linden Any plans to run the CDN with Quic/HTTP2/3 instead of old HTTP1.1 Pipelining? It does not serve https:// so the default http/2 stuff obviously does not work yet. With the newish multithreaded texture decoders the fetching can be a bottleneck, so some more concurrency without head-of-line blocking would be nice to have. Link to comment Share on other sites More sharing options...
Coffee Pancake Posted July 24, 2021 Share Posted July 24, 2021 10 hours ago, Monty Linden said: That said, I have a story from a recent adventure. CDN issues were reported involving Ukrainian users, VPNs, Akamai, and some other stuff. One of the things that showed up was a certain Ukrainian residential ISP was providing DNS and other services, as expected. However, their DNS was hijacking requests for certain Akamai DNS names and returning IPs for their own hosts. For whatever reason, they had set up their own CDN in front of Akamai. The performance of this Potemkin Village of a CDN was on the order of what the user is experiencing. *Many* seconds for certain requests to even start or just fail. Never trust your ISP. Grab that Raspberry Pi you've been meaning to do something useful with. Install this - https://pi-hole.net/ Set the pi as your local DNS server. Profit. Link to comment Share on other sites More sharing options...
Facebook
Instagram
Twitter
YouTube
Flickr
Recommended Posts
Please take a moment to consider if this thread is worth bumping.
Please sign in to comment
You will be able to leave a comment after signing in
Sign In Now