Why is UDP texture delivery available again?

[Monty Linden]

6 hours ago, Kathrine Jansma said:

@Monty Linden Any plans to run the CDN with Quic/HTTP2/3 instead of old HTTP1.1 Pipelining? It does not serve https:// so the default http/2 stuff obviously does not work yet.

With the newish multithreaded texture decoders the fetching can be a bottleneck, so some more concurrency without head-of-line blocking would be nice to have.

All on our radar but needing time and resources.  Head-of-line problems really aren't a problem and you can read why starting at https://bitbucket.org/lindenlab/viewer/src/master/indra/llcorehttp/README.Linden#lines-593

 

[arabellajones]

23 hours ago, Monty Linden said:

Okay, the bigger problem of Sad-in-the-UK.  128kbps is pretty much insane.  If UDP over a sea cable to a throttled simulator on the west coast of the US is faster than an unthrottled, local (though possibly unpopulated) HTTP cache something is fundamentally wrong.  HTTP from Svalbard would be faster.

User changed network provider and problem persisted.  Well, if we trust that one provider is not simply reselling services from the other, this tends to point to a local problem.  ISP change usually changes the CDN Point-of-Presence and routing to same.  Still the same CDN supplier but a good piece of the final hops tends to change.

 

More a general point, but most of the UK's internet uses the Openreach (subsidiary of British Telecom) physical network, and that has a couple of odd features. One if that most exchanges (the local switching nodes, telephone and IP) are one internet hop from every other. So it doesn't much matter where in the country you are. Ping times to the CDN node I use are around 13ms. I did some runs with Speedtest, all under 25ms, and the UK is effectively one rather small fuzzy blob which confuses internet location tracking.

We do have physically independent networks used by some ISPs for some physical locations. Kcom was the local telephone company for Kingston-upon-Hull, essentially still is. For most of us, the Openreach physical network is all there is, whoever has put their name on the bill, and the set up seems better regulated than anything in the USA. BT owns Openreach, provides their own ISP, but isn't allowed privileged pricing or access.

While it can get a bit geeky, it's not hard to find an IP address to get DNS from elsewhere than your ISP. It may be tricky changing that setting on the black box your ISP provides, but it doesn't depend on more extreme solutions such as VPNs and Pi-Hole. Where it might be a significant barrier is if you're getting cable TV and Internet through the same connection. (This paragraph may not be limited to the UK.)

 

 

[Ardy Lay]

Given @Monty Linden's advice, some more analysis was performed.  The problem was found to be in a local system, not the CDN or elsewhere.  When the HTTP transfers from the CDN were flowing, the software analyzing the contents for threats was unfamiliar with the JPEG-2000 format and giving it the full treatment, using up a lot of CPU time, which, with concurrent transfers, exhausted the host's resources, slowing the transfers drastically.  The operator of the system has applied an exception rule to prevent this unnecessary analysis and now the end user sees 40Mbps of flows from CDN for a while, tapering off as the viewer's request density tapers off.

Sorry to cause such a kerfuffle with this bit of troubleshooting and thanks for the answer to my original question, Monty! 

[Monty Linden]

3 hours ago, Ardy Lay said:

When the HTTP transfers from the CDN were flowing, the software analyzing the contents for threats was unfamiliar with the JPEG-2000 format and giving it the full treatment

Excellent outcome!  Can you share the name of the anti-virus package?

[Ardy Lay]

6 hours ago, Monty Linden said:

Excellent outcome!  Can you share the name of the anti-virus package?

Sure can.  The target machine is running Windows 10 and using Microsoft's security software.  I use the same with no ill effects and have added the same exceptions to seek improvement but noticed none at all on my Intel i9-9900k.  We are, however, running on vastly different hardware and the other party was also running ESO, which I think is Elder Scrolls Online.

This experience reinforces my opinion on why some people have very bad experiences with Windows 10 on their computers.  I can replicate these painful symptoms on some older computers I have otherwise retired.  (Intel Q6600 and Intel W3550) They 'run' Windows 10 but chug for significant durations after updates to malware signatures and detection heuristics.  I suspect that, if I were to try running SL on them I would also get the choked behavior during HTTP texture downloads if I do not exempt them from analysis.  I know I do see them hit very high CPU utilization when downloading files to them and that load is from either a Windows Defender process or a Malicious Software Removal Tool process.

Edited July 26, 2021 by Ardy Lay

[Coffee Pancake]

2 hours ago, Ardy Lay said:

I know I do see them hit very high CPU utilization when downloading files to them and that load is from either a Windows Defender process or a Malicious Software Removal Tool process.

This is sadly expected behavior and the results observed depend heavily on the PC's I/O performance (chipset & drives).

FS have a wiki page listing how to whitelist the cache for this reason (and this works for all viewer) not just theirs.

https://wiki.firestormviewer.org/antivirus_whitelisting

[arabellajones]

The internet generally is overloaded with rubbish, such as advertising, malware, and tools to track you. At least the CDN for SL content is pretty safe. I am a bit less confident about some of the the games and tools in SL, which depend on an external server. There have been cases where these were abusive. There are some that are OK. It's not always easy to tell which are which. I remember past incidents, both Viewers and add-ons, which were very serious, and there have been cases, not SL-related, involving such things as web browsers.

Even open-source isn't a guarantee. How many people have the knowledge to check the code, and how many of them will bother to look?

Some people, I trust. Some have blind spots. Some, I don't have a long enough bargepole to not touch them with. And the latter class, the Lindens don't seem to like it if you name them. The biggest problem of today's internet is "Quis custodiet ipsos custodes?"

[Coffee Pancake]

8 hours ago, arabellajones said:

The internet generally is overloaded with rubbish, such as advertising, malware, and tools to track you. At least the CDN for SL content is pretty safe. I am a bit less confident about some of the the games and tools in SL, which depend on an external server. There have been cases where these were abusive. There are some that are OK. It's not always easy to tell which are which. I remember past incidents, both Viewers and add-ons, which were very serious, and there have been cases, not SL-related, involving such things as web browsers.

Even open-source isn't a guarantee. How many people have the knowledge to check the code, and how many of them will bother to look?

Some people, I trust. Some have blind spots. Some, I don't have a long enough bargepole to not touch them with. And the latter class, the Lindens don't seem to like it if you name them. The biggest problem of today's internet is "Quis custodiet ipsos custodes?"

An aggressive antivirus scanning everything will protect you from non of these things.

[Ardy Lay]

On 7/24/2021 at 1:11 AM, Monty Linden said:

That said, I have a story from a recent adventure.  CDN issues were reported involving Ukrainian users, VPNs, Akamai, and some other stuff.  One of the things that showed up was a certain Ukrainian residential ISP was providing DNS and other services, as expected.  However, their DNS was hijacking requests for certain Akamai DNS names and returning IPs for their own hosts.  For whatever reason, they had set up their own CDN in front of Akamai.  The performance of this Potemkin Village of a CDN was on the order of what the user is experiencing.  *Many* seconds for certain requests to even start or just fail.  Never trust your ISP.

Nortel Networks tried to sell to me a product specifically for modifying web pages in-transit to replace existing ads, insert additional ads, and do even more unthinkable hooliganism.  The CEO was interested, but I, the CTO, told him to not even think about it or the company WOULD fail.  We were busy building a Fiber to the Home network in 2001.  We really could not afford to alienate subscribers in an environment where people thought 1Mbps was more than they could ever want and we were offering that as the absolute bottom tier.  Looks like I am still here, in the same building, still at the core of the network like a good little packet pusher.  I have and will continue to defend the subscriber's ability to use which ever DNS resolver(s) they want, while providing a pair of on-net resolvers that are 8ms near or better, to each served region.  So, as you can imagine, the CDN hijacking you described disturbs me.  What's to stop any of the publicly accessible DNS resolver operators out there from doing the same, or worse?  What would be their motivation?  No, scratch that.  Somebody will reply with their political rantings.  Could the Second Life Viewer, and derivative works, have some sort of CDN validation mechanism?  I know that anycast-IP routing can be used to hijack CDN queries without modifying DNS queries or results.  Could a method be employed to thwart such efforts?  What would be the desired action in the viewer when such hooliganism is detected?

[Monty Linden]

On 7/27/2021 at 2:06 PM, Ardy Lay said:

Could the Second Life Viewer, and derivative works, have some sort of CDN validation mechanism?  I know that anycast-IP routing can be used to hijack CDN queries without modifying DNS queries or results.  Could a method be employed to thwart such efforts?  What would be the desired action in the viewer when such hooliganism is detected?

End-to-end encryption but you have to be willing to pay for it.