Whatever happened to "Project Interesting"?

[animats]

Whatever happened with the "Project Interesting" viewer from 2013? This was supposed to load nearby visible objects first, avoid holes in roads, and such. Yet we still have the common experience of standing in front of a sign and waiting a full minute for the high-rez version to come in.

[Qie Niangao]

My pain point is loading of textures, not objects. AFAIK, the objects load in the "Interesting" order, so nearest ones first (more or less). Texture loading, on the other hand, is now so abysmal that I can't detect any order at all, whereas years ago textures were noticeably prioritized for surfaces under a mouse hover or click.

[Candide LeMay]

1 hour ago, Qie Niangao said:

My pain point is loading of textures, not objects. AFAIK, the objects load in the "Interesting" order, so nearest ones first (more or less). Texture loading, on the other hand, is now so abysmal that I can't detect any order at all, whereas years ago textures were noticeably prioritized for surfaces under a mouse hover or click.

Hasn't the very recent Firestorm improved that? The release notes say "Texture previews opened from inventory will now load much faster" which implies some changes to the texture loading prioritization. Maybe @Beq Janus can tell us.

[animats]

2 hours ago, Qie Niangao said:

Texture loading, on the other hand, is now so abysmal that I can't detect any order at all, whereas years ago textures were noticeably prioritized for surfaces under a mouse hover or click.

Yes. I've made the remark before that throttling at Akamai may be the problem. Akamai's content delivery network is intended to be used for web pages. SL assets are retrieved from those servers as if going to a new region was browsing to a web page with a few thousand images. Akamai may treat that as an overload and throttle bandwidth to the viewer. Akamai says they throttle "intelligently", which probably means if the access pattern doesn't look like a web browser, it gets throttled. I brought this up at Server User Group once, and the LL people hadn't thought about that.

I personally have 1Gb/s networking, and still get 30 second texture loading stalls. There's one thing that seems to work particularly badly - signs. You can walk right up to a sign, then wait, and wait, and wait for the text to load at high detail. Vendors, too. Whatever prioritizes this does not understand that when a texture is filling most of the screen it really ought to be loaded promptly.

[Aishagain]

1 hour ago, animats said:

I've made the remark before that throttling at Akamai may be the problem. Akamai's content delivery network is intended to be used for web pages. SL assets are retrieved from those servers as if going to a new region was browsing to a web page with a few thousand images. Akamai may treat that as an overload and throttle bandwidth to the viewer. Akamai says they throttle "intelligently", which probably means if the access pattern doesn't look like a web browser, it gets throttled.

Yes just this.  I was at the Cyberpunk Fair tonight (UK time) and after a certain number of textures had arrived (quite rapidly) the remaining textures (there were a lot) arrived in a slow dribble and it took 30 minutes before nearly all textures were rezzed, and even then there were a few that still had not changed from blank grey rectangles.

All the ojects had appeared and what I noticed most was that textures attached to avatars appeared to be among the first to rez.

[Ardy Lay]

@animats & @Aishagain I am beginning to suspect some bum CDN components.  Maybe there is a way to test them?  The one I get almost never pulls what you are describing but I have seen it happen.  It was quite jarring when it happened.  I thought "the Internet was down".  Panic set in until I tried some other stuff.  (I work for the ISP that serves my home.  If it's not working, I get to fix it.)

[animats]

3 hours ago, Aishagain said:

after a certain number of textures had arrived (quite rapidly) the remaining textures (there were a lot) arrived in a slow dribble

Yes. Turn on the operating system's graph of network traffic, and then teleport to a region you haven't visited recently. There's lots of traffic for about 10 seconds, and then it slows way down, even though there's more content to be delivered. Something in the asset delivery chain is throttling. Hard to tell what.

[Ardy Lay]

1 hour ago, animats said:

Yes. Turn on the operating system's graph of network traffic, and then teleport to a region you haven't visited recently. There's lots of traffic for about 10 seconds, and then it slows way down, even though there's more content to be delivered. Something in the asset delivery chain is throttling. Hard to tell what.

That is more of a demonstration than a test.  You know it happens.  When I repeat your method, I get textures dumped on me at a rate exceeding 50 Mbps until they have all loaded, except in the very infrequent case when I get something somewhat like you describe.  Sounds like it happens to you more frequently than it happens to me.  I am pondering a repeatable test that could be used more uniformly than what I did, dumping cache and relogging.  Would a test program be suitable or would such a thing violate ToS or be 'bad' in some other way?

curl has not only deprecated HTTP/1.1 pipelining, pipelining has been eliminated from curl as of 7.65.0.  curl enables HTTP/2 multiplexing by default since 7.62.0.  Has the Second Life Viewer been updated to use HTTP/2 and multiplexing?  I don't think so.  Second Life Viewer appears to be using HTTP/1.1 and pipelining.  Could this be part of the issue?  I just don't know.  If your client doesn't support multiplexing and your server doesn't support pipelining, guess what you get.  One-time use TCP sessions.  That's a problem.

[Henri Beauchamp]

1.- The interest list has been implemented (so the project viewer itself has been adopted). It got nothing to do with textures loading, however it changed how objects get rezzed and there have been changes in how objects are cached too, which may cause objects not rezzing after a TP or login: they are ”there” but you need to right-click at their (empty) position to see them pop up into view. The Cool VL Viewer got a workaround for this issue (it refreshes the visibility of all objects in the object list when it receives the parcel info, which happens a few seconds after TP/login; you may also refresh them manually with ALT SHIFT R, a bit like an objects ”rebake”).

2.- The texture fetcher and the texture prioritization algorithm are suboptimal and would deserve a rewrite, from the ground up. In the Cool VL Viewer I recently added a textures fetching boost feature (which also benefits from its new multi-threaded image decoder), which basically forces the refresh of the textures priority way more often (more textures have their fetching priority updated at each frame), causing partly loaded textures in close view to get loaded MUCH faster; the drawback is an impact on the frame rate while the boosting happens. This feature kicks in after login and TP (duration of the boost is configurable), and on demand (CTRL B), with also an optional ”camera speed proportional boost” feature (the faster the camera moves, the higher the boost factor: great when using a vehicle at high speed).

3.- I do not see the issue described here (i.e. the textures network traffic slowing down to a trickle), but it might depend on the CDN serving your region/country. If you are unlucky enough to hit a badly configured CDN, you may change your DNS address for one in a neighboring region/country, which will cause another (pool of) CDN to be used.

4.- The viewers are still using a curl version with HTTP/1.1 (pipelining) enabled. I recently updated the curl version for my viewer (for Linux and macOS ) to a patched v7.64.1 which got HTTP/1.1 re-enabled (via a one-liner patch), so that OpenSSL v1.1 could be used as well (instead of the deprecated v1.0) and suggested to LL to use that combination: they are working on it.

[Qie Niangao]

3 hours ago, Ardy Lay said:

If your client doesn't support multiplexing and your server doesn't support pipelining, guess what you get.  One-time use TCP sessions.  That's a problem.

Hmm. Such a problem could hit different users differently, right? Depending on LAN hardware, client machine configuration, maybe even backing up to the local Internet service. Might this be why some of us starve for textures every damned teleport, and others encounter it once in a blue moon? I'm not network engineer enough to know, but I am a little skeptical of the "luck of the CDN draw" explanation unless there's actual evidence specific to network geography.

[Henri Beauchamp]

28 minutes ago, Qie Niangao said:

Depending on LAN hardware

Maybe if you are using a cheap router (usually seen with WiFi so-called ”routers”) that support only a limited amount of simultaneously opened ports (viewers can use dozens of them, depending on their configuration): you might manage to work around it by reducing the configured numbers of max requests for textures and meshes (see your viewer network preferences or debug settings), but the definitive solution is to replace that cheap router with a true one, that can keep all 65535 ports opened simultaneously !

Quote

Client machine configuration

Maybe, if configured with a local proxy for HTTP connections: it would totally ruin textures fetching, don't do that !  However, such proxies are usually configured only for standard web ports (80, 443, typically) and won't affect SL's HTTP traffic that happens on non-standard ports.

Quote

maybe even backing up to the local Internet service.

Yes, if your computer is behind a firewall using its own proxy for HTTP connections (same as above).

Quote

but I am a little skeptical of the ”luck of the CDN draw” explanation unless there's actual evidence specific to network geography.

Different CDN pools = different administrators. You cannot rule-out local ”optimizations” (that might be great to serve standard web pages but would break SL's usage of HTTP for assets serving, including textures).

It won't surprise me the least if some admins, in some countries/regions would have had a ”great idea” that ruins it for SL.

It's easy to verify whether you are in this case or not, by configuring a different DNS (from a different country).

Edited May 11, 2021 by Henri Beauchamp

[Qie Niangao]

4 hours ago, Henri Beauchamp said:

It's easy to verify whether you are in this case or not, by configuring a different DNS (from a different country).

Are we sure that's how Akamai PoPs get associated with an originating IP? (I mean I tried it anyway, moving from my usual Google DNS server addresses to those Bell Canada is using here, with no observable difference, but even if that landed me on a different PoP, which I question, that still might not be my particular bottleneck, so I can't claim to have performed a real test of anything.)

The "one-time use TCP session" hypothesis wouldn't necessarily result in a huge number of simultaneous sessions if they churn with each request. Staring at TCP Connections (not sessions) in Windows Resource Monitor, it doesn't really look that bad although some stale connections come and go. It appears that the viewer ends up connected to a couple different Akamai endpoints simultaneously, but that's probably completely expected; I never looked at any of this before.

I definitely do see the huge burst of traffic at the start of texture download (e.g., a teleport), tapering off precipitously... but just turning the cam can generate a new burst, so it seems as if the flood of requests from the viewer start being processed all at once but are then starved for some capacity until the next burst. Naively, I can see how this might result from a CDN configuration optimized for web traffic as @animats suggests.

[Henri Beauchamp]

1 hour ago, Qie Niangao said:

@animatsAre we sure that's how Akamai PoPs get associated with an originating IP? suggests.

It depends if the DNS you are using uses the Anycast protocol (which allows to select a server on the basis of the lowest number of 'hops') or not...

I would assume Google's uses it, and you won't see a difference in name resolutions for their CDN server with a local DNS not using Anycast.

One thing is certain, I could convince SL to use different CDN pools (AKA ”PoP”: point of presence) for me by changing DNS, but I never use ”big DNS” servers, and even most often use DNScrypt servers.

Edited May 11, 2021 by Henri Beauchamp

[Kathrine Jansma]

8 hours ago, Henri Beauchamp said:

4.- The viewers are still using a curl version with HTTP/1.1 (pipelining) enabled. I recently updated the curl version for my viewer (for Linux and macOS ) to a patched v7.64.1 which got HTTP/1.1 re-enabled (via a one-liner patch), so that OpenSSL v1.1 could be used as well (instead of the deprecated v1.0) and suggested to LL to use that combination: they are working on it.

Looking at the openssl 1.1 patch i really wonder a bit about compile options: (and really hope it is 1.1.1 as 1.1 is out of support as well...)

             # disable idea cypher per Phoenix's patent concerns (DEV-22827)
             perl Configure "$targetname" no-asm no-idea zlib threads -DNO_WINDOWS_BRAINDEATH \
                 --with-zlib-include="$(cygpath -w "$stage/packages/include/zlib")" \
                 --with-zlib-lib="$(cygpath -w "$stage/packages/lib/release/zlib.lib")"

 

• zlib is kind of "YES, give me CRIME attacks". So most Linux distros disable TLS compression and LL should too, gives a nice CVE (https://nvd.nist.gov/vuln/detail/CVE-2012-4929 ) so better build with no-comp. Even if it works around the lack of compression support on the HTTP layer but well, textures should be highly compressed anyway, and you want HTTP/2 with the efficient Header compression in the end.
• no-asm is also not really a smart thing to do, as it disables AES-NI support which costs a lot of CPU time (see https://www.openssl.org/docs/faq.html Listed at "Does OpenSSL use AES-NI or other speedups?")
• LL could be way more liberal in disabling dead algorithms and unused stuff (e.g. Triple-DES, RC4, and a ton of others you do not need)
• No one needs SSLv3 anymore, and while at it kill TLS 1.0 & TLS 1.1 defaults as well (https://datatracker.ietf.org/doc/html/rfc8996)

 

[Henri Beauchamp]

45 minutes ago, Kathrine Jansma said:

Looking at the openssl 1.1 patch i really wonder a bit about compile options: (and really hope it is 1.1.1 as 1.1 is out of support as well...)

             # disable idea cypher per Phoenix's patent concerns (DEV-22827)
             perl Configure ”$targetname” no-asm no-idea zlib threads -DNO_WINDOWS_BRAINDEATH \
                 --with-zlib-include=”$(cygpath -w ”$stage/packages/include/zlib”)” \
                 --with-zlib-lib=”$(cygpath -w ”$stage/packages/lib/release/zlib.lib”)”

 

• zlib is kind of ”YES, give me CRIME attacks”. So most Linux distros disable TLS compression and LL should too, gives a nice CVE (https://nvd.nist.gov/vuln/detail/CVE-2012-4929 ) so better build with no-comp. Even if it works around the lack of compression support on the HTTP layer but well, textures should be highly compressed anyway, and you want HTTP/2 with the efficient Header compression in the end.
• no-asm is also not really a smart thing to do, as it disables AES-NI support which costs a lot of CPU time (see https://www.openssl.org/docs/faq.html Listed at ”Does OpenSSL use AES-NI or other speedups?”)
• LL could be way more liberal in disabling dead algorithms and unused stuff (e.g. Triple-DES, RC4, and a ton of others you do not need)
• No one needs SSLv3 anymore, and while at it kill TLS 1.0 & TLS 1.1 defaults as well (https://datatracker.ietf.org/doc/html/rfc8996)

 

In my latest build (for Linux and macOS, used in last Cool VL Viewer release), I used  ”zlib threads no-idea”, so nasm is enabled.

”zlib” could probably be dispensed with, and is not used at all for textures (which need to be downloaded via range requests, since a J2C file does not need to be entirely downloaded when you only need its lower LODs, and the viewer does use this property), neither for meshes (which data is already gzipped anyway: it's a llsd+gz encoding).

You are right about SSL (3 and 2), as well as for weak cyphers, but not sure which cyphers are in use by LL, so before disabling a bunch of them, better letting LL do it; plus, as long as LL is not using any such weak protocol or algorithm, it does not hurt (it just makes the libraries a bit larger)...

IPv6 could also be disabled (both at OpenSSL and Curl levels), since SL (and the viewers) only knows about IPv4 and having IPv6 enabled could slow down DNS resolution in some configurations; I am currently testing OpenSSL and Curl builds without IPv6 and they will be used for the next Cool VL Viewer release (at least Linux and macOS ones, if I do not have the time or patience to build them for Windoze).

Edited May 11, 2021 by Henri Beauchamp

[Beq Janus]

On 5/10/2021 at 10:00 PM, Candide LeMay said:

Hasn't the very recent Firestorm improved that? The release notes say "Texture previews opened from inventory will now load much faster" which implies some changes to the texture loading prioritization. Maybe @Beq Janus can tell us.

Coming very late to this. But to answer the question asked to me directly by @Candide LeMay(Henri has answered the original question about "project interesting"), the changes added to Firestorm prioritise textures loaded from inventory specifically, it has nothing (sadly) to do with what is right in front of you, but more "what you have in your handbag". We prioritise images that you are intentionally pulling out of your handbag (inventory) so that clicking textures from your inventory, photos etc, will be prioritised. This includes things like outfit thumbnails in the outfit gallery display, which used to be so slow as to be unusable but is now tolerable for the most part.

This solution makes sense because it is a direct response to an action you have taken, double clicking on the texture or opening the outfits floater. It has a separate priority queue for these because what used to happen was that, in heavy texture regions (and these days that can be pretty much anywhere), the "on-demand" textures like these would be drowned in the flood of trees and grass, and of course the pointlessly oversized textures slapped on nipple rings, and brass aglets on your bootlaces.

 

[Gavin Hird]

In my experience many ISPs inserts Cloudflare between you and the CDN, so that is another hurdle for your content to load. I could very well be that Cloudflare actually throttles.

Usually, when I arrive in SL after having been absent for some days, it takes minutes for a region to even start loading textures on objects, and I suspect a combination of the CDN has discarded the content from cache and has to dig deep in the storage to fetch it, or even replicate from another peer, in addition to Cloudflare starting to fill its caches en route.  

Edited May 21, 2021 by Gavin Hird