Are others noticing very slow loading of textures the last week or so?

[Jennifer Boyle]

For the last week or so, I have noticed that, very often, textures load very slowly. For example, if I change outfits, it may take a minute or two before all of the textures for the new outfit load. Textures on all manner of objects also are slow to load, and avatars are slow to rez. This is a marked change from the way it was.

Is anyone else noticing this?

[Ardy Lay]

Maybe, just maybe, the content delivery mechanisms that deliver assets to Second Life users are markedly more busy than usual.

Yes.  I have noticed the uncharacteristic delays too.

[Ansariel Hiller]

Are you using the most current release of the Linden Lab viewer (6.6.10)?

[Odaks]

9 hours ago, Jennifer Boyle said:

Textures on all manner of objects also are slow to load, and avatars are slow to rez.

Definitely! That is using Firestorm 6.6.8.68380. Maybe Firestorm has not yet caught up with the very latest wizardry?

[Jennifer Boyle]

7 hours ago, Ansariel Hiller said:

Are you using the most current release of the Linden Lab viewer (6.6.10)?

No. I'm using the most current release of Firestorm.

[Rowan Amore]

Spring break so more internet traffic?

[Arielle Popstar]

I've noticed the same slowness both in S/L and Opensim using the corresponding Firestorm 6.6.10 (68758)  viewers for each.

To me it seems especially bad when changing any BoM layers and tattoo's. Some just seem to drag.

Edited April 5, 2023 by Arielle Popstar
added note

[Aishagain]

I wonder...with the enthusiastic adoption of BoM, the baking service is being more taxed and is seemingly slowing as a consequence.

Is that the phenomenon being experienced?

[Love Zhaoying]

I just rezzed a building and was surprised how slow it was for the textures to load.

I installed the newest official viewer yesterday.

[Janet Voxel]

Slow and a lot of texture thrashing

[Kaleb Nightingale]

Yes I have noticed textures being slow to load. That is all textures, BoM, clothing, objects and avatars rezzing. It does seem to revolve around time of SL day, so when it is midnight and early hours of the morning, SL time, the issue seems worse.

I am using the LL Viewer, Second Life Release 6.6.11.579153 (64bit).

[animats]

On 4/3/2023 at 7:17 PM, Ardy Lay said:

Maybe, just maybe, the content delivery mechanisms that deliver assets to Second Life users are markedly more busy than usual.

Yes.  I have noticed the uncharacteristic delays too.

I've been testing something for my own viewer that involves much file fetching. (I'm trying OpenJPEG, which is a JPEG 2000 decoder that's free but historically buggy. We got some of the bugs fixed.  So I'm running lots of SL content through it, fixing bugs, repeat.)

In terms of stalls, midafternoon 1400 to 1700 SLT, no stalls. Late evening, 2100 SLT and later, no stalls. Early evening, around 1900 SLT, stalls of 1-2 seconds. I'm on gigabit fiber from Sonic.net, and it really is gigabit both ways with no throttling. Akamai's own documents say that they do throttle, but during off-peak periods, the throttle is not very restrictive.

If you're seeing viewer stalls outside of peak periods, it's probably not the asset system. But during peak periods, there are occasional stalls.

File loading test. 44,000 textures, read from 48 threads over gigabit fiber and decompressed on 12 CPUs. 2130 SLT.

Note how flat the network traffic line is.

(This is s stress test for the asset loader. It simulates the worst case, visiting a cluttered area starting from an empty cache with a big draw distance with the LOD factor turned way up. Like a big shopping event.)

[Love Zhaoying]

I am finally integrating / synthesizing the different issues with Mainland vs. Private Regions and the new functionality. Let me know if this sounds correct or if I am wrong in some ways.

a) Since there are LSL functions that allow scanning from other parcels and other nearby regions, merely denying entry into a parcel will not prevent scripted agents from scanning data in a parcel.

b) Unless the new "deny Region entry to Scripted Agents" feature also prevents Scripted Agents from using the same  LSL functions from neighboring regions, denying entry to a Region would not prevent Scripted Agents from collecting data on mainland Regions with the new settings.

c) The new "deny access to Scripted Agents" feature works for Private Regions because there is no way to scan / use the LSL functions that collect data from "Neighboring" Regions, because they are "Private".

d) "Modified viewers" / "scraping" directly from viewers is completely outside of this new functionality, and in fact is completely irrelevant to Scripted Agents.

e) The above a),  b) and c) are why "deny entry to Scripted Agents for Estates" only applies to Private Regions and the entirety of Bellisseria. Bellisseria, being a collective of Regions as a "continent", assumedly does not include the possibility of using the LSL "scanning" functions from neighboring non-Belliseria Reguons. 

[Love Zhaoying]

. Wrong thread

[Ardy Lay]

I suppose this could also be caused by network congestion at any of the links along the paths that various data takes to reach the client.  So many various paths...  Client at the destination can't really determine what the actual path taken by data from the server was.  Using traceroute and the like from the client location only reveals the path from client to server.  I could look at BGPv4 advertisements for the IPv4 address of a CDN node that I get content from but that just shows a path of Autonomous Systems.  Hmm, close enough, I guess.

Hmm, so that BGP trail of Autonomous Systems is changing during the peak hours.  That's going to be disruptive.  But that's the way some Autonomous Systems operate.  I see some discarding routes when the volume of traffic traversing their network causes congestion.  I see others advertising paths they probably should not because better paths are available, but the original advertiser of the better path has prepended their own Autonomous System number several times making hijacking traffic that wants to get to them trivial, then the route hijacker forwards traffic along through the path that pays them the most.  And I see one Autonomous System that has reinvented the router.  They always send replies out via the path upon which they received the queries.  That seems reasonable at first, but the best path client to server is not always the best path from server to client.  The Internet is an asymmetrical mess because some of the companies operating the Autonomous Systems manipulate traffic flow to maximize their own revenue.

Edited April 6, 2023 by Ardy Lay
clarifying some confusing pronouns

[Jennifer Boyle]

Yesterday, textures were loading normally again.

[Profaitchikenz Haiku]

If they're in the cache this shouldn't be an issue., but I have a feeling that what gets sent along the network is enough details about all the textures on the faces for the client to work out if it has it or if it needs to ask for it. It's not easy to work out if the internet is choked or not as a simple speed test only examines your router to ISP link, there are quite a few other bottlenecks between there and the Amazon servers.

 

There are some other threads in other sub-forums that are touching on this and I am now starting to wonder if the number of network requests has somehow increased, because in general the textures on places stay the same, so once they're in cache you shouldn't be needed half the amount of bandwidth to show them.

[animats]

I've been testing in this area, and I've found some things. In about 1 in every 1000 texture asset requests, the request times out. A retry will work. An internal error message from my own program:

2023-04-09T04:05:35.098Z ERROR [jpeg2000_decoder::fetch] Retrying asset url http://asset-cdn.glb.agni.lindenlab.com/?texture_id=00c130f7-15e5-3235-a58b-CENSORED
Error encountered in the status line,
Error: "timed out reading response",
2 of 3 retries left. 

That means the 15 second timer for getting the first response from an HTTP request timed out without getting even the HTTP status line. The data transfer never started. Lost packets alone wouldn't cause that. Lost packets get re-transmitted much sooner than 15 seconds.

Retry has always succeeded on the first retry. So this problem is completely recoverable. Viewers that don't keep a large number of requests in flight might experience a stall when this happens.

I'm trying to work out out the optimal strategy for asset fetching for a multi-threaded viewer. I am using 18 fetch/decode threads. No more than 6 decodes can be in progress, to avoid using up all the CPU time in asset decompression. Thus, there are 12 HTTP requests in progress most of the time. This pulls a steady 220Mb/sec from the asset servers. More concurrent requests (I've tried up to 40) does not increase throughput. Fewer requests cause reduced throughput when the server drops a request. I'm essentially doing a workaround for asset request stalls here.

Not sure how other viewers behave. This might, or might not, be responsible for user-visible stalls. Comments?

(Note this is all about the asset servers. Those are just AWS web servers front-ended by caches. They have plenty of bandwidth. Requests to the sim servers are very different.)

[Jennifer Boyle]

This is way over my head. I am very glad that I was able to stimulate discussion of this issue by people who are so much smarter and more knowledgeable than me. I shall continue to watch with interest and, likely, bewilderment.

[DakotaLeRae]

I am glad it’s not just me. I have noticed it for a few weeks. Even noticed sometimes an object won’t even load I have to jump to another location and jump back to see a object or clothes.

[Kathrine Jansma]

7 hours ago, animats said:

In about 1 in every 1000 texture asset requests, the request times out.

Maybe check if that aligns with the keepalive duration of the HTTP connection?

The server might have killed off the connection and the client failed to notice. Sometimes the connection state can get lost in the byzantine maze of TCP handshakes, TLS handshakes and HTTP keepalive & pipelining.

[RoxanaSaphire]

I have same issue. I tried with FS and SL Viewer... but could not fix it.

When i log in or TP somwehere, body part are flying around for maybe 30 seconds or 1 minute, and then suddenly everything loads.

 

I talked to FS support, and they gave me different advices like clearing cache, whitelisting antivirus and so on. I follwod all, but still same.

Any solution?