Really slow asset loading

[animats]

It took 5 minutes to load this scene. Most of the time the network was almost idle. 45 Mb/s is available.

Texture loading was very, very slow today. I went to New Babbage and buildings were showing with opaque windows for the first two minutes while textures loaded. Above is Robin Loop, which I drive around every few days. This time, the road texture never showed as I drove the 13-region loop. Equally bad with cloud and non-cloud regions.

Are others seeing this?

[Anna Nova]

1 hour ago, animats said:

It took 5 minutes to load this scene. Most of the time the network was almost idle. 45 Mb/s is available.

Texture loading was very, very slow today. I went to New Babbage and buildings were showing with opaque windows for the first two minutes while textures loaded. Above is Robin Loop, which I drive around every few days. This time, the road texture never showed as I drove the 13-region loop. Equally bad with cloud and non-cloud regions.

Are others seeing this?

Yes, it's been getting worse and worse.  Some days I have to wait 10 minutes for everything to appear when I log in.  I had just assumed it was my rural European location, with speed, but long latency, for some reason, but that isn't actually logical.  Maybe it's Linux?

[PekeNL]

I'm seeing the benefits of going to a cloud enviroment already. /s

[Solar Legion]

@Anna Nova: Not likely to be the OS unless there's something quite wrong with a module or configuration.

[Odaks]

Very slow loading indeed, and does seem to have been getting worse recently.

If what @PekeNL says is true for AWS regions, let's hope for a swift and smooth transfer of all regions soon.

[animats]

2 hours ago, Solar Legion said:

@Anna Nova: Not likely to be the OS unless there's something quite wrong with a module or configuration.

Right. The viewer is happy, system load is just the viewer and is low. System memory usage is about 60% on the graph. The viewer is just sitting there waiting for assets to arrive. Viewer is the current Firestorm beta, the one with EEP.

Clicking "edit" on the no-texture areas doesn't bring them up. That normally increases the priority of showing an object, so if it's loaded but not in the GPU due to GPU memory space problems, it gets brought in when you edit it. But there's no texture info the GPU can find.

Speculation: Akamai, which runs the caching system used for SL assets, has detection for overloads, and will throttle requests from an IP address if some threshold is exceeded. Maybe that's it. Akamai's system is tuned to deliver web pages. Teleporting somewhere in SL looks to Akamai like loading a web page with a thousand images or so. That's an abnormal event. A quote from Akamai: "An online video game download consumes as much traffic as about 30,000 web pages, so that's why Akamai is working with companies such as Microsoft and Sony to reduce global Internet traffic during hours of unusually high demand." Due to the US elections, we're probably in a period of unusually high demand, with news pages constantly changing and being reloaded.

I've brought this up at Server User Group, and it's on the list of things to look at after "cloud uplift".

[Solar Legion]

Well there's a more or less major difference: I'm not using the EEP Beta.

[Ardy Lay]

I am not _currently_ experiencing slow asset loading.

[Odaks]

Oh dear! It is so easy to go blamimg LL for malfunctions, especially when we know that they're tinkering with the system.

@animats's theory about Akamai is interesting, and reminds us that there are several other fingers in this pie!

 

I'm in England, so maybe have more potential hiccups than others. I'm using FS (non-EEP).

Edited November 6, 2020 by Odaks
More info

[Mazidox Linden]

There are two potential causes that occur to me immediately: One is as animats says, there maybe just be more traffic globally right now. I know after a long day in the bug mines I've been refreshing the news non-stop.

The other is that we are tracking a bug internally regarding which objects get sent to a viewer when from uplifted regions. I don't believe there's been an external BUG for it yet (the closest we've come to people mentioning it before was a Book Club island where the entire build wouldn't rez unless you right clicked where it should be). If this persists beyond the current flurry of (presumed) internet traffic I would definitely recommend filing a BUG so we can keep an eye on this (hopefully) rare occurrence.

[VenKellie]

On 11/6/2020 at 3:26 PM, Mazidox Linden said:

There are two potential causes that occur to me immediately: One is as animats says, there maybe just be more traffic globally right now. I know after a long day in the bug mines I've been refreshing the news non-stop.

The other is that we are tracking a bug internally regarding which objects get sent to a viewer when from uplifted regions. I don't believe there's been an external BUG for it yet (the closest we've come to people mentioning it before was a Book Club island where the entire build wouldn't rez unless you right clicked where it should be). If this persists beyond the current flurry of (presumed) internet traffic I would definitely recommend filing a BUG so we can keep an eye on this (hopefully) rare occurrence.

its happening to me too but mostly on the old data center regions, uplift regions seem to be near instant loading. I just figure it was just traffic from you guys making backups of regions to send to the cloud.

[animats]

If it's at the Akamai level, whether you see overload throttling depends on where you are and to which cache server you're connected.

Akamai's world of caches.

This is what makes it possible for everyone to view the home page of CNN or Fox at the same time.  It only helps for SL if more than one Second Life user happens to be both in the same area of SL and the same area in RL. Or some of the people in the same area in RL are seeing some of the same items in SL.

Edited November 10, 2020 by animats

[Haruka Mistwood]

I'm definitely having the same issue, when I log into my sim it's generally 5-10 minutes before every texture is loaded

[Ardy Lay]

I must be cheating.  My round trip time to asset-cdn.glb.agni.lindenlab.com and back is 13ms.  Textures usually load in very little time unless the scene is so texture heavy it pushes the discard bias.

[LittleMe Jewell]

I've definitely experienced more and more of this over that past few days.  It actually feels like I'm back in my early days of SL, back in 2007/2008 when pretty much everything loaded slowly all the time.

[SneakyBooger]

Experiencing this also.

[Sharie Criss]

Assets are still loading very slowly. The issue has nothing to do with bandwidth, it's still slow on 1G symmetric fiber. Playing around with the bandwidth slider in the viewer including dropping it quite low is not effective (it's one of the support hints.) The annoying part is that the viewer cache system is just FUBARed completely, Zoom far away, let it load, hit Esc to reset view to close, wait wait wait while everything that had been rezzed (including your own avi) is now gray again and has to load all over as if you just TPed into the region. Even ancient old system library textures are affected.

Time to dust off that old Squid caching proxy to do what the viewer cache should be doing...

LL: You know, if you fix the viewer cache, that will drop your content delivery service costs to a fraction of current as right now, everyone is re-loading the same textures over and over and over and over and over.......

[LittleMe Jewell]

10 minutes ago, Sharie Criss said:

Assets are still loading very slowly. The issue has nothing to do with bandwidth, it's still slow on 1G symmetric fiber. Playing around with the bandwidth slider in the viewer including dropping it quite low is not effective (it's one of the support hints.) The annoying part is that the viewer cache system is just FUBARed completely, Zoom far away, let it load, hit Esc to reset view to close, wait wait wait while everything that had been rezzed (including your own avi) is now gray again and has to load all over as if you just TPed into the region. Even ancient old system library textures are affected.

Time to dust off that old Squid caching proxy to do what the viewer cache should be doing...

LL: You know, if you fix the viewer cache, that will drop your content delivery service costs to a fraction of current as right now, everyone is re-loading the same textures over and over and over and over and over.......

This has been driving me nuts the past few weeks, but last night was super horrid.  I truly felt like I was back in the early days of SL. It has been a long, long time since I'd TP in to a region and have to wait minutes, rather than a few seconds, to see objects and textures.

 

[Chic Aeon]

Agreed. Sometimes it seems like things will never appear. The more complex coalesced seem the most problematic. Simple "one prim" boxes are fine for me.  It can't be the "weight" of the mesh as I have the same issues with my stuff and I make some very low complexity items. 

[Profaitchikenz Haiku]

You're not going to like this, but I have also experienced assets taking a while to load ---

On my own locally-hosted standalone.

It's not bandwidth, it's not the cloud, it's simply the amount of time it takes the server to access lists of objects in the region and the amount of time your viewer takes to go through the cache and texture those it already knows about. In my case, the problem has worsened recently after I pretty much doubled the number of objects in the region. When I took some of the newer builds back to inventory, emptied trash, and cleared cache then spun around re-caching everything, the time taken to redraw the scenes reduced.

[Sharie Criss]

If asset load time hadn't CLEARLY increased dramatically, Everywhere, including my own non-changing skybox, I could go with that explanation, but the issues are beyond that. Asset load times have always been an issue, but what we are taking about (mostly) is that it's worse now than it used to be when everything else is the same. If it was simply efficiencies in the server / viewer as you described, I would expect behavior to be different. Cached textures, once the UUID's are identified, should be immediately loaded from cache. They are not - they start gray, then slowly load and clarify as the textures are progressively loaded from low-rez to the final high-rez version. If the cache is being used properly, you would see the cache textures instantly load rather than progressively.

Again, in my zooming example, the viewer shouldn't be dumping all that scene data prematurely (which I believe is called the Interest List) which forces it to be reloaded. Fix that and everything else gets better.

Whatever the root cause of the recent slowness, it needs to be addressed. Then start working on resolving inefficiencies in the server / viewer that are long-time problems. In SL, it seems like age-old issues are swept under the carpet never to be addressed while working on the next whiz-bang feature. I would really like to see new feature rollout paused while important basic issues - some that span over 14 years - are fixed.

Bottom line, I'm not saying you are wrong, clearly there are issues as assets have Always loaded slower than they should, but I do think the increase in load times is something else.

Edit: Considering how much higher performing modern systems are, with super fast video cards, NVMe drives, modern CPU's, etc. than they used to be along with the viewer updating to 64bit, the increase in scene complexities is mostly compensated for on the client side with the exception of the viewer code itself which is still largely stuck in 2009 outside of cosmetic UI changes and additions for new features. It's time for the entire cache system to get a makeover. Checking for cached data and loading cached data should be a near instantaneous action on modern systems. I should not be waiting over a minute to load my 200 prim skybox in a nearly empty region and low draw distance.

Edited December 2, 2020 by Sharie Criss

[Profaitchikenz Haiku]

14 hours ago, Sharie Criss said:

Cached textures, once the UUID's are identified, should be immediately loaded from cache. They are not - they start gray, then slowly load and clarify as the textures are progressively loaded from low-rez to the final high-rez version. If the cache is being used properly, you would see the cache textures instantly load rather than progressively.

This depends on several things, disk access speed (people have reported sticking the cache on an SSD or Ramdisk gives best performance. Size of textures is a big factor, lots of 1024x1024's take a while to throw around the buses to and from disk/Ram/Cpu/Gpu, and then there's how many of them have to be processed. I don't know what the prioritisation or selection or order is, and watching scenes slowly un-grey themselves I can't spot any obvious order. 

[arabellajones]

Using an SSD does make a big difference to caching, but I have had a number of occasions when the viewer has switched back to the default location, for no obvious reason. So that's the first thing I check when textures are slow to load. I know, it shouldn't happen, but it does. And you can say that about too many things involved with Second Life.

[Nick0678]

19 hours ago, Sharie Criss said:

I should not be waiting over a minute to load my 200 prim skybox in a nearly empty region and low draw distance.

A minute to load a 200 prim skybox? Maybe you should keep a backup and have a look at your SL client settings? It takes about 10 seconds for me to load one of my skyboxes (298 prims) after login.

(Without having any crazy specs , just my 10 years old HP laptop using an i5 460M 2.5Ghz / 6GB Ram and a mechanical 5400 HDD.)

 

[Profaitchikenz Haiku]

3 hours ago, arabellajones said:

but I have had a number of occasions when the viewer has switched back to the default location, for no obvious reason

I have this when I log in on a viewer using an alt that has never logged in on that client before, the cache and chat-log locations get set to default. Once you have corrected the entries in preferences and successfully logged out it should be good for the future. (crash to desktop doesn't seem to save all the details of where you last were, what you had on, ...)