Downloading the cache

[EliteData Maximus]

excuse my 7 year absence.

wouldnt it be great if those of us with Terabytes of storage could download all or selected mainland/private/world/sim caches that have not already been cached to disk, during the times we are not active on Second Life ?

this would greatly improve the experience when exploring the Second Life world.

perhaps this could be an added feature for premium users at a small or no additional cost ?

perhaps an option to download the caches for mainland only or private sims ?

now if i had the entire cache for all the mainland sims (LL owned), exploring them would be pie since the hit ratio for the cache would be significantly higher only necessitating the need to update avatars and modified objects/textures stored in the cache instead of having to download them during active use of the viewer.

[Nova Convair]

... and you think traffic is completely free and if not others should pay that for you yes? (small fee hmmm)

Sounds like the idea to download the internet - just a few dimensions lower.

A cache stores only stuff that you have to download anyways and optimizes traffic in a way that you don't have to download the same stuff over and over again. Perfect for a constantly changing environment like SL.

Another observation is that LL invented the worlds slowest cache. In my case there is nearly no difference in downloading vs loading from cache. That is if the cache is on a HD. If the cache is on a SSD that makes a significant difference. I am pretty sure that the viewer cache will fail miserably at a multi terabyte size, not to mention that ssd arrays of this size are not so common.

 

[EliteData Maximus]

the specs of my (cable) internet: 100Mbit down, 35Mbit up.
my tests seem to indicate asset downloading (that has not already been downloaded) from any region doesnt seem to exceed 1Mbit/s maximum at the server level.
its my opinion that region asset downloading to the viewer, limited at 1Mbit/s, hinders the Second Life experience because you have to wait for most assets to download and render before you can actually explore the region.
1Mbit/s speed would be a valid workable speed 10 years ago when most home internet connections did not exceed 6Mbit/s but fast forward to the present where 1GBit/s connections are available for the home consumer such as Fiber Optic.
my disk cache access, read and write times are exceptional, that is a non-issue, the problem is downloading assets to the cache that have not already been downloaded, the speed from the asset server is terrible.
when i enter a new region to explore, i generally sit and wait for about 3 minutes for about 90% of the assets to have downloaded to my disk cache before i begin to explore the region, thats a long wait.
i keep my far clip (draw distance) at 128m, imagine the asset download time if the far clip was set to 512m ?
Linden Research can upgrade and update their code for the regions and viewers and their hardware that supports the regions but how about the connection speed for downloading assets to the viewer ?
as far as i can remember, the asset server speed was always at 1Mbit.
i never thought to question the speed because the last time i was on Second Life 7 years ago, 1Mbit/s speed for the asset server seemed appropriate at the time.
i think 10Mbit/s would be a million times better than 1Mbit/s for todays standards, heck even 5Mbit/s is better than 1Mbit/s.

[Nova Convair]

Nowadays content is distributed over a CDN (Content Distribution Network) and is by far not limited to 1 MBit. It can make use of a 100 MBit line.

If you don't use an outdated viewer or blocked CDN in the settings I don't know why your connection to the CDN is that bad - if there is any. Thats not expected behaviour. So thats your real problem - not the caching.

A little Detail: Have you ever calculated how long you download 1 TB at a transfer speed of 1 MBit? Over 100 Days.

[EliteData Maximus]

i could make use of wireshark and TCP/UDP connection list and reverse DNS and traceroute the true identity and location of all relevant connections to any specific server that retrieves content for assets such as textures, objects, sounds, etc.

i could then narrow down the issue to a specific server in a specific location by measuring the downstream speed of that particular server.

i am currently using the Firestorm client set to "Phoenix", the latest version.

the bandwidth settings within the client is really just a simplified setting for the amount of multiple simultaneous connections the client can make to any given server for asset retrieval.
there are network speed throttle settings within the debug menu that also affect upload and download connection speeds as well but despite changing any of these settings inside the menu and debug menu, the network speed observed inside the statistics window of the client still shows a maximum speed of 1Mbit/s (changing texture retrieval from UDP to TCP makes no difference)

its my opinion and thought, despite any additional detailed testing that i may perform, the results i will receive will be indicative of a server that is speed limited to 1Mbit/s, a latency issue in the traceroute path or a peering issue.
i have measured the same instances with the latest official stable release of the SL client as well.
now i dont doubt that this could be a peering/latecny issue between my ISP and CDN servers and/or LL's servers, the statistics settings does not measure the latency of independent connections, only as a whole measurement for latency and packet loss.

average packet loss measured in the statistics window is less than 1%, latency averages 300mS, after complete loading of 128m far clip area of a region, typically less than 5 minutes.

it would be nice to see what others network speed is observed through the statistics menu.

ill get around to performing more detailed tests to try and pin down the actual issue though im very confident the issue is not my network and client, the issue is either a peering issue between the CDN/LL and my ISP or that one or more servers are simply throttling their outbound speed to a single IP despite multiple connections (its common in the IT industry to do this)

i also dont see any method to change the VFS size for the cache either (similar to virtual memory), it appears to be hard-coded to 1GB.

since i have visited many regions over the weekend, their assets have been saved to the cache, re-entering those regions provides instant load-up of those assets with extremely minimal load time, current cache size is 4.1GB, only changed > objects, sounds, textures and avatars are refreshed, non changed assets are retrieved from the cache.

at the current time, i have the viewer configured to draw "grey" outlines of unloaded assets, its a far cry from the "missing image" days.

 

[Callum Meriman]

---

EliteData Maximus wrote:

wouldnt it be great if those of us with Terabytes of storage could download all or selected mainland/private/world/sim caches that have not already been cached to disk, during the times we are not active on Second Life ?

---

 

Sadly the VFS file system the lab invented to be a cache is one of the world's worst implementations of an LRU cache. It fails after about 2GB of size, becoming exponentially slower, and also buggier to the point the whole index is broken.

We can dream of an efficient LRU cache, but sadly the Lab are putting too many of their programmers onto the white elephant of Sansar.

 

In terms of performance, the textures should stream as fast as you can blink. Turn your bandwidth DOWN to 500 to stop useless UDP traffic flooding the texture loads (which are pipelined TCP from a coral CDN), and try again.

Most people crank the bandwidth up, but that just slows down textures and prioritises the reporting of where objects are. Bandwidth ios a setting where you make it as SMALL as possible for the best performance.

The other thing to reduce is cache. A 1GB cache will perform better than a 7GB cache. Substantially better.

 If you do this, then you get textures very fast now.

 

---

EliteData Maximus wrote:

it would be nice to see what others network speed is observed through the statistics menu.

 

---

Again, that is just the UDP data. It is not the important multiple pipelined TCP/IP channels that are doing the work.

You want your network speed to be as absolutely low as possible, but still let you walk. A setting of 500 for a 100Mb connection is about the right sweet spot.

 

---

EliteData Maximus wrote:

ill get around to performing more detailed tests to try and pin down the actual issue though im very confident the issue is not my network and client, the issue is either a peering issue between the CDN/LL and my ISP or that one or more servers are simply throttling their outbound speed to a single IP despite multiple connections (its common in the IT industry to do this)

 

---

There are a number of debug settings to allow you to adjust the number of pipelines you have open,and the depth of each pipeline. The CDNs are not throttled. But you must reduce your UDP traffic, hence the lowest possible value on the bandwidth slider you can walk with.

I wont go into the many debug settings because the Firestorm team have optomised them very well. For me, there is no load time. Textures are just there.

---

EliteData Maximus wrote:

i also dont see any method to change the VFS size for the cache either (similar to virtual memory), it appears to be hard-coded to 1GB. 

---

Its on the netowrk tab. It can be very large, but as mentioned 1GB is the sweet spot. You only need enough for a 1024M or less arc in front of your camera. Oh, speaking of camera, keep it to 128meters to start and slowly increase after you test the other things.

A cache of any bigger than about 1-2GB and there is a huge drop in performance.

You are welcome to waste time and try of course.

[Monty Linden]

---

Callum Meriman wrote:

---

EliteData Maximus wrote:

wouldnt it be great if those of us with Terabytes of storage could download all or selected mainland/private/world/sim caches that have not already been cached to disk, during the times we are not active on Second Life ?

---

Sadly the VFS file system the lab invented to be a cache is one of the world's worst implementations of an LRU cache. It fails after about 2GB of size, becoming exponentially slower, and also buggier to the point the whole index is broken.

We can dream of an efficient LRU cache, but sadly the Lab are putting too many of their programmers onto the white elephant of Sansar.

---

Or...  It could be that Sansar provides secondary duty as a rapid-cycle development testbed to proof out ideas that could then be re-implemented in SL with fewer iterations.

Various cache aggregation schemes have been mooted internally as long as I've been here.  And Sansar is actually demonstrating that something other than a global, dynamic, "LRU" cache might be a better experience in some cases.  But there are some SL-specific aspects that might be attacked first with greater impact:

• Too many styles of caches.  Nearly everything that caches does so in a unique and special (and buggy) way.
• VFS system is limited in size and has some performance and serialization issues.
• VFS system solves a problem (poor native filesystem performance for many small files) that doesn't exist on supported platforms in this decade.
• Image cache (PNG, textures) has an index that limits the size of cached textures.  This may also be a source of silent and deadly bugs.
• Processing of cached data to ready-to-wear data slows viewer.
• Caching of post-processed data might improve experience.
• Eviction logic might be better.
• Inventory caching could use a review.

There's stuff in the above that could be a nice win for a TPV...