CDN broken again?

[Whirly Fizzle]

28 minutes ago, Ayesha Askham said:

The report is @ https://jira.secondlife.com/browse/BUG-100683

First thing I noticed from your logs is you are using the same cache location on the LL viewer & the RLV viewer.
It's a really bad idea to share the cache between viewers, especially when one viewer uses OpenJPG and another uses KDU - this will cause texture corruption.
You have both the RLV viewer & the LL viewer using the custom cache location: F:\RLV Cache\

[Ayesha Askham]

@Whirly

That is news to me, it certainly WASN'T shared before!  I shall look at this and correct it promptly!

And fixed. D'oh!  It take it that RLV uses OpenJPG?

 

Edited May 14, 2017 by Ayesha Askham
needs more information

[Whirly Fizzle]

12 minutes ago, Ayesha Askham said:

@Whirly

That is news to me, it certainly WASN'T shared before!  I shall look at this and correct it promptly!

And fixed. D'oh!  It take it that RLV uses OpenJPG?

 

As far as I'm aware yes. Pretty sure only the LL viewer & Firestorm use KDU.  A KDU license is unfortunately pretty expensive.
Go to Help -> About Second Life/viewer name & look at the "J2C Decoder Version" line to see if a viewer uses OpenJPG or KDU.

Your LL viewer session log did look like you will have seen some corrupted textures that session.
Example:
 

2017-05-14T14:24:36Z INFO: LLKDUMessage::put_text: KDU Error: Kakadu Core Error:

2017-05-14T14:24:36Z INFO: LLKDUMessage::put_text: KDU Error: Attempting to access a non-existent resolution level within some tile-component.  Problem almost certainly caused by trying to discard more resolution levels than the number of DWT levels used to compress a tile-component.
2017-05-14T14:24:36Z WARNING: #TextureLLTextureFetchWorker::callbackDecoded: DECODE FAILED: 3947a1e9-c5a1-f99f-67e3-0d5efedefd8c Discard: 6
2017-05-14T14:24:36Z WARNING: #TextureLLTextureFetchWorker::doWork: 3947a1e9-c5a1-f99f-67e3-0d5efedefd8c: Decode of cached file failed (removed), retrying
2017-05-14T14:24:36Z INFO: LLKDUMessage::put_text: KDU Error: Kakadu Core Error:

2017-05-14T14:24:36Z INFO: LLKDUMessage::put_text: KDU Error: Attempting to access a non-existent resolution level within some tile-component.  Problem almost certainly caused by trying to discard more resolution levels than the number of DWT levels used to compress a tile-component.
2017-05-14T14:24:36Z WARNING: #TextureLLTextureFetchWorker::callbackDecoded: DECODE FAILED: 3947a1e9-c5a1-f99f-67e3-0d5efedefd8c Discard: 6
2017-05-14T14:24:36Z WARNING: #APRll_apr_warn_status: APR: The system cannot find the file specified.  
2017-05-14T14:24:36Z WARNING: #APRLLAPRFile::remove:  Attempting to remove filename: F:\RLV Cache\texturecache\3\3947a1e9-c5a1-f99f-67e3-0d5efedefd8c.texture

This is the texture that couldn't be decoded from cache
http://texture-service.agni.lindenlab.com/3947a1e9-c5a1-f99f-67e3-0d5efedefd8c/320x240.jpg/

 

[Ayesha Askham]

Yes, you are quite correct, Whirly and what is more disturbing is that there appears to be widespread ignorance of this issue inworld from what I can gather.

I am surprised and concerned that RLV should use the same Cache location as the SLV. That needs attention, in my view.

Edited May 21, 2017 by Ayesha Askham
addition

[arabellajones]

I don't know just how CDN works, but whatever the CDN servers are doing, they can't know which content you need. The servers have to send you the URL. (I have a vague recollection of this being the reason to use HTTP instead of UDP).

And it's the same whether the asset is in the CDN or in your local cache.

What I have been seeing is sometimes very slow appearance of textures from my local cache. I've been in sim-A, TP to sim-B which loads about as usual, and when I TP back to sim-A, everything is grey for what seems like an eternity before everything starts loading in a rush, with no sign of it coming over the network

It's the same viewer (I can't even get the SL Viewer to load on my Linux system). The cache is 10GB. And It's one continuous login.

So I am a bit doubtful about it being the CDN. 

 

[arabellajones]

On 2017-5-14 at 5:10 PM, Whirly Fizzle said:

As far as I'm aware yes. Pretty sure only the LL viewer & Firestorm use KDU.  A KDU license is unfortunately pretty expensive.
Go to Help -> About Second Life/viewer name & look at the "J2C Decoder Version" line to see if a viewer uses OpenJPG or KDU.

 

I am puzzled. A texture arrives from SL (or the CDN). Why should the file format need changing when it is written to the cache? I can see why a different JPG implementation might write a different version, but if the two versions are incompatible, how can they both be meeting the documented standard? If a file is being changed between leaving SL and leaving the local cache, why?

[Whirly Fizzle]

10 hours ago, Ayesha Askham said:

I am surprised and concerned that RLV should use the same Cache location as the SLV. That needs attention, in my view.

Yes it does.
I honestly thought you'd accidentally set the same custom cache location for the LL viewer & the RLV viewer, but no.
I installed the RLV viewer to check & it's even worse then that.  The RLV viewer is using the same settings files as the LL viewer too. It doesn't created it's own cache folder & will use by default whichever cache location is set for the LL viewer.
That should be reported to Marine.

[Whirly Fizzle]

4 hours ago, arabellajones Resident said:

What I have been seeing is sometimes very slow appearance of textures from my local cache. I've been in sim-A, TP to sim-B which loads about as usual, and when I TP back to sim-A, everything is grey for what seems like an eternity before everything starts loading in a rush, with no sign of it coming over the network

It's the same viewer (I can't even get the SL Viewer to load on my Linux system). The cache is 10GB. And It's one continuous login.

So I am a bit doubtful about it being the CDN. 

If uncached textures are loading quickly but cache textures are being slow to load then I don't see how this can be anything to do with a CDN problem. Something is causing slow cache reads.

Which viewer are you using?
Have you whitelisted the viewer cache folder in your antivirus software?
Which Firewall & antivirus software are you using?

[arabellajones]

14 hours ago, Whirly Fizzle said:

If uncached textures are loading quickly but cache textures are being slow to load then I don't see how this can be anything to do with a CDN problem. Something is causing slow cache reads.

Which viewer are you using?
Have you whitelisted the viewer cache folder in your antivirus software?
Which Firewall & antivirus software are you using?

Just re-read what I wrote.

The viewer has to be talking to the sim server to know which textures to get, whether it's from the CDN or from local cache. If that data is slow to arrive, either because of a problem with the sim server or the internet, there doesn't have to be anything wrong with the CDN or the cache. But blaming them is a bit of a not-my-problem solution.

Since I use Linux, I wouldn't expect you to know anything about the available tools, but it would help to have the cache on a different physical drive. I don't know what tools Windows might have to monitor a particular folder, and I don't know the details of how the viewer uses the cache. It might need some specific debug setting, maybe a special version. No, I am not going to compile my own debug viewer. Just because I use Linux doesn't magically turn me into a geek. But if you want to give me Windows fixes I shall ignore you.

[Ayesha Askham]

On ‎14‎/‎05‎/‎2017 at 4:42 PM, Whirly Fizzle said:

First thing I noticed from your logs is you are using the same cache location on the LL viewer & the RLV viewer.

Just to close out this issue of cache sharing, Marine Kelley has reacted quickly and has issued an updated RLV which no longer shares its default cache with the Linden SLV.  It DOES require that anyone who has previously been using the default viewer to change the cache location from preferences PRIOR to the first login with the new viewer, but once that is done all will be well.

[Whirly Fizzle]

15 minutes ago, Ayesha Askham said:

Just to close out this issue of cache sharing, Marine Kelley has reacted quickly and has issued an updated RLV which no longer shares its default cache with the Linden SLV.  It DOES require that anyone who has previously been using the default viewer to change the cache location from preferences PRIOR to the first login with the new viewer, but once that is done all will be well.

Great!  Thanks for the update Ayesha.

[Monty Linden]

On 5/21/2017 at 7:05 PM, arabellajones Resident said:

I am puzzled. A texture arrives from SL (or the CDN). Why should the file format need changing when it is written to the cache? I can see why a different JPG implementation might write a different version, but if the two versions are incompatible, how can they both be meeting the documented standard? If a file is being changed between leaving SL and leaving the local cache, why?

The cache (or more correctly, the caches, there are many, all different), isn't a simple file cache as with a squid/http cache.  Right or wrong, we implemented a bunch of special little snowflake caches each with its own challenges.  One common one is that the compiled, in-memory structure of objects is often reflected in the cache.  We have an imperfect method of indicating changes between LL viewer releases but none between viewers.  The only solution there is separate caches.  (Extra credit:  discuss 32 versus 64 bit builds.)