Firestorm cache on a solid state drive

[animats]

I recently had to replace the hard drive in my Linux machine, and I took that opportunity to add a medium-sized solid state drive to the rotating hard drive. The solid state drive currently holds Firestorm's caches, and nothing else. As expected, texture loading is much improved. Before, the hard drive was constantly busy. Now it's almost idle when Firestorm is running.

There was an unexpected benefit. Avatar and vehicle turning is much smoother. Much of the time in Firestorm, turning is slower than the frame rate, a lot slower, and may happen in huge jumps. After putting the cache on a solid state drive, that stopped, and Firestorm became more responsive.

The way the cache system works, all the disk I/O is supposed to be done off the main thread, so drawing doesn't wait for the disk. This improvement is a strong indication that somewhere there's an I/O wait in the main thread, or a wait on a lock for another thread doing disk I/O. It would only take one locking bug like that to make this happen.

Anyone set up to profile Firestorm? Look for stalls in the main thread on a machine with a slow hard drive.

[bigmoe Whitfield]

I have 3 ssds in this machine now and one of them I dedicated to just the cache, so much improved.  firestorm lives on one ssd, it's cache on another lol.

[bigmoe Whitfield]

@Whirly Fizzle

[Whirly Fizzle]

I plonked this thread in the Firestorm dev chat.

[Chic Aeon]

Just a note that if you install Firestorm on your SSD (mine is "C") then the cache shows up there automatically. I have had it that way for three years with no issues. All my other programs are there too and it runs wonderfully so not sure that having a SSD "just" for you cache is a giant step up.  Not my strong area though. 

[animats]

2 hours ago, Whirly Fizzle said:

I plonked this thread in the Firestorm dev chat.

Thanks. It's striking how much of an effect this has on response time to arrow keys. That ought to be unrelated.

[Candice LittleBoots]

I have a ASUS ROG laptop with 16GB RAM. C-drive = SSD - D-drive = HDD. All applications, including Firestorm's cache, are on the C-drive.

This thread now has me thinking ..

With the general feeling of speed improvement I experience following a cache deletion, added to the logistics/overkill/expense of adding a second dedicated SSD purely for Firestorm's cache, I am going to set up a RAMDRIVE exclusively for Firestorm's cache instead. No hassle, no costs incurred, making use of resources already onboard.  

Two birds with one stone 
.. or is it?

[KT Kingsley]

If I remember correctly, the downside of a RAM drive was the extra time it takes to save it on shutdown and restore it on startup. This was a while ago, though, and it did go to an HDD when the computer was off: maybe saving to an SSD would be less of a pain. Also, if you only reboot once a week or so, it'll be something you can live with.

[Candice LittleBoots]

23 minutes ago, Candice LittleBoots said:

I have a ASUS ROG laptop with 16GB RAM. C-drive = SSD - D-drive = HDD. All applications, including Firestorm's cache, are on the C-drive.

This thread now has me thinking ..

With the general feeling of speed improvement I experience following a cache deletion, added to the logistics/overkill/expense of adding a second dedicated SSD purely for Firestorm's cache, I am going to set up a RAMDRIVE exclusively for Firestorm's cache instead. No hassle, no costs incurred, making use of resources already onboard.  

Two birds with one stone 
.. or is it?

I just set up a RAMDISK and it runs really well.

I think I'll keep it set up like this for a while and trial it over a week or two.

My RAMDISK will lose all its data upon shutdown/restart which, as mentioned above, is one of the reasons why it appeals to me.

I tend to reboot once per day, or longer in-between.

[Beq Janus]

10 hours ago, Whirly Fizzle said:

I plonked this thread in the Firestorm dev chat.

I've always assumed that it was well established that spinning platters were bad, it stands to reason that this is the case when you are streaming content into the viewer. Years ago I was a videographer on the Designing Worlds show in SL it was painfully clear that onloading and offloading content was a major contributor to rubber banding. Before I could afford SSD I was using a RAM drive (when the viewer was 32 bit) to keep my cache away from the disk to avoid this. Other advice like keep your cache on a separate drive to your system drive so that you are spreading the load across multiple spindles was common knowledge. I guess somewhere along the line such concepts have been lost.

I am sitting on a patch that was submitted to us, that claims (and I say that not because I doubt it works but because I have not tested it in any way) to improve the performance of the cache (on Windows only). I am ignoring it for now because a complete rewrite of the cache subsystem is coming down the line from the lab soon(tm). I do recall a comment from Oz a month or two back when we were discussing how that work was coming along that "in some extreme cases  it was faster to not use cache at all". So there's lots of room for improvement but until the lab land their new implementation it is not worth spending time on because it will all change.

What would be interesting would be to find a measure of "Cache friendliness" that can be applied to scene/region. It is strongly correlated to texture and asset reuse of course. If every time you turn around there's a new set of 1024x1024 textures to pull in (no doubt for an object with exaggerated LODs that in reality you can barely see) your viewer will have to cycle all the new textures from the disk cache into memory.

I should also note though that profiling the viewer when using SSD shows that most of the stalls (more than 50% of your time) is spent waiting on Main RAM so in many cases, while you are speeding up the big delays you are finding yourself up against the RAM speed bottleneck

On 1/20/2019 at 7:17 PM, animats said:

The way the cache system works, all the disk I/O is supposed to be done off the main thread, so drawing doesn't wait for the disk. This improvement is a strong indication that somewhere there's an I/O wait in the main thread, or a wait on a lock for another thread doing disk I/O. It would only take one locking bug like that to make this happen.

As I said, don't waste time on this right now as it is expected to completely change in the near(ish) future. I've had no sight of the new implementation so far though but they've been talking about it for about a year or more, testing various different implementations and so forth. The existing cache implementation has definitely outlived its viability. 

During the Animesh tuning I eliminated a couple of lock conflicts between the threads but those had been mostly introduced during animesh development and were part of the logging subsystem. I have no doubt at all that there are many other dependencies 

 

Edited January 22, 2019 by Beq Janus

[Fionalein]

12 hours ago, Chic Aeon said:

Just a note that if you install Firestorm on your SSD (mine is "C") then the cache shows up there automatically. I have had it that way for three years with no issues. All my other programs are there too and it runs wonderfully so not sure that having a SSD "just" for you cache is a giant step up.  Not my strong area though. 

This thread gave me the idea to get a SSD for cache and SWAP purposes only... I think many old folks just don't trust SSDs yet - I for my part don't. Statistics show me I'm wrong but every one who had a flash card fail before will propably kinda understand my reluctance...

[Chic Aeon]

46 minutes ago, Fionalein said:

This thread gave me the idea to get a SSD for cache and SWAP purposes only... I think many old folks just don't trust SSDs yet - I for my part don't. Statistics show me I'm wrong but every one who had a flash card fail before will propably kinda understand my reluctance...

Well my 3 year old (from what turned out to be a company that turned from great to way below par) SSD just died a few days ago. Power blip that the surge protector didn't protect.  But I run my machines very hard and 3 years is pretty good for me.  I believe SSDs are much more stable now. When I got mine it was a pretty new thing.  Does work great.     And if you are going to lose something -- programs are the best as usually easily installed again (well not if you have as many as "I" do but still easily).  DATA is what you don't want to lose :D.

Edited January 22, 2019 by Chic Aeon

[Fionalein]

2 minutes ago, Chic Aeon said:

 And if you are going to lose something -- programs are the best as usually easily installed again (well not if you have as many as "I" do but still easily).

If you ever had to compile GROMACS localy you would disagree

 

Edited January 22, 2019 by Fionalein

[animats]

8 hours ago, Beq Janus said:

I've always assumed that it was well established that spinning platters were bad, it stands to reason that this is the case when you are streaming content into the viewer. Years ago I was a videographer on the Designing Worlds show in SL it was painfully clear that onloading and offloading content was a major contributor to rubber banding.

In theory it shouldn't cause rubber banding and stalls. That it does is a bug. Fuzzy textures and greyness are legit results from slow texture I/O, but rubber banding is not.

Quote

As I said, don't waste time on this right now as it is expected to completely change in the near(ish) future. I've had no sight of the new implementation so far though but they've been talking about it for about a year or more, testing various different implementations and so forth. The existing cache implementation has definitely outlived its viability. 

You have more faith in LL than I do. But that area does need attention. The design dates from the early days when textures came from the sim servers over UDP connections. Now they come from Amazon AWS via Akamai caches over HTTP connections, and people have more network bandwidth, so the throttling is too strong.

As I've pointed out before, the texture system is quite clever and tries to do the right stuff. The trouble is that its policy on what to do first is spread out through the system and not documented well, so it's tough to tune it for better performance. That's why it has bugs like "you're standing right in front of a sign looking at it and it takes a full minute before it loads". The texture system doesn't prioritize loading the biggest thing in your field of view.

[Fionalein]

2 hours ago, animats said:

Now they come from Amazon AWS via Akamai caches over HTTP connections, and people have more network bandwidth, so the throttling is too strong.

What makes you so sure about that?

[steeljane42]

8 hours ago, Fionalein said:

This thread gave me the idea to get a SSD for cache and SWAP purposes only... I think many old folks just don't trust SSDs yet - I for my part don't. Statistics show me I'm wrong but every one who had a flash card fail before will propably kinda understand my reluctance...

HDDs fail too, though. And not in all cases they give you enough of time to safely save all information as they slowly develop bad sectors and other noticeable issues. Sometimes controller just dies and in best case you'll be able to find exactly same HDD or PCB from exact same model for "donor purposes" and replace it  (or find someone who can do it for you). I did it twice in the past... never again, these days I have 3 copies of backups for all my important data.

Can always start with SSD for games/SL/cache and other random stuff and see how it goes for you, less reasons to worry that way.

44 minutes ago, Fionalein said:

What makes you so sure about that?

Most reports show year to year increase usually. Like here, not the most recent one, but https://www.akamai.com/uk/en/about/news/press/2017-press/akamai-releases-first-quarter-2017-state-of-the-internet-connectivity-report.jsp

From there " Global average connection speed was 7.2 Mbps (an increase of 15% year over year). "

And if compare to 15 or even 10 years ago... it's quite the difference in my city at least. 10 years ago we still had relatively slow (2-5mbps on average) ADSL connections here as the most common ones, these days it's usualy cable or fiber with 100mbps+, plus options to get up to 1gbps if someone actually needs that much. Even my backup mobile connection can reach ~65mbps outside of peak hours (during those it's usually around 30-35mbps), ping is not as good of course and jitter is quite noticeable, so it's not very suitable for games, but for SL it's totally fine.

So it would be great of LL finally started to work/implementing some caching changes along with some rework for SL bandwidth system. FS wiki recommends 1500kbps max, which with current internet speeds is very slow. I suppose the reason for it is how cache works, to not choke viewer with too much data.

[Fionalein]

1 hour ago, steeljane42 said:

 And if compare to 15 or even 10 years ago... it's quite the difference in my city at least. 10 years ago we still had relatively slow (2-5mbps on average) ADSL connections here as the most common ones, these days it's usualy cable or fiber with 100mbps+, plus options to get up to 1gbps if someone actually needs that much. Even my backup mobile connection can reach ~65mbps outside of peak hours (during those it's usually around 30-35mbps), ping is not as good of course and jitter is quite noticeable, so it's not very suitable for games, but for SL it's totally fine.

Well your connection speeds up... but what about those folks that now juts came to SL which did not have any ADSL 10 years ago? Do you think all those Brazilians that recently joined SL have 1GBit internet at their fingertips if they only pay enough? I doubt that. Even in the US you will find regions where you still have nothing like VDSL because there is simply no ISP willing to provide a cable to connect to that region. When the Lab improves their service to faster connection speeds all that will happen is that badly designed content will spread even more as some people will not notice anymore how much bandwidth their cheaply designed gacha trophies really take up. Folks that have no higher connection speeds available will then be lagged so much that they will simply leave SL...

Edited January 23, 2019 by Fionalein

[steeljane42]

46 minutes ago, Fionalein said:

Well your connection speeds up... but what about those folks that now juts came to SL which did not have any ADSL 10 years ago? Do you think all those Brazilians that recently joined SL have 1GBit internet at their fingertips if they only pay enough? I doubt that. Even in the US you will find regions where you still have nothing like VDSL because there is simply no ISP willing to provide a cable to connect to that region. When the Lab improves their service to faster connection speeds all that will happen is that badly designed content will spread even more as some people will not notice anymore how much bandwidth their cheaply designed gacha trophies really take up. Folks that have no higher connection speeds available will then be lagged so much that they will simply leave SL...

I doubt caching improvements will push badly designed content to spread, not more than it already does at least. Because I don't think people who already make (or just "port") said bad content care if their content causes any issues, be it with caching or bandwidth.

And if use that logic, then better not improve SL at all in most cases. Recent animesh might lag some people with slow PCs even more, it's not too heavy from what I've seen, but it's just more mesh and geometry around in general. Upcoming EEP might too (not that sure about that, but it should come with some rendering engine changes I believe, including godrays and some more). And I'm sure when SL just got mesh many years ago, it also forced some people to leave, permanently or temporarily, because their machines couldn't handle mesh.

Most platforms/games designed for average specs in mind, unless it's something aimed for either extremely low or high end. Now what is average in SL these days, PC specs and bandwidth, only LL knows, so we can only speculate. But I would be surprised if LL aimed for the lowest end when they make plans on how to improve SL. Or, you know, if go back to the bandwidth example, there are people on satelite connections in the middle of nowhere too (actually met a few of those in SL), who get unstable 256kbps on a good not rainy/snowy day. Should LL stop all possible improvements just so those would stay in SL?

[Beq Janus]

16 hours ago, animats said:

You have more faith in LL than I do. But that area does need attention. The design dates from the early days when textures came from the sim servers over UDP connections. Now they come from Amazon AWS via Akamai caches over HTTP connections, and people have more network bandwidth, so the throttling is too strong.

Oh it has nothing to do with faith, more that this has been on my list for a while, we have a patch from someone that (apparently) addresses some performance aspects of the underlying windows file system, but none of it is worth wasting limited time on if the lab are about to rip it out, even if they replace it with something worse the fact it gets replaced at all means it is not worth spending too much time on this before that. All that said, I shall be enquiring on the status of that rewrite.

[Paul Hexem]

I'm on a couple M.2 drives myself, it really makes a difference.

[bigmoe Whitfield]

On 1/21/2019 at 8:31 PM, KT Kingsley said:

If I remember correctly, the downside of a RAM drive was the extra time it takes to save it on shutdown and restore it on startup. This was a while ago, though, and it did go to an HDD when the computer was off: maybe saving to an SSD would be less of a pain. Also, if you only reboot once a week or so, it'll be something you can live with.

Starting up with this system that I built in 2015, is 7 seconds from power to desktop,  shutdown, like 2 if I can catch it before it hits the black screen lol. 

 

CPU: Intel(R) Core(TM) i7-5820K CPU @ 3.30GHz (3300 MHz)
Memory: 32680 MB
OS Version: Microsoft Windows 10 64-bit (Build 17134)
Graphics Card Vendor: NVIDIA Corporation
Graphics Card: GeForce GTX 980/PCIe/SSE2

 

Viewer never tells it right,

4.5GHZ water cooled for 3 half years. 

Dual 980's.

 

Samsung 850 evo 256gb ssd (fire storm lives here)

Crucial mx500 500gb ssd (currently firestorm cache lives here by it's self)

Team Group l5 lite 480GB (I dont recommend this ssd, it does not have a dram controller and does not use nand III)

Edited January 25, 2019 by bigmoe Whitfield
Added infoers

[Anna Nova]

On 1/22/2019 at 2:30 AM, animats said:

Thanks. It's striking how much of an effect this has on response time to arrow keys. That ought to be unrelated.

Have you also noticed how hyper-sensitive Native Linux Firestorm64 is to arrow keys, compared with, say, Windows Firestorm64 under Wine?  I mention this as I am struggling with trying to find a way of getting the responses to be the same.  Driving SL land vehicles under Native Linux Firestorm64 is almost impossible due to the hyper-sensitivity, and there doesn't appear to be a control for it.

I don't have any spinning drives at all now, just SSDs and cloud-backups...

[animats]

1 minute ago, anna2358 said:

Have you also noticed how hyper-sensitive Native Linux Firestorm64 is to arrow keys, compared with, say, Windows Firestorm64 under Wine?  I mention this as I am struggling with trying to find a way of getting the responses to be the same.  Driving SL land vehicles under Native Linux Firestorm64 is almost impossible due to the hyper-sensitivity, and there doesn't appear to be a control for it.

"Hyper-sensitive" in what way? I drive land vehicles from native Linux Firestorm 64-bit all the time. Over 1000km on motorcycles.

[Anna Nova]

13 hours ago, animats said:

"Hyper-sensitive" in what way? I drive land vehicles from native Linux Firestorm 64-bit all the time. Over 1000km on motorcycles.

Like you give the minimum dab on left arrow and the vehicle turns 90 degrees left.

Also when just flying (as an avatar, no vehicle, so no script) impossible to regulate how high a minimum dab up or down will take you.

It could be my Arch Linux setup.  But I have no idea where to start.

Edit:  I found one thing.  Somehow, despite it being turned off in preferences, the viewer has Flight Assist turned on.  Had to turn it on, save, turn it off, save.  And now I can fly properly.  Vehicles seem a little less violent too.  Sorry for hijacking the discussion.

Edited February 2, 2019 by anna2358

[animats]

I'll check this out tomorrow. More later.