It's time for a place for technical discussions on viewers

[Kathrine Jansma]

3 hours ago, Love Zhaoying said:

Is sarcasm, or is it really much different than using a solid-state drive?

It obviously depends on how good your system is with its disk cache.

The hardware differences are still huge:

	Latency	Throughput
DDR5 RAM	20 ns	80 GB/s
SSD NVMe	200000 ns	2 - 15 GB/s (PCIe 3 to 5)

Add to that the extra latency of the driver stack and its a massive difference. BUT..., if your OS has a decent disk cache it may make the difference vanish. But it is pretty easy to get data kicked out of the disk cache, so unless you have giant amounts of RAM and a sufficiently huge disk cache, the chances are that a dedicated RAM disk is faster.

Does it really matter? Rarely.

It is squeezing out the last extra bits of performance. With a RAM disk you do not really need to multithread your file access all that much, as you can do a lot more sequential I/O calls in the same timeframe, while with NVMe you would need to get the concurrency up to make up for the slower hardware to compete. As NVMe allows a huge amount of parallel requests (>> 64k), you can recover a lot of the latency differences if you are able to keep the I/O queue full. But viewers do not really do that, so RAM wins.

Telling your AV solution to keep its hands off your disk cache is typically an order of magnitude more important. 

[Love Zhaoying]

It's interesting to me because, RAM disk is soooo old a concept (get off my lawn!) and I never had failure with my SSD yet (knock on wood), didn't think how it's so much slower.

Thanks!

[Henri Beauchamp]

17 minutes ago, Love Zhaoying said:

It's interesting to me because, RAM disk is soooo old a concept (get off my lawn!) and I never had failure with my SSD yet (knock on wood), didn't think how it's so much slower.

And Kathrine used an NVMe SSD as an example... A SATA SSD is even slower (about 4x to 30x slower) !... I personally do not use NVMe drives, because they are way too fragile, and unpractical to assemble in a RAID. I use a RAID5 of 3 SSDs instead (multi-layer MLC ones, of good quality and high endurance)...

OSes use the free RAM as a cache for disks, but Windows is especially inefficient in this respect, and even the Linux caching scheme will end up writing the data on your disks, which will slow down things anyway...

The use of a RAM-disk is especially interesting nowadays to spare your SSDs and avoid using up their precious (and very limited, in the case of TLC or, horror (!), QLC cells) write cycles for what is, in the end, temporary and throw away data you do not care about the least on the long term.

A cache such as the SL viewer cache endures a lot of small writes (if just to update the cache entries ”last accessed” time stamp, so that the cache can be freed starting from oldest entries first when room must be made in it), which are very toxic for SSDs (or any mass storage based on Flash memory).

Edited May 6, 2023 by Henri Beauchamp

[Love Zhaoying]

Ironically, I only got the SSD to cache SL..

[animats]

9 hours ago, primerib1 said:

"TCP retransmit avalanche"

Not seeing that with Linux netstat -s. Well over 99% of TCP segments are  being received normally.

9 hours ago, Henri Beauchamp said:

RAM-disk

You can, but that's not the bottleneck here. The disk cache in Sharpview is currently .png files, and I may change that to something more elaborate. On a solid state disk, it's not the bottleneck. On a rotating hard drive, it is. Biggest single hardware upgrade that helps SL is putting the cache on an SSD. Even a small SSD.

The big wins in this are algorithmic, not local optimization. There's this ocean of content out there. But there are only so many screen pixels.  Figuring out what really needs to be loaded when, and at when level of detail, and how to fake what's far away, is what's needed to look like an AAA title.

[Ardy Lay]

There are good SSDs and there are junk SSDs.

[Henri Beauchamp]

20 minutes ago, Ardy Lay said:

There are good SSDs and there are junk SSDs.

It becomes harder to distinguish one from another... I recently (last year) experienced a SSD failure for a brand I had in high consideration (Samsung, with several older SSDs running wonderfully for years with 0 reallocated/dead sector). The model was an EVO 870 which failed with bad (unreadable) sectors after only 6 months of rather light use (3TB written only, for a 600TB advertised endurance), and I'd recommend anyone to avoid this particular model (just search for ”Samsung EVO 870 failure” on the web if you are not convinced)...

Note that it is not much better with hard disks: I have seen well known brand models (the three IBM Deskstar, AKA ”Deathstar” I ever bought in the past, almost every Seagate 7200.11 I bought (just one still ”live”), and a WD Blue 500GB drive) failing lamentably after only a few months or a couple of years...

You'd better use a RAID5 (even for SSDs, because they are prone to sudden and instant failures) and do incremental weekly backups of your data, like I do (thanks to this policy, I never lost an important file).

[Coffee Pancake]

6 hours ago, Ardy Lay said:

There are good SSDs and there are junk SSDs.

Even a garbage junk super cheap SSD is orders of magnitude better than spinning rust in all but one measurement.

Spinning rust will hold the data longer when powered off and shoved in a drawer.

Don't confuse "brand" notions of "quality" with reliability. Make backups.

[primerib1]

On 5/7/2023 at 9:09 PM, Henri Beauchamp said:

You'd better use a RAID5 (even for SSDs, because they are prone to sudden and instant failures) and do incremental weekly backups of your data, like I do (thanks to this policy, I never lost an important file).

How I wish Windows support ZFS.

Hands down the best filesystem if file integrity is of the utmost importance.

[ST33LDI9ITAL]

On 5/1/2023 at 2:46 AM, Henri Beauchamp said:

This is not Firestorm's fault: Windoze does not offer any mean to find out the actual frequency for the CPU core running a software, and believe me, I tried hard to get it for my own viewer... ”Modern” processors with turbo mode simply get reported the TSC frequency, sadly.

Amusingly, I managed to find a method to get the turbo frequency while the Windows build of my viewer runs via Wine under Linux (i.e. Wine folks got the corresponding system call right, unlike Micro$oft) ! 🤣

Windoze does provide. This PS scriptlet will list core speeds:

Quote

$MaxClockSpeed = (Get-CimInstance CIM_Processor).MaxClockSpeed

While($true){
    $CoreCount = (Get-CimInstance -ClassName Win32_Processor).NumberOfCores
    While($CoreCount--){
        $ProcessorPerformance = (Get-Counter -Counter "\Processor Information(0,$CoreCount)\% Processor Performance").CounterSamples.CookedValue
        $CurrentClockSpeed = $MaxClockSpeed*($ProcessorPerformance/100)
        
        Write-Host "`n Core ${CoreCount}: $CurrentClockSpeed" -ForegroundColor Yellow -NoNewLine
    }
    Write-Host "`n"
    Sleep -Seconds 2
}

https://learn.microsoft.com/en-us/windows/win32/perfctrs/performance-counters-portal