some odd behaviour in FS

[Jackson Redstar]

Computer specs are below

What I have been noticing, and I am not sure their is totaly viewer related or maybe my computer related, but I will be at a event maybe 30-40 people there with shaders on, ALM, shadows on, and be getting maybe 5-6 FPS. GPU running 100%. I turn off the shaders, which turns off ALM and shadows and FPS climbs back up, no surpise there, (I have my render limiter set to 34fps). Swith back on shaders, ALM and shadows, the FPS often, but now always, will stay at about 32-33fps. If I move cam position sometimes the FPS holds, but often, a new cam position will take FPS back down to the 5-6 FPS range, and I do the shader cycling again and FPS goes back up. Seems to me something is "stuck" Pretty hard to explain. when the new performance viewer code came out, never really had a problem holding at least 20 FPS in a crowded event. Now it seems its mostly back to how the viewer was before the performace code. Any thoughts or ideas? things i might want to check or stats I should be looking at?

I have whitelisted FS and cache locations in windows firewall and on Win defender. FS and cache are all on the same drive - samsung SSD on a m.2 slot

 

CPU: Intel(R) Core(TM) i7-10700K CPU @ 3.80GHz (3792 MHz)
Memory: 32671 MB
Concurrency: 16
OS Version: Microsoft Windows 11 64-bit (Build 22621.900)
Graphics Card Vendor: NVIDIA Corporation
Graphics Card: NVIDIA GeForce RTX 3070/PCIe/SSE2
Graphics Card Memory: 8192 MB

Windows Graphics Driver Version: 31.0.15.2756
OpenGL Version: 4.6.0 NVIDIA 527.56

RestrainedLove API: (disabled)
libcurl Version: libcurl/7.54.1 OpenSSL/1.1.1l zlib/1.2.11.zlib-ng nghttp2/1.40.0
J2C Decoder Version: KDU v8.2
Audio Driver Version: FMOD Studio 2.02.09
Dullahan: 1.12.4.202209142021
  CEF: 91.1.21+g9dd45fe+chromium-91.0.4472.114
  Chromium: 91.0.4472.114
LibVLC Version: 3.0.16
Voice Server Version: Vivox 4.10.0000.32327.5fc3fe7c.571099

Settings mode: Firestorm
Viewer Skin: Firestorm (Grey)
Window size: 2560x1417 px
Font Used: Deja Vu (96 dpi)
Font Size Adjustment: 0 pt
UI Scaling: 1
Draw distance: 120 m
Bandwidth: 30000 kbit/s
LOD factor: 2
Render quality: High (5/7)
Advanced Lighting Model: Yes
Texture memory: Dynamic (4096 MB min / 10% Cache / 10% VRAM)
Disk cache: Max size 9984.0 MB (100.0% used)
Built with MSVC version 1934
Packets Lost: 0/3,343 (0.0%)

[Wulfie Reanimator]

This may or may not be vaguely related, but I would get pretty severe freezing while moving my camera (on an SSD and an i5-13600 CPU), until I changed the cache location to a different drive than my operating system.

I would suspect Win11 as the problem, but there should be a lot more complaints if it was that simple.

Edited December 23, 2022 by Wulfie Reanimator

[Jackson Redstar]

1 hour ago, Wulfie Reanimator said:

This may or may not be vaguely related, but I would get pretty severe freezing while moving my camera (on an SSD and an i5-13600 CPU), until I changed the cache location to a different drive than my operating system.

I would suspect Win11 as the problem, but there should be a lot more complaints if it was that simple.

interesting I will try that as that is one of the issues i been noticing as well lately

[Odaks]

On 12/23/2022 at 4:45 PM, Wulfie Reanimator said:

pretty severe freezing

Yes, very interesting. I too am getting a type of freezing, mainly noticable with trying to move my cursor around, since "upgrading" to W 11. It does  not happen all the time, but when it does, it is really frustrating. This is using Firestorm on fairly decent laptop with an i7 8750H paired with a GTX 1070 Max-Q and 32GB DDR4 Ram. My caches are on a 256GB  M.2, PCIe NVMe SSD. I had been suspecting throttling of some sort because it does run hot; CPU temperatures in the 80 - 85C range are common and spikes up to 90C do happen.

I need to try different viewers and, when I can lay my hands on one, a different computer. 

[Wulfie Reanimator]

2 hours ago, Odaks said:

My caches are on a 256GB  M.2, PCIe NVMe SSD.

Is this separate from where Win11 is installed? Once I did that, all problems disappeared and I haven't had time to see if that was just a fluke.

[Odaks]

Just now, Wulfie Reanimator said:

Is this separate from where Win11 is installed? Once I did that, all problems disappeared and I haven't had time to see if that was just a fluke.

No, not separate. I figured the caches should be on the fastest drive, and that's where they defaulted to. However, given your experience, it will surely be worth a go to put the caches on the other drive which is a 2TB SEAGATE FIRECUDA 2.5" SSHD, which should still provide a reasonably speedy performance. I'll try this out over the holiday period and see what results.

[Jackson Redstar]

I tried moving the cache to another .m2 SSD i have on the system Im not really using and for me didnt seem help much. using the alt mouse buton to move the cam around in a crowded enviroment really geives the stutters and freeze frames. I am trying to decode the texture console to see if I can detect any patterns, but I suspect this is prob a win 11 update issue

 

[Ardy Lay]

Hmm, I don't use Firestorm.  I just use LL's release or maybe a release candidate and sometimes a "project viewer".  Recently I used Second Life Release 6.6.9.577220 (64bit) at a party with about 40 avatars present.  I was asked to stream the event online so even more could watch.  I received several compliments saying my stream looked clear and smooth.  I did notice some texture evictions and reloads using this viewer.  It doesn't keep as much texture data in memory as Firestorm does.  I recently compared this viewer to a LL project viewer that keeps a whole lot more texture data in memory and noticed that one was stuttering at times.  The fast timers console was identifying those stutters as time spent in "swap".  When two of my stream viewers compared that streaming run with their own performance at the same venue at the same time running Firestorm, they said they got about the same stuttering and about the same frame rate when it's not stuttering.  What I find interesting is I am using an i9-9900k with an nvidia GTX 1070 and they are both running newer CPUs and GPUs than I run.  Both are using RTX 3080 GPUs.  All three of the compared systems are running on Windows 11.

[Ardy Lay]

Oh, I forgot to say I have OS and SL cache on a single volume on a Samsung SSD that has DRAM cache and command queuing.

[Odaks]

On 12/25/2022 at 8:05 AM, Odaks said:

put the caches on the other drive which is a 2TB SEAGATE FIRECUDA 2.5" SSHD

Just to report back, I did move the caches and have been running like this for about 2 weeks now.

I have the impression that things are somewhat better but not completely solved. This is far from a scientific-based observation though.

[Jackson Redstar]

3 hours ago, Odaks said:

Just to report back, I did move the caches and have been running like this for about 2 weeks now.

I have the impression that things are somewhat better but not completely solved. This is far from a scientific-based observation though.

with modern drives SSD and especially if they are on a .m2 socket, I really cant see how moving cache files around would help all that much

[Wulfie Reanimator]

6 hours ago, Jackson Redstar said:

with modern drives SSD and especially if they are on a .m2 socket, I really cant see how moving cache files around would help all that much

Even back when I was running Phoenix on HDD, the stuttering didn't exist even remotely on the same level as in this thread.

While the problem isn't the speed of the drive, Windows 11 is doing something weird which seems related to high activity on the same drive as the OS. (I've had one instance of the same problem with a different program.)

[VenKellie]

On 12/23/2022 at 11:45 AM, Wulfie Reanimator said:

This may or may not be vaguely related, but I would get pretty severe freezing while moving my camera (on an SSD and an i5-13600 CPU), until I changed the cache location to a different drive than my operating system.

I would suspect Win11 as the problem, but there should be a lot more complaints if it was that simple.

same issue regardless if i cache from my SSD or HDD.

[Kathrine Jansma]

16 hours ago, Jackson Redstar said:

I really cant see how moving cache files around would help all that much

In theory you can break every kind of disk performance if you issue frequent fsyncs() to the drive. Or if you disable the write cache on HDDs, e.g. if you tried to use SL on a Windows Domain Controller. 😉

Other example: The Firefox browser used a gazillion of fsyncs() for its internal SQLite Database (https://bugzilla.mozilla.org/show_bug.cgi?id=421482) which made it a total mess. The cache layer of the SL Viewer might have similar lurking issues with fsyncs.

Pushing the cache to a different drive if you have such fsync() storms running might help.

The generic filesystem code of the viewer is pretty dumb, it does not really use the high performance async I/O options you would need to really get the benefits of a modern NVMe SSD. Usually that does not matter, but sometimes it does. Modern I/O works best when you have lots of operations in flight via thread pools or modern async APIs like IOCompletionPorts/io_uring. The viewer just runs a small threadpool for disk access, far away from the queue depths of 65536 possible parallel operations a typical NVMe SSD supports. Most of the time thats just fine and fast enough. But you could do better.

[Henri Beauchamp]

20 hours ago, Kathrine Jansma said:

The generic filesystem code of the viewer is pretty dumb .../... Most of the time thats just fine and fast enough. But you could do better.

It uses basic C-library calls (or boost libraries ones, in some viewers such as LL's, which might actually prove to be slower), which are indeed fast enough for such a small amount of file read/writes as seen in the viewer (we are not in the same order of magnitude as what is seen with database servers, for example), especially once you got those file operations pushed to a thread.

This said, the use of io_uring/IOCP could (slightly) improve things (but requires that your file operation can indeed be threaded/async, meaning it won't help for synchronous reads/writes still sometimes needed in the viewer main thread)...

 

On 1/9/2023 at 9:51 PM, Odaks said:

a 2TB SEAGATE FIRECUDA 2.5” SSHD

Being an SSHD, you may encounter two slow-down issues:

1.- When the SSD storage part of the disk is full, the controller will write to the hard disk platters instead, causing a noticeable slow down in file operations.

2.- When the firmware managing the SSD storage part is not using the TRIM command after a sector gets erased, subsequent writes to that sector will be slower. It is easier to do proper TRIMing with genuine SSDs, since this can be configured at the OS driver level.

 

For the viewer cache, and provided you got enough RAM (32GB or more), you could use a RAM-disk instead, which would spare you any slow down and would save your SS(H)D endurance.

Edited January 11, 2023 by Henri Beauchamp