Dual Logging Causes Crashes?

[BondJamesBond]

The title mostly says it.  I'm using the most recent 64 bit version of Firestorm, I'm running a GTX 1050 TI...  I can run one viewer for days without a crash, but if I open a second instance and log in another account...   it's incredibly unstable.  Sometimes it lasts for a few minutes, other times it crashes within seconds of logging in.   Only the second account to log in ever crashes.   I get the bugsplat crash report window after most of the crashes.   After many of the crashes, I get the bug where I log back in at a location I was at several TPs ago, and my avatar never loads (stays the orange cloud forever) until I relog.   When this bug occurs...  the game is stable!   No crash!   But if I relog to force my avatar to load, back to crashing every few seconds/minutes.   

I've talked to a couple of friends who've had similar problems dual logging.   Has anyone had this problem and found a fix?   

[Ardy Lay]

I don't use that particular third-party viewer, but the method I use to keep multiple sessions from corrupting each other's files should work with any viewer.  I create an operating system login for each additional session I want to run.  As an example, Ardy and Leslie have Windows 11 accounts.  Ardy is logged in to Windows 11 and running Second Life Viewer.  Ardy launches Second Life Viewer as another user and selects Leslie's Windows 11 account and proceeds to login as Leslie.  This is not a Second Life specific solution.  It works for other software too.  I don't use voice chat so cannot state that Leslie's Second Life Viewer will have access to voice chat.

[Ardy Lay]

If you can keep the viewers from trying to use the same local data and log files you should be able to run multiples.  For example, different third-party viewers might use completely different directory names for these files.  I used to build a viewer myself just for this purpose.  I modified the file paths so the viewer I built would not interfere with Second Life Viewer.  I think that Cool VL Viewer does use its own directories instead to stomping on Second Life Viewer's directories.

The viewer you are using MIGHT allow you to make filesystem path changes per-instance.  I do not know.

[Kathrine Jansma]

The logfile might show more insights, why you crash. Inventory / Cache File corruption as mentioned by Ardy Lay is surely one issue that might happen.

Another possible issue would be lack of VRAM. A 1050 Ti has just 4 GB VRAM. Depending on how good the free VRAM detection of the viewer works, you might run out of VRAM and crash. If this is the case, you might be able to set a fixed amount thats good enough for two sessions.

 

[Beq Janus]

10 hours ago, BondJamesBond said:

The title mostly says it.  I'm using the most recent 64 bit version of Firestorm, I'm running a GTX 1050 TI...  I can run one viewer for days without a crash, but if I open a second instance and log in another account...   it's incredibly unstable.  Sometimes it lasts for a few minutes, other times it crashes within seconds of logging in.   Only the second account to log in ever crashes.   I get the bugsplat crash report window after most of the crashes.   After many of the crashes, I get the bug where I log back in at a location I was at several TPs ago, and my avatar never loads (stays the orange cloud forever) until I relog.   When this bug occurs...  the game is stable!   No crash!   But if I relog to force my avatar to load, back to crashing every few seconds/minutes.   

I've talked to a couple of friends who've had similar problems dual logging.   Has anyone had this problem and found a fix?   

Can you let me know (private message if you prefer) the name of the alt that was logged in as the second instance when you crashed and submitted the bugsplat(s)? I can sift through them and see if there is an answer. 

Can you check if the same issue happens when running the default (LL) viewer. If it does not then I'll probably be asking you to raise a JIRA (jira.firestormviewer.org) so that I can track this.

At the very least I need to know more about the system how much RAM, VRAM, what are your graphics settings etc. 

Thanks

Beq

 

[Henri Beauchamp]

5 hours ago, Ardy Lay said:

I think that Cool VL Viewer does use its own directories instead to stomping on Second Life Viewer's directories.

In fact, no, not for everything (e.g. it uses the same directories for the IM and chat logs on purpose, so that you can switch between SL's official viewer and mine without loosing any part of your conversations in your logs, but these are ”per account” files, so they won't risk to collide in two simultaneously running sessions of any viewer). It also uses the same global logs and settings directory, but with different file names than any other viewer.

However, the caches (which structure and format are often different) are held in different directories than any other viewer.

So, yes, in the end, it cannot collide/conflict with other viewers, not even with a different version of itself or any number of running instances of the same version (I took great care to detect such occurrences in the code, to avoid fighting for caches, logs or settings writes), so it is safe to use in multiple instances on the same computer.

Edited July 18, 2023 by Henri Beauchamp

[Beq Janus]

12 hours ago, Henri Beauchamp said:

However, the caches (which structure and format are often different) are held in different directories than any other viewer.

Henri, in Cool VL Viewer do you get many complaints about this from the entitled users that we all tend to have at least a few of? 

I'm curious because I've been planning on making discrete caches and logs for a while, but I keep holding back, wondering whether I end up with another 50/50 situation where half the people think it's good and half the people complain that now their caches are eating all their disk space. The plan would be to have a per-user cache, which would, therefore also avoid the read-only cache issues that we have today for second and subsequent instances.

[Ardy Lay]

Can each instance have a unique cached texture header index yet share the texture particle files for data beyond byte 600 in each encoded texture?

I have often wondered if the texture header index for the first 600 bytes of each texture really improves anything.  I remember having to slog through that when asked to test Second Life Viewer caches for corruption in the textures.  I had to assemble the things from the index file and the particle files then pipe them to a reference decoder then gather the resulting errors and warnings to give developers a list of textures that the current build was corrupting.  That was some bug-hunt.

Maybe the texture cache doesn't even do that dance now.  I haven't checked recently.

[Henri Beauchamp]

7 hours ago, Beq Janus said:

Henri, in Cool VL Viewer do you get many complaints about this from the entitled users that we all tend to have at least a few of?

I never had any such complaint, no... But my viewer user base is in no way comparable to Firestorm's either (there is likely a difference of two orders of magnitude in number of users), so...

True, for users of several different viewers, it means more disk space taken by separate caches, but with today's cheap storage solutions, it should not be much of a bother...

7 hours ago, Beq Janus said:

The plan would be to have a per-user cache, which would, therefore also avoid the read-only cache issues that we have today for second and subsequent instances.

Interesting idea, but this might be an overkill for people using a lot of accounts (this time, filling up their storage too much).

If anything, I would recommend making such a feature a configurable option. 😉

Edited July 19, 2023 by Henri Beauchamp

[Beq Janus]

Just returning here to give a public update. @BondJamesBond contacted me privately, and I've been able to look at the bug splat reports. They do (as @Kathrine Jansma suspected) suggest an OOM exception in the NVidia drivers. The next step will be to review the settings and see if we can tighten them up a bit to allow a dual operation in 4GB VRAM. I've not yet seen any preferences etc so I have no idea what settings are in use, but I thought I'd update here with what we have so far. 

@Henri Beauchamp I recall that you've mentioned a couple of times in forum posts in the past about "broken dynamic memory" but I've not seen any details on that. What is the broken behaviour you are alluding to? I can take a look at it if I know what I am looking for. Of course, at this stage, I don't know that we are even using dynamic VRAM here. 

As a general note, all viewers that are largely based on the LL viewer are increasingly able to use more of the VRAM more of the time, this is a trend that is going to continue with PBR. 

[Henri Beauchamp]

14 hours ago, Beq Janus said:

I recall that you've mentioned a couple of times in forum posts in the past about ”broken dynamic memory” but I've not seen any details on that. What is the broken behaviour you are alluding to? I can take a look at it if I know what I am looking for. Of course, at this stage, I don't know that we are even using dynamic VRAM here. 

I do not think I wrote it was ”broken”, but yes, the new ”dynamic” algorithm implemented in FS' latest release seems to be causing issues with VRAM consumption for quite a few people, and as I could notice myself; my suggestion to people encountering a problem to turn it off proved to solve their issue...

The problem with VRAM is that when you fill it up, the GPU driver starts spilling its memory allocations over to the RAM, and then there must be heavy transfers performed on the PCI bus, causing ”hiccups” or right out seconds long freezes. With the old algorithm (fixed and moderate VRAM amount dedicated to the viewer), you do not risk to spill memory as much as you do with the ”dynamic” one.

Also, I noticed that, at least with NVIDIA (I do not have any AMD card to test it against), there are times when, after a seconds-long overshoot of the VRAM amount, the driver was never quite able to ”fully recover” and migrate back all its structures to the VRAM once enough of the latter gets freed (e.g. after teleporting away from a busy place to a sky box), and the only solution left was to relog in a new session (or endure much lower fps rates)...

Feel free to have a look and ”borrow” the code I implemented in the Cool VL Viewer to try and keep everything in line, but be aware that it involves a rather complex set of algorithms with discard bias adaptive adjustments and amortizing, anticipation/extrapolation of the VRAM and GL textures consumption, as well as force-freeing of stale bound GL textures (which otherwise cause a ”leak” of VRAM over time), among other more subtle/indirect adjustments (in my viewer, the texture discard bias also influences the objects retention and push delays in the object cache, for example)...

I am also extremely skeptical about LL's PBR viewer new code for texture discard bias and VRAM management (it causes the VRAM to fill up to the brim), thus why I preferred to (deeply) change the old algorithm for my viewer...

Edited July 21, 2023 by Henri Beauchamp

[JeromFranzic]

On 7/18/2023 at 9:57 AM, Kathrine Jansma said:

The logfile might show more insights, why you crash. Inventory / Cache File corruption as mentioned by Ardy Lay is surely one issue that might happen.

Another possible issue would be lack of VRAM. A 1050 Ti has just 4 GB VRAM. Depending on how good the free VRAM detection of the viewer works, you might run out of VRAM and crash. If this is the case, you might be able to set a fixed amount thats good enough for two sessions.

 

I'm on a 1050 with 4 GB of VRAM, in a laptop. For two Firestorm viewers, I set one at 2048 MB and another at 512 MB. I usually take this on when working with photos using my alts + myself.

For fun... I did try this out late in 2022 in a more active way. One viewer was logged into SL, and another was logged into Open Sim. Both scenes were indoors at clubs with lots of activity. I didn't experience either instance crashing on me, ran it for about an hour with both viewers streaming audio (haha). I used 512 for SL and 2048 for Open Sim. I used Mid graphic settings and lowered object complexity to 2048 for both instances. I normally use 4096.

Of course my frame rate suffered (went down to 10-15 fps on average) and the laptop fans... but whatever, it was fun having two parties on at once and I have a decent cooling pad with powerful fans.