Objects not showing up properly is starting to ruin SL for me

[animats]

Ref: https://jira.secondlife.com/browse/BUG-233107 Technical details there.

This has been driving people up the walls for weeks. Especially anybody involved with viewer development, because it looks like a viewer bug. By now, it's been seen in the LL viewer, Firestorm, my experimental Rust viewer, and a few of the others. As of last week's server user group meeting, everybody, including the Lindens, now seems to be convinced it's server side. It's probably related to the optimization that tries to send the viewer only objects the viewer really needs to display.

[Henri Beauchamp]

17 hours ago, Rick Daylight said:

Although... Cool VL refused to show two of my three animesh bunnies until I relogged. Possibly the missing mesh bug... or just a random glitch. Or shy bunnies.

If the ALT SHIFT R workaround (or ”Advanced” -> ”Rendering” -> ”Refresh visibility of objects”) does not work, then it is most likely an issue with missing data in the interest list sent by the server...

You (normally) do not need to relog to work around it: provided you disabled the zoom restrictions (”Preferences” floater -> ”Input & Camera” tab, ”Camera” sub-tab, ”Disable camera zooming constraints” checked), simply ALT-left-mouse-button-click on a nearby object and zoom far beyond draw distance (i.e. until nothing shows on your screen) by moving your mouse backward (with the left button still pressed), then stay like that for 20-30 seconds or so and press SHIFT ESC to bring back your camera. The server should then send again the object data as the viewer reports its new FOV to it, and the missing objects should appear...

Edited January 16, 2023 by Henri Beauchamp

[Rick Nightingale]

@animats yes, that missing item bug is a pain and I've been seeing it too, all over the place.

The stuck Lowest LOD bug is worse for me though. It happens all the time on my land. I hardly get to move around without something showing it and there's no 'cure' that lasts longer than it takes to turn around. @Beq Janushas seen it here, and confirmed it doesn't happen in LL's viewer.

[NiranV Dean]

Just another thing i just noticed while reading through the OP.

The FPS display (at least in FS as far as i recall) does not show you your actual FPS. It shows you your average FPS. 30 FPS average doesn't mean you have smooth 30 FPS, it means you can be jumping between 15 and 45 FPS periodically if these jumps happen in the measured timeframe and then averaged. This is why in BD i chose to show the actual current FPS (hence why it jumps around a lot and seems super unstable). Most games i know show you either your current framerate or at least some very very tight average to which usually catches smaller FPS instabilities but shows you framespikes.

On 1/15/2023 at 3:45 PM, Rick Daylight said:

@Henri Beauchamp I finally got around to trying Cool VL, and can confirm it does not have this stuck low LOD issue or the delayed drawing of objects when I turn around. Nor does Black Dragon from a ten minute test (although that gave me motion sickness with the swaying effect when I move my camera, lol). Seems these issues are Firestorm-only.

Although... Cool VL refused to show two of my three animesh bunnies until I relogged. Possibly the missing mesh bug... or just a random glitch. Or shy bunnies.

It's called camera smoothing and can be configured in Preferences - Camera. Its set higher compared to other Viewers by default. I do not like the jerky direct camera in third person, that's something for Mouselook or Build mode.

[Wulfie Reanimator]

5 hours ago, NiranV Dean said:

The FPS display (at least in FS as far as i recall) does not show you your actual FPS. It shows you your average FPS. 30 FPS average doesn't mean you have smooth 30 FPS, it means you can be jumping between 15 and 45 FPS periodically if these jumps happen in the measured timeframe and then averaged.

This is correct, but you can see a simple frame-by-frame timing breakdown in the statistics window. (You know, the funky rising red dots that are spread all over in most cases. Each dot is a single frame.)

[Ardy Lay]

Yes, spread all over.  How does one plan for a distribution like that?  How does one accomplish a distribution like that?  Is this thing not at all deterministic?  If my broadcast video cameras dumped frames out at random intervals like that, they would be considered useless.

[Henri Beauchamp]

2 hours ago, Ardy Lay said:

Yes, spread all over.  How does one plan for a distribution like that? How does one accomplish a distribution like that?

No one planned for it and this is therefore not an accomplishment... 😛

2 hours ago, Ardy Lay said:

Is this thing not at all deterministic?

No, it is not, and will never be unless some day, the rendering is entirely done in a secondary thread (or set of threads) dedicated to it.

The problem is that the main viewer thread is used both for rendering and for all ancillary tasks the viewer needs and that cannot be pushed to threads, because the various steps need to be done in order until a coherent and non-varying (i.e. not modified by other threads) set of data is available for doing the rendering.

These tasks range from updating the objects data (based on what is sent by the server), launching the fetching of the various assets data (meshes, textures, etc) by sending requests to other threads responsible for their fetching, recovering from other threads the tasks that have been completed since the last frame, etc, to doing non-rendering related things such as updating your inventory, playing sounds, exchanging events with the OS, etc...

The result is a totally random repartition of the tasks and their duration at any given frame. LL and TPVs developers have been working to reduce the frame rates ”hiccups”, by pushing more tasks to threads (I recently pushed LLVOCache file writes to a thread, for example, in the Cool VL Viewer, meaning many 10-500ms ”hiccups” suppressed, when you cross sim borders or TP). Alas, not everything can be smoothed out.

Another thing you may use to reduce the hiccups occurrences, is to limit your frame rate with the frame limiter (provided your viewer got one that is not just the lame VSync trick): if you got a 60 Hz VSync monitor, limit your frame rate to 60fps with the software limiter of the viewer, and your should experience a much smoother SLing...

Here is what I get with 60fps-limiting on in the Cool VL Viewer in my parcel:

Its fps-limiter is fine-grained and measures the time each frame needs to render, then performs supplementary iterations of any unfinished ”ancillary task” (texture prioritizing, fetching, decoding, caching, for example) in 1ms time slices (sleeping the CPU by this amount of time when no work is available/completed by other threads) until the required minium frame duration is reached.

But if, in the same place, I let it free-running, then I get the classical fps ”random” distribution:

Edited January 17, 2023 by Henri Beauchamp

[Jackson Redstar]

I get this often but in weird and unpredicatable ways that kinda defy explanation. For instance I was shooting a wedding and there is a table at the alter, All fine before wedding. my 'LOD' is set to default  2 - btw i have 8gig vrm and 32 gig ram, but then,  after zooming to other areas to film and coming back to alter - no table. Right click on it = the 'wireframe' of it is still there, but cant get it to rezz all.  And my cam is focused on the alter area for at least the next 20 min and table never rezzed. Later on after reception, couple goes back to do some photos there - vola! table is back! I get this weirdness on occasion, just weird, can't explain it. Just SL I suppose. Forgot to mention while filming I always keep 'unchecked' the dynamic adjust level of detail' in the phototools panel 

Edited January 17, 2023 by Jackson Redstar

[Rick Nightingale]

@Jackson Redstar that sounds the missing object bug, rather than my stuck LOD bug. Yes, it's a pain. I've had a large part of my build do that and not come back until a relog. Sometimes the stuff doesn't even show up in wireframe.

[Rick Nightingale]

This is what my framerate looks like, limited to 60fps by the viewer (not vsync), when it starts to judder. The number at the top corner of the screen still insists it's 60+fps, rarely dipping below 60.

Edited January 17, 2023 by Rick Daylight

[Henri Beauchamp]

1 hour ago, Rick Daylight said:

This is what my framerate looks like, limited to 60fps by the viewer (not vsync), when it starts to judder.

Yes, this is what you get whenever the viewer cannot render every frame at the set limit (it can prevent faster frames, but obviously cannot speed up slow ones).

This said, you will notice that the limiter does work by preventing the frames to be rendered too fast (no red dot seen above 63fps or so, the +3fps inaccuracy being due to the sleep() accuracy itself, which is precise at -0/+1ms or so); this does limit the jitter in the frame rate despite the fact the scene is ”too heavy” to render at a stable 60fps rate.

1 hour ago, Rick Daylight said:

The number at the top corner of the screen still insists it's 60+fps, rarely dipping below 60.

This other number is averaged over a second or two (depending on the viewer), so it is less accurate than the one found in the Statistics floater.

[Love Zhaoying]

Was there a "Golden Age" of Second Life where objects DID show up "properly"?

 

[Henri Beauchamp]

23 minutes ago, Love Zhaoying said:

Was there a ”Golden Age” of Second Life where objects DID show up ”properly”?

There has been an ”all attachments in the butt” age (back in 2007, almost at each sim crossing), that's for sure ! 🤣

Edited January 17, 2023 by Henri Beauchamp

[Ardy Lay]

"Interest list bugs" have been a thing since somebody decided to make the simulator responsible for preventing the viewer from asking for data about region objects that the viewer should not be concerned with at the moment.  When was the "interest list" first implemented in Second Life simulators?

The severity of the "interest list bugs" has varied over the years.  I do not recall experiencing them until well after I joined Second Life, but maybe they existed when I joined, and I didn't notice because everything was covered in "MISSING IMAGE" once I left Orientation Island.  At that time LL was packaging Orientation Island textures with the viewer and I was on dial-up Internet access.  As soon as I arrived in an area with "seasoned residents" I was exposed to large amounts of gesture playing resulting in my Internet connection being clogged with sound clip downloads resulting in texture fetch failures.

Edited January 17, 2023 by Ardy Lay

[Henri Beauchamp]

55 minutes ago, Ardy Lay said:

”Interest list bugs” have been a thing since somebody decided to make the simulator responsible for preventing the viewer from asking for data about region objects that the viewer should not be concerned with at the moment.  When was the ”interest list” first implemented in Second Life simulators?

Sometime in April or May 2013... The first mention I made about it in the release notes of my forum was on 2013-05-19 for a bugfix following recent changes to the server-side ”interest list” code.

1 hour ago, Ardy Lay said:

The severity of the ”interest list bugs” has varied over the years. 

The ”interest” of the interest list is that it reduces the burden on the renderer (and also reduced the bandwidth needed by the server to transmit data to the viewer, at a time when FTTH was not even in your dreams and you had to cope with slow ADSL1 or even dial-up connections).

It became all the more ”needed” with the multiplication of meshes and the increase of the objects allotment (either directly or via the LI system introduction) per sim.

It does have a few race condition issues, due to the network delays, in excess of server-side bugs that have varied over years, and viewer-side bugs (such as in LLVOCache code), that should now be fixed in all maintained viewers.

[Ardy Lay]

Hmm, region object cache files typically are not huge.  I wonder if futzing around with the interest list is making a mountain of trouble out of a molehill of data.

[Henri Beauchamp]

10 hours ago, Ardy Lay said:

Hmm, region object cache files typically are not huge.

Not all objects get written on file but only those that are in your LLVOCache at the time the region is disconnected...

10 hours ago, Ardy Lay said:

I wonder if futzing around with the interest list is making a mountain of trouble out of a molehill of data.

The Cool VL Viewer now lets you the option to disable objects cache file writes (but it is not really worth disabling them, since those writes are now threaded in this viewer and therefore not causing fps ”hiccups” any more) and/or file objects cache reads; the latter can be interesting, when you travel across many main land sims and you got a very fast network connection: the fps rate is no more affected by hiccups at sims connection, whenever a cache file already exists for them and would otherwise have to be synchronously (*) read.

 

Note that the object cache and the interest list protocol have two different purposes: the first is to lower the network traffic when you connect to an already visited region, and therefore speed up rezzing whenever your network is slow (which is not the case anymore nowadays for most SLers), while the interest list itself helps with higher frame rates and lower memory consumption (including VRAM, since less objects loaded = less textures to keep around).

Granted, the benefits of the objects cache usage are nowadays rather... disputable.

--------------

(*) Sadly, I could not manage to thread cache reads properly, because the SL sim servers are apparently not even waiting for the handshake reply from the viewer to the UDP message they send to trigger that read, before they start sending object data, and if you thread the latter, you sometimes receive data before the cache can be properly loaded, causing race conditions and failed objects rezzing, up to an entire sim, sometimes.

Edited January 18, 2023 by Henri Beauchamp

[MarissaOrloff]

On 12/7/2022 at 10:55 AM, Henri Beauchamp said:

Workarounds range from reducing the RenderVolumeLODFactor (so that less vertices get rendered, with the hope that it will suffice to stay below the limit) to increasing RenderMaxNodeSize (which works better, but will cause a higher memory consumption both at the GPU and GPU levels).

Thank you Henri! Increasing RenderMaxNodeSize solved a major headache I was having with a highly detailed item.

[sweetandsassy1988]

Hi this is sweetandsassy1988    I am having major problems with firestorm not showing walls or floors    .  when i just use second life viewer i see everything   so the problem is with Firestorm.  i have read where other people have had this problem   but they get a temporary fix  i cant even get that.  i dont know how to fix and yes it does ruin the experience i have on my second life.    advice anyone???????   thank you

[primerib1]

On 1/19/2023 at 11:15 AM, sweetandsassy1988 said:

Hi this is sweetandsassy1988    I am having major problems with firestorm not showing walls or floors    .  when i just use second life viewer i see everything   so the problem is with Firestorm.  i have read where other people have had this problem   but they get a temporary fix  i cant even get that.  i dont know how to fix and yes it does ruin the experience i have on my second life.    advice anyone???????   thank you

Have you upgraded to the latest FireStorm?

If you still having problems, try clearing cache.

If that still gives you problem, you might want to consider using other Viewers (as you mentioned SLV does not give you the same issues).

If you're on Windows, you're in luck because you have a whole bunch of them to try out ... Kokua, Alchemy, Black Dragon, Cool VL Viewer, Genesis, etc.

If you're on Linux ... well ... that's a bit tough.

[animats]

On 1/17/2023 at 7:10 PM, Ardy Lay said:

Hmm, region object cache files typically are not huge.  I wonder if futzing around with the interest list is making a mountain of trouble out of a molehill of data.

This has nothing to do with the region object cache files in the viewer. Even the "give me everything" request a viewer can send to the server fails to tell the viewer about some objects.

There's some optimization of the interest list, apparently for not showing small objects far from the viewpoint. I suspect that optimization is broken. Sometimes huge objects aren't delivered.

(A bit of explanation: What's really going on here is a sync problem, like syncing up contact lists. Each sim server knows about all the objects in its region. When a viewer first connects to a sim server, it has no info about the region.  The viewer sends the sim servers in range an Agent Update: "My camera is here, pointed in this direction, and my draw distance is N meters. The up arrow key is pressed, and I'm walking". The server takes that info, and starts sending lots of Object Updates: "Object UUID is at LOCATION with these FEATURES". The "interest list" is the total of all those Object Updates. When you first enter a region, there is a flood of object updates for about 15 seconds. Sim and viewer are then supposed to be in sync.The viewer can't tell if it is in sync; it's completely dependent on what the sim server chooses to tell it about. So viewers cannot detect the problem. It's pretty clear at this point that the sim servers are not telling the viewers about some objects. But no one knows why yet.

As the viewpoint moves, the viewer continues to send Agent Updates, and the sims in range send more Object Updates appropriate to the new camera location. So there's no single "interest list" - it's just what the sim servers decide the viewer needs to know right now. This is why moving around can sometimes clear up the problem - it changes what's "interesting". Relogging, teleporting in and out, and leaving the area and returning all force an update of what's interesting. However, clearing the cache or reinstalling the viewer appears to be irrelevant to this problem.)

Go to Server User Group (Denby, 1200 SLT, every Tuesday) to discuss.

Edited January 23, 2023 by animats

[Ardy Lay]

2 hours ago, animats said:

Go to Server User Group (Denby, 1200 SLT, every Tuesday) to discuss.

Sorry.  I have a day job.

[animats]

I think I found the problem. See BUG-233107. I put a temporary workaround in my experimental Rust viewer. Went from 4 fails in 10 logins to 0 fails in 25 logins.

It's a sim side bug. A viewer side workaround is possible, but not a good solution. See the JIRA for details.

[Monty Linden]

Yeah, that's a server-side problem to be fixed if true.

[Henri Beauchamp]

I added more info to the BUG-233107 that I gathered during my attempt to thread LLVOCache file reads... I won't be surprised if the bug was not (only/always) a race condition, but a bad assumption by the server about the quality of the camera FOV info (which is extrapolated/approximated by the viewer on login and far TPs) sent via the initial AgentUpdate message...

Edited January 25, 2023 by Henri Beauchamp