New version of Sharpview viewer available

[animats]

My experimental Sharpview viewer can now display multiple regions. Unlike previous versions, you can now move across regions.

This is a limited viewer. Avatars are still blocks, but most other objects render properly. No audio, no editing. Just move and view. It's a completely new implementation, written entirely in Rust, and uses many threads. The goal is to effectively use the full resources of a gamer PC with a good GPU with 6GB or better.

This version displays the four regions nearest to the avatar. All four regions are displayed at High level of mesh detail. Modern GPUs have enough power to do this. The eventual goal is to show region-sized impostor meshes for distant regions beyond the nearby four, much like sim surrounds.

I recommend Sharpview only to people doing development work on SL or the Other Simulator. Downloading requires a password, which you can request by sending me a message. See the release notes. I'd like to have a few more people testing it, but it's nowhere near being a daily driver. Sharpview should be used only with a low-value alt. It doesn't do anything hostile, but it hasn't been tested enough for general use.

Runs on Linux and Windows. Some tests indicate it can be built for MacOS, but as yet that's not supported.

[Phil Deakins]

I suppose there's no chance of you making it run on an ARM processor?

[randakong]

Gonna give a try when full release.

[randakong]

Im can't find any YouTube gameplay on this viewer. Anybody care to upload some gameplay up?

[Rowan Amore]

58 minutes ago, randakong said:

Im can't find any YouTube gameplay on this viewer. Anybody care to upload some gameplay up?

https://video.hardlimit.com/w/ceXf7Xr1b8f29VvyV4zqEF

[animats]

On 3/8/2024 at 5:34 AM, Phil Deakins said:

I suppose there's no chance of you making it run on an ARM processor?

Actually, it should run on anything that Rust can compile for that can handle the graphics. Some tests have been run on Apple silicon.

For now, though, I'm only producing Linux and Windows versions. This is still early.

Edited March 26 by animats

[Henri Beauchamp]

10 hours ago, animats said:

Actually, it should run on anything that Rust can compile for that can handle the graphics. .../... For now, though, I'm only producing Linux and Windows versions.

It would likely compile just fine on a Linux aarch64 SBC (Orange Pi5B, Rock Pi5B, Raspberry Pi5, etc), and might run fine on them too, though at low FPS rates.

With the sources (apparently not yet available), I could give it a try on my Orange Pi5B...

Edited March 26 by Henri Beauchamp

[Phil Deakins]

@Henri Beauchamp

I've been considering getting a Raspberry Pi5. If you have any success, post here and let us know please. Maybe we could do what we did before with Radegast, and try to get it running on the Pi.

Edited March 26 by Phil Deakins

[Henri Beauchamp]

11 hours ago, Phil Deakins said:

If you have any success

I don't have the sources, to start with...

[animats]

I'm not promoting Sharpview for wide use at this time. It's a tech demo at this point.

Running it on a Raspberry PI is pointless. It's intended for gamer machines with a good GPU. Minimum is about an NVidia 1060 6GB. It's intended to answer the question, can we make Second Life look like an AAA title on a machine that can run AAA titles. The answer appears to be yes. When I started, people were saying that was inherently impossible. It's not. It does, however, take a lot of GPU memory and network bandwidth.

While I've been working on this new implementation, (Sharpview is 100% safe Rust, all new code) though, the C++ viewers have improved quite a bit. They now have ALM as standard, PBR is close, multi-threaded asset decompression is now in several C++ viewers, and in general things are looking up.

Looking ahead, the really hard problem is to figure out some way to bake avatars, with all those mesh layers, down to one mesh and skeleton, like an animesh. Like Bakes On Mesh, and done at the same time, when clothes change. That's really hard. But it's necessary if we don't want things to choke every time we get 50-100 avatars in the same region. At least for avatars further away than 2-3 meters, where we can get away with some loss of detail, it's probably possible.

[animats]

New version out: 0.7.0. Download from here. Windows and Linux versions are available.

Again, I'm not promoting Sharpview for general use at this time. It's still a tech demo. Use it with a low-value alt. Download requires a password; IM me for access.

In this version, you can explore the world of Second Life. Sitting, flying, and driving work. Region crossings work (mostly). Right-clicking on things brings up a pie menu, like Firestorm. The only option that does anything is "Sit". Can't do "Touch" yet, and clicking on things is rather low-rez; it's based on bounding boxes only in this version.

This version is intended for testing what's really going on at region crossings. There's new logging. If you open the "Developer->Statistics" window, you can watch the status of regions as they connect and disconnect. Region crossing failures do occur, and when they do, the Statistics window should, below the current region and location, show, in red, the name of the region that seems to be giving trouble. Most region crossing failures seem to involve the avatar's control region (the one that gets the key presses) differing from the region in which the avatar is present. That happens momentarily at each region crossing. If that condition lasts for more than a second, something has gone wrong. Don't know why that happens, but it can be detected and displayed.

Incidentally, the old "EstablishAgentCommunication" server side bug, the one that delays region appearance in Sharpview, appears to be fixed in a newer Second Life simulator in test on the beta grid. So the end of that problem is in sight.

[AmeliaJ08]

6 hours ago, animats said:

Incidentally, the old "EstablishAgentCommunication" server side bug, the one that delays region appearance in Sharpview, appears to be fixed in a newer Second Life simulator in test on the beta grid. So the end of that problem is in sight.

Is that the bug responsible for the teleport failure & subsequent disconnect issue?

[Monty Linden]

Nope.  EAC problem is bad implementation (mostly).  The teleport failure problem is bad design (mostly).

[animats]

13 hours ago, Monty Linden said:

Nope.  EAC problem is bad implementation (mostly).  The teleport failure problem is bad design (mostly).

Right.

Second Life uses four basic mechanisms to keep this distributed system in sync.

1. Lockstep. One side says to A, "do X". A replies "did X". Sim to viewer examples are RegionHandshake/RegionHandshakeReply,  and CrossedRegion/CompleteAgentMovement.
2. Eventually consistent. No matter what order the messages come in, the result is the same at the end. ObjectUpdate works this way, mostly. The first ObjectUpdate from a sim for the object creates it in the appropriate region. If the update for a child prim arrives before the root prim, the child is, temporarily, an "orphan" and goes on hold, not appearing on screen because its position info is relative to the root prim, so the viewer doesn't know where to put it yet. When the parent arrives, orphans are placed with their parents and appear in world. This works well; in Sharpview I check for orphans that never found their parent, and, in normal operation, don't find any.
3. Loss-tolerant. As objects move around, their current position is sent via ImprovedTerseObjectUpdate messages. These are shorter than full ObjectUpdate messages. They're sent as "unreliable", and not re-transmitted, because, if you lose one, you want to use a later one, not a stale one that was retransmitted.
4. Reliable and in-order. Events on the event channel, which is a TCP connection, should be both reliable and in order. Due to some implementation problems and middleware, discussed previously, they're not. Monty LInden is working on this.

Some of the things that happen around region crossings and teleports aren't in any of those categories. They're "eventually consistent if everything happens in the normal sequence". This is the underlying cause of most of the things that fail intermittently.

The theory on how to do this reliably wasn't well developed until 2011. Second Life was designed well before that. There were no standard patterns to follow at the time.

My own thinking on this, and this is just me, is this:

• Single teleports and simple region crossings seem to have been designed to be eventually consistent. Where they are not, that's an ordinary bug. They mostly work, although under overload things go wrong. We see failures at crowded clubs when too many inbound teleports are underway, for example. We see avatars spending too much time in pink cloud mode.
• Fast multiple teleports and double region crossings just aren't going to work reliably without some kind of lockstep interlock to prevent a second one from starting until the first one has finished. Trying to make all the cases there eventually consistent is a very tough problem. Preventing a second one from starting too soon is easier. Yes, under certain circumstances this will briefly delay some movement. But if there's any significant delay at a double region crossing, it's going to fail anyway.

[animats]

Sharpview 0.10.0 has some serious problems at some region crossings. Here's a video. In a double region crossing, it's possible to get a situation where two regions are both sending ObjectUpdate messages for the main agent object. In Sharpview 0.10.0, that produces terrible visuals and eventually a disconnect.

I put a check in my development version, 0.10.1, which ignores bogus ObjectUpdate messages which try to change the main agent's region away from the region that the most recent CrossedRegion message said was in charge. This workaround helps a lot. No more vibrating between two regions. I still log the bogus messages, of course. I suspect that the C++ viewers have a similar check.

I've been testing this by logging in an alt with Sharpview and seating them in a YavaPod or SLRR train, then just letting that run for an hour or so.I get about one fail per hour. It falls just like it does in Firestorm - the avatar is Left Behind, seated on the ground, while the YavaPod or train goes on without them. Avatars come off at the usual spots, such as that spot on the Heterocera SLRR main line where the tracks cross over a region corner. So I've achieved bug compatibility.

Monty Linden has some fixes to region crossings in test. Right now, they're up on a few regions on the beta grid. I've tested and haven't had any failures there so far. This is a big step forward.

[Monty Linden]

On 4/13/2024 at 3:46 AM, animats said:

I put a check in my development version, 0.10.1, which ignores bogus ObjectUpdate messages which try to change the main agent's region away from the region that the most recent CrossedRegion message said was in charge. This workaround helps a lot. No more vibrating between two regions. I still log the bogus messages, of course. I suspect that the C++ viewers have a similar check.

I haven't looked into this specific problem but one universal throughout SL is that there's very little ordering guarantee between the various message streams.  UDP messages aren't ordered with respect to TCP (UDP deprecated) messages.  Non-llmessage events are not ordered relative to UDP messages.  Independent Caps not ordered with llmessage or events, etc.  Most of the time there will appear to be an order but it's accidental.  This gets learned when something odd crops up and a defense is put in place that was always needed.  The problem and its patch gets labeled a weird behavior of the network when it's actually a designed-in necessity and should have been described in the protocol.  But so it goes.

If you're building a fresh transport or viewer implementation, you're going to be going down the rediscovery path.  That includes the SL mobile viewer, libremetaverse, and Sharpview, to name three.  An aid to finding these conditions would be fuzzing the network timing a bit.  Delay UDP with respect to TCP (12043, 12046) and vice versa.  Randomly delaying selected responses within the TCP stream.  (Difficult without man-in-the-middle techniques.  Might need to do something within the client.)  This should reveal assumptions earlier and at a time of your choosing.

 

On 4/13/2024 at 3:46 AM, animats said:

Monty Linden has some fixes to region crossings in test. Right now, they're up on a few regions on the beta grid. I've tested and haven't had any failures there so far. This is a big step forward.

Thanks, I'm glad that's looking good in your testing although I'm not certain it's related to the message problem above.  I get these in testing constantly (both pre- and post-work).  They're like Keystone Cop shorts:  avatars go flying off of vehicles, attachments fall uselessly to the ground.  It's a madhouse.  Vehicles just add an additional layer of order and timing assumptions that simply aren't right.  Getting to robust is going to take a few releases....

[Henri Beauchamp]

On 4/13/2024 at 5:46 AM, animats said:

I suspect that the C++ viewers have a similar check.

In fact, those spurious ObjectUpdate's messages have mainly been an issue with agent attachments, for C++ viewers. Gory details here (in paragraph 1). So yes, most TPVs do have code to ignore those bogus ”kill attachment” messages, in one form or another...

However, the agent stays rooted to the region it arrived in, even if the departure sim says otherwise via a bogus ObjectUpdate: the agent only changes region on ”AgentMovementComplete” (which callback is process_agent_movement_complete() in llviewermessage.cpp), and therefore any position received from another region is ignored.

You could do the same thing with Sharpview...

[animats]

4 hours ago, Henri Beauchamp said:

the agent only changes region on ”AgentMovementComplete” (which callback is process_agent_movement_complete() in llviewermessage.cpp), and therefore any position received from another region is ignored.

That's what I'm doing now. Good to know the C++ viewers do it too.

A few bogus messages from the old region are normal at a region crossing, but in the video i linked, the old region or regions kept sending bogus ObjectUpdate messages for tens of seconds after the avatar and vehicle had left the area. Hundreds of ObjectUpdate messages from the wrong simulator. I have detailed logs for that video if anybody cares. I've seen this at two region corners. It's an indication that the simulator to simulator handoff somehow became confused. The viewer sees fallout from that, but it's not a problem with simulator-to-viewer network traffic.

Edited April 15 by animats

[Monty Linden]

I was a little wary of watching a video at hardlimit.com but that was interesting.  I'd agree about the simulator<>simulator handoff and would love to capture that sometime - more is going wrong in there.  Very familiar with sitting in the dust after a bad crossing.

[animats]

Yes, that was strange. I suspect that happens occasionally and viewers compensate for it. I'll find the part of the log file that matches the video and make it available.

(I use Hardlimit because it's free, and doesn't insert ads. YouTube has too many interruptions now.)

[animats]

Another Sharpview demo video: A walk through two regions of Fantasy Faire 2024.

Technical notes, mainly for viewer devs.

• Viewpoint is first person, and there's no camera smoothing. So it's a bit bouncy.
• No environment rendering, lighting, or new PBR materials yet. Avatars are still blocks.
• There are still frame rate drops when moving. Rendering is 60FPS when stationary and the asset loader threads are idle. Asset loading is supposed to be concurrent with rendering. For that to work, Sharpview, Rend3, WGPU, Vulkan, and the GPU all have to support that. Right now, Sharpview and Vulkan are ready, WGPU isn't ready, and Rend3 needs some minor fixes once WGPU is fixed. See this posting in WGPU discussions for more details.

I'm trying to get a high frame rate without sacrificing detail. It's not easy, but it's possible.