[Obscure question: when does the simulator send EstablishAgentCommunication to the viewer?]

Just now, Henri Beauchamp said:

I'm pretty sure it's a missing viewer message or bad timing in message exchanges, on the contrary...

Huh. What are you thinking is missing or mis-timed?

For anyone who is really, really bored, and knows how to read viewer logs, here's Sharpview's log of a few successful double region crossings followed by a failed crossing. Formats are different, but it's not that different from what you'd get in an LL or TPV log.

Notes:

• Lines beginning "From:" are traffic from sim to viewer. "Message" means it was a UDP message. Just the message type is printed, because many of those are big. "Event" means it came in via an event message poller. The two numbers in parentheses are the location of the sim in-world that sent the message.
• Lines beginning "To:" are UDP traffic from viewer to sim. All the data is displayed.
• Look for CrossedRegion to see activity around region crossings. In the vicinity of those events, the other activity that makes a region crossing go will show up.
• Also look for RegionHandshake and EnableSimulator messages. All four regions at the corner have been live for minutes at this point, so there's no good reason for those to have been sent to the viewer.
• This is a viewer under development, and you'll see messages about message processing not yet implemented.

Yes, I know this is all terminally boring, but it's how stuff gets fixed.

(I'm still getting over some non-COVID flu-like virus, I'm not going anywhere, and it's rained a lot lately.)

[Obscure question: when does the simulator send EstablishAgentCommunication to the viewer?]

1 minute ago, Henri Beauchamp said:

Ideally, you'd need four neighbouring regions in a square, empty of any object, for a start: then you could try and compare your viewer logs with the Cool VL Viewer's with the proper DEBUG tags activated: you won't have all the objects updates spam then, and would only see those related with your own avatar.

Then, you could see what message is missing from your viewer, or what race condition happens...

I don't think we'll see this in the viewer traffic. This looks like a sim to sim problem, which we, as users, cannot see.

[Obscure question: when does the simulator send EstablishAgentCommunication to the viewer?]

OK, more boring stuff to finish out the year.

Here's an annotated log of several successful double region crossings, and one failed one. This is from Sharpview.

 

====> Beginning of successful double region crossing <====
====> Thes are all at the same corner.
====>  [Bronlen] (462848,306688)
====>  [Charlesville] (462848,306944)
====>  [Vallone] (462592,306944)
====>  [Kama Center] (462592,306688)
====> This first one is from Vallone to Bronlen via Charlesville.

02:49:40 [WARN] From (462592,306944), event CrossedRegion: MsgServerEvent { region_handle: (462592,306944), event: CrossedRegion(EventCrossedRegion { look_at: <0.392944,-0.919562,0>, position: <1.00946,0.236431,35.6884>, socket_addr: 34.222.128.86:12035, seed_capability: "https://simhost-05f87cd2d2a7b0b7a.agni.secondlife.io:12043/cap/be1edc3a-5ba8-4b50-7ffc-b75380dcec9c", region_box: RegionBox { region_handle: (462848,306944), size: [256, 256] }, session_id: a0f40a7c-a5f3-11ee-9453-0242ac110003, agent_id: b5fde908-eebf-4505-b686-ca74c3f91979 }) }
02:49:40 [WARN] CrossedRegion: leaving region (462592,306944), entering live region [Charlesville] (462848,306944)
02:49:40 [WARN] CrossedRegion from (462592,306944) to (462848,306944), distance <256,0,0>
02:49:40 [WARN] CrossedRegion from [Vallone]
====> Vallone sim has handed over control of the avatar to Charlesville. Fine.
====> These errors are some late object updates straggling in from Vallone. That seems to be normal. Those are ignored.
02:49:40 [ERROR] Main agent object is in region (462592,306944) but current CrossedRegion region is (462848,306944)
02:49:40 [ERROR] Main agent object is in region (462592,306944) but current CrossedRegion region is (462848,306944)
02:49:40 [ERROR] Main agent object is in region (462592,306944) but current CrossedRegion region is (462848,306944)
02:49:40 [ERROR] Main agent object is in region (462592,306944) but current CrossedRegion region is (462848,306944)
02:49:40 [ERROR] Main agent object is in region (462592,306944) but current CrossedRegion region is (462848,306944)

====> Here are some seemingly bogus EnableSimulator messages. This is Charlesville telling the viewer to start talking to Bronlen and Kama Center.
====> The viewer has been talking to all four regions for several minutes at this point, so this is redundant. Harmless, though.

02:49:41 [ERROR] Event error: Duplicate add of region (462848,306688), event MsgServerEvent { region_handle: (462848,306944), event: EnableSimulator(EventEnableSimulator { region_box: RegionBox { region_handle: (462848,306688), size: [256, 256] }, socket_addr: 35.87.129.151:13013 }) }
02:49:41 [ERROR] Event error: Duplicate add of region (462592,306688), event MsgServerEvent { region_handle: (462848,306944), event: EnableSimulator(EventEnableSimulator { region_box: RegionBox { region_handle: (462592,306688), size: [256, 256] }, socket_addr: 34.215.138.142:13007 }) }
02:49:42 [WARN] From (462848,306944), event CrossedRegion: MsgServerEvent { region_handle: (462848,306944), event: CrossedRegion(EventCrossedRegion { look_at: <-0.332161,0.943223,0>, position: <4.8429,254.94,35.6402>, socket_addr: 35.87.129.151:13013, seed_capability: "https://simhost-043e6020eb4baf499.agni.secondlife.io:12043/cap/60ea19c3-0600-310b-3f10-56c9e008f05a", region_box: RegionBox { region_handle: (462848,306688), size: [256, 256] }, session_id: a0f40a7c-a5f3-11ee-9453-0242ac110003, agent_id: b5fde908-eebf-4505-b686-ca74c3f91979 }) }
02:49:42 [WARN] CrossedRegion: leaving region (462848,306944), entering live region [Bronlen] (462848,306688)
02:49:42 [WARN] CrossedRegion from (462848,306944) to (462848,306688), distance <0,-256,0>
02:49:42 [WARN] CrossedRegion from [Charlesville]
02:49:42 [ERROR] Main agent object is in region (462848,306944) but current CrossedRegion region is (462848,306688)
02:49:43 [ERROR] Event error: Duplicate add of region (462592,306688), event MsgServerEvent { region_handle: (462848,306688), event: EnableSimulator(EventEnableSimulator { region_box: RegionBox { region_handle: (462592,306688), size: [256, 256] }, socket_addr: 34.215.138.142:13007 }) }


=====> Here's another successful double region crossing - Bronlen to Vallone via Charlesville, the reverse of the previous move.
=====> This is a motorcycle with avatar going forwards and backwards right across the region corner.
02:49:58 [WARN] From (462848,306688), event CrossedRegion: MsgServerEvent { region_handle: (462848,306688), event: CrossedRegion(EventCrossedRegion { look_at: <-0.504536,0.863391,0>, position: <0.915858,1.00684,35.6747>, socket_addr: 34.222.128.86:12035, seed_capability: "https://simhost-05f87cd2d2a7b0b7a.agni.secondlife.io:12043/cap/be1edc3a-5ba8-4b50-7ffc-b75380dcec9c", region_box: RegionBox { region_handle: (462848,306944), size: [256, 256] }, session_id: a0f40a7c-a5f3-11ee-9453-0242ac110003, agent_id: b5fde908-eebf-4505-b686-ca74c3f91979 }) }
02:49:58 [WARN] CrossedRegion: leaving region (462848,306688), entering live region [Charlesville] (462848,306944)
02:49:58 [WARN] CrossedRegion from (462848,306688) to (462848,306944), distance <0,256,0>
02:49:58 [WARN] CrossedRegion from [Bronlen]
02:49:58 [ERROR] Main agent object is in region (462848,306688) but current CrossedRegion region is (462848,306944)
02:49:59 [WARN] From (462848,306944), event CrossedRegion: MsgServerEvent { region_handle: (462848,306944), event: CrossedRegion(EventCrossedRegion { look_at: <-0.504536,0.863391,0>, position: <254.984,2.26188,35.6209>, socket_addr: 34.212.163.17:13019, seed_capability: "https://simhost-04148d0076d95659a.agni.secondlife.io:12043/cap/2de81e9e-b7a6-53aa-66c8-f6bd753aa6c2", region_box: RegionBox { region_handle: (462592,306944), size: [256, 256] }, session_id: a0f40a7c-a5f3-11ee-9453-0242ac110003, agent_id: b5fde908-eebf-4505-b686-ca74c3f91979 }) }
02:49:59 [WARN] CrossedRegion: leaving region (462848,306944), entering live region [Vallone] (462592,306944)
02:49:59 [WARN] CrossedRegion from (462848,306944) to (462592,306944), distance <-256,0,0>
02:49:59 [WARN] CrossedRegion from [Charlesville]
02:49:59 [ERROR] Main agent object is in region (462848,306944) but current CrossedRegion region is (462592,306944)
02:49:59 [ERROR] Main agent object is in region (462848,306944) but current CrossedRegion region is (462592,306944)
02:49:59 [ERROR] Main agent object is in region (462848,306944) but current CrossedRegion region is (462592,306944)
02:49:59 [ERROR] Main agent object is in region (462848,306944) but current CrossedRegion region is (462592,306944)


====> And another successful double crossing, Vallone to Bronlen  via Charlesville.====
02:50:21 [WARN] From (462592,306944), event CrossedRegion: MsgServerEvent { region_handle: (462592,306944), event: CrossedRegion(EventCrossedRegion { look_at: <-0.0494177,0.998778,0>, position: <1.00549,0,35.6919>, socket_addr: 34.222.128.86:12035, seed_capability: "https://simhost-05f87cd2d2a7b0b7a.agni.secondlife.io:12043/cap/be1edc3a-5ba8-4b50-7ffc-b75380dcec9c", region_box: RegionBox { region_handle: (462848,306944), size: [256, 256] }, session_id: a0f40a7c-a5f3-11ee-9453-0242ac110003, agent_id: b5fde908-eebf-4505-b686-ca74c3f91979 }) }
02:50:21 [WARN] CrossedRegion: leaving region (462592,306944), entering live region [Charlesville] (462848,306944)
02:50:21 [WARN] CrossedRegion from (462592,306944) to (462848,306944), distance <256,0,0>
02:50:21 [WARN] CrossedRegion from [Vallone]
02:50:21 [ERROR] Main agent object is in region (462592,306944) but current CrossedRegion region is (462848,306944)
...
02:50:22 [ERROR] Main agent object is in region (462592,306944) but current CrossedRegion region is (462848,306944)
02:50:28 [WARN] From (462848,306944), event CrossedRegion: MsgServerEvent { region_handle: (462848,306944), event: CrossedRegion(EventCrossedRegion { look_at: <-0.0494177,0.998778,0>, position: <2.10215,254.998,35.6366>, socket_addr: 35.87.129.151:13013, seed_capability: "https://simhost-043e6020eb4baf499.agni.secondlife.io:12043/cap/60ea19c3-0600-310b-3f10-56c9e008f05a", region_box: RegionBox { region_handle: (462848,306688), size: [256, 256] }, session_id: a0f40a7c-a5f3-11ee-9453-0242ac110003, agent_id: b5fde908-eebf-4505-b686-ca74c3f91979 }) }
02:50:28 [WARN] CrossedRegion: leaving region (462848,306944), entering live region [Bronlen] (462848,306688)
02:50:28 [WARN] CrossedRegion from (462848,306944) to (462848,306688), distance <0,-256,0>
02:50:28 [WARN] CrossedRegion from [Charlesville]
02:50:28 [ERROR] Main agent object is in region (462848,306944) but current CrossedRegion region is (462848,306688)
...
02:50:29 [ERROR] Main agent object is in region (462848,306944) but current CrossedRegion region is (462848,306688)

====> And another successful double region crossing, back from Bronlen to Vallone via Charlesville
02:51:08 [WARN] From (462848,306688), event CrossedRegion: MsgServerEvent { region_handle: (462848,306688), event: CrossedRegion(EventCrossedRegion { look_at: <-0.744412,-0.667721,0>, position: <0,1.00766,35.6432>, socket_addr: 34.222.128.86:12035, seed_capability: "https://simhost-05f87cd2d2a7b0b7a.agni.secondlife.io:12043/cap/be1edc3a-5ba8-4b50-7ffc-b75380dcec9c", region_box: RegionBox { region_handle: (462848,306944), size: [256, 256] }, session_id: a0f40a7c-a5f3-11ee-9453-0242ac110003, agent_id: b5fde908-eebf-4505-b686-ca74c3f91979 }) }
02:51:08 [WARN] CrossedRegion: leaving region (462848,306688), entering live region [Charlesville] (462848,306944)
02:51:08 [WARN] CrossedRegion from (462848,306688) to (462848,306944), distance <0,256,0>
02:51:08 [WARN] CrossedRegion from [Bronlen]
02:51:08 [ERROR] Main agent object is in region (462848,306688) but current CrossedRegion region is (462848,306944)
02:51:08 [ERROR] Main agent object is in region (462848,306688) but current CrossedRegion region is (462848,306944)
02:51:08 [ERROR] Main agent object is in region (462848,306688) but current CrossedRegion region is (462848,306944)
02:51:14 [WARN] From (462848,306944), event CrossedRegion: MsgServerEvent { region_handle: (462848,306944), event: CrossedRegion(EventCrossedRegion { look_at: <-0.744412,-0.667721,0>, position: <254.998,2.55342,35.6153>, socket_addr: 34.212.163.17:13019, seed_capability: "https://simhost-04148d0076d95659a.agni.secondlife.io:12043/cap/2de81e9e-b7a6-53aa-66c8-f6bd753aa6c2", region_box: RegionBox { region_handle: (462592,306944), size: [256, 256] }, session_id: a0f40a7c-a5f3-11ee-9453-0242ac110003, agent_id: b5fde908-eebf-4505-b686-ca74c3f91979 }) }
02:51:14 [WARN] CrossedRegion: leaving region (462848,306944), entering live region [Vallone] (462592,306944)
02:51:14 [WARN] CrossedRegion from (462848,306944) to (462592,306944), distance <-256,0,0>
02:51:14 [WARN] CrossedRegion from [Charlesville]
02:51:14 [ERROR] Main agent object is in region (462848,306944) but current CrossedRegion region is (462592,306944)
02:51:14 [ERROR] Main agent object is in region (462848,306944) but current CrossedRegion region is (462592,306944)


====> Beginning of a failed double region crossing <====
====> This time, though, we start in Vallone and go through Kama Center, rather than Charlesville.  Stuck in Kama Center. Bronlen not talking. <====
====> Many unnecessary EnableSimulator messages.
====> Need to check if the server is trying to change the IP address or UDP port.
====> These messages are from Vallone, telling the viewer about Charlesville and Kama Center, which are neighbors of Vallone; they share an edge.
02:52:15 [ERROR] Event error: Duplicate add of region (462848,306944), event MsgServerEvent { region_handle: (462592,306944), event: EnableSimulator(EventEnableSimulator { region_box: RegionBox { region_handle: (462848,306944), size: [256, 256] }, socket_addr: 34.222.128.86:12035 }) }
02:52:15 [ERROR] Event error: Duplicate add of region (462592,306688), event MsgServerEvent { region_handle: (462592,306944), event: EnableSimulator(EventEnableSimulator { region_box: RegionBox { region_handle: (462592,306688), size: [256, 256] }, socket_addr: 34.215.138.142:13007 }) }
02:52:20 [WARN] From (462592,306944), event CrossedRegion: MsgServerEvent { region_handle: (462592,306944), event: CrossedRegion(EventCrossedRegion { look_at: <0.0256889,0.99967,0>, position: <255,254.989,35.6911>, socket_addr: 34.215.138.142:13007, seed_capability: "https://simhost-0e6e8ee1f4104630e.agni.secondlife.io:12043/cap/6ec6de3f-73fd-03a5-fe35-f73f1cc09a23", region_box: RegionBox { region_handle: (462592,306688), size: [256, 256] }, session_id: a0f40a7c-a5f3-11ee-9453-0242ac110003, agent_id: b5fde908-eebf-4505-b686-ca74c3f91979 }) }
02:52:20 [WARN] CrossedRegion: leaving region (462592,306944), entering live region [Kama Center] (462592,306688)
02:52:20 [WARN] CrossedRegion from (462592,306944) to (462592,306688), distance <0,-256,0>
02:52:20 [WARN] CrossedRegion from [Vallone]
02:52:21 [ERROR] Main agent object is in region (462592,306944) but current CrossedRegion region is (462592,306688)
02:52:21 [ERROR] Main agent object is in region (462592,306944) but current CrossedRegion region is (462592,306688)
02:52:21 [ERROR] Main agent object is in region (462592,306944) but current CrossedRegion region is (462592,306688)
====> Got to Kama Center OK. Then stuck.
====> Now this is Kama Center telling the viewer about its neighbors, Bronlen and Vallone.
====> Same UDP port as previously used for Bronlen, so the UDP connection is not being restarted.
02:52:22 [ERROR] Event error: Duplicate add of region (462848,306688), event MsgServerEvent { region_handle: (462592,306688), event: EnableSimulator(EventEnableSimulator { region_box: RegionBox { region_handle: (462848,306688), size: [256, 256] }, socket_addr: 35.87.129.151:13013 }) }
02:52:22 [ERROR] Event error: Duplicate add of region (462848,306944), event MsgServerEvent { region_handle: (462592,306688), event: EnableSimulator(EventEnableSimulator { region_box: RegionBox { region_handle: (462848,306944), size: [256, 256] }, socket_addr: 34.222.128.86:12035 }) }

====> First crossing was successful, second one never started.

The main thing to note about this is that nothing unusual happened. It just stopped working. Notes:

• The second crossing at a region corner just never happened, according to the info the viewer sees. All the messages for the first crossing are correct and on the happy path. There's just nothing about a second crossing. To the viewer, it looks like the avatar just never left the second region for the third.
• There's some "noise". There are unnecessary EnableSimulator events. Sometimes those show up on successful crossings. Sometimes they don't show up. There are also unnecessary RegionHandshake messages. It's not in this log, but there are also unnecessary KillObject messages for objects leaving a sim. Sometimes those are present, sometimes not. Viewers discard all this junk routinely. Henri Beauchamp reports seeing this stuff, too.
• The "noise" suggests imperfect synchronization between neighboring simulators. We can't tell from the outside, but this may be a symptom of some inter-simulator communication issue. That's speculative. However, it's something concrete that the simulator-side devs could look at. When we all looked hard at EstablishAgentCommunication problems, that turned up things that needed to be fixed. This may be similar. I have a vague suspicion that there may be some kind of low-level inter-simulator timeout or reset which causes some loss of shared state, and those excess messages are side effects of recovery from that. But that's speculation on my part.
• Sharpview, Firestorm, and Cool VL all log similar message patterns. I've looked at Firestorm logs in the past, and Henri Beauchamp has looked at Cool VL logs. Nothing seems to be wrong. Successful crossings and failed crossings look similar right up to where progress stops.
• Viewer <-> Sim protocol for all this seems sound. The lockstep handoff for ChangedRegion/CompleteAgentMovement works fine. The state change for RegionHandshake/RegionHandshakeReply works fine. The system by which objects move from region to region works fine. There's a brief moment of confusion because control handoff (who has the keyboard and mouse?) and object movement (where is the avatar?) are not synchronized, but those fall into sync in a second or so.The architects of these parts did a good job.

I was hoping to find something clearly wrong that could be fixed or worked around viewer side. Or even just see different behavior in  Sharpview than what the C++ viewers provide, which would give us some ideas to follow up. Nope.

What's probably needed to fix this is to find four regions on the beta grid where there's a clear corner and very little activity. Then log everything the simulators do while an avatar on a vehicle is moved back and forth across the corner. Then look over the logs in great detail. Somewhere in the sim to sim traffic just before a stall, there will be something of interest.

It's not simulator to viewer.

 

[Can anyone get the SL viewer to run under Wine on Linux any more? Crashes with uncaught C++ exception during login.]

Where is SLVersionChecker.py source, anyway? I can't find it on Github. Whatever is causing a subscript out of range over there can't be too hard to fix.

[Can anyone get the SL viewer to run under Wine on Linux any more? Crashes with uncaught C++ exception during login.]

Hm. I just downloaded 7.1.2.71xxx, the PBR maint viewer, and it crashed. I have the debug info.

First, the updater crashes with a subscript out of range in Python code. (Where's the source code for the updater, anyway.)

Subscript out of range in Python. Should be easy to fix.

Then the CEF plugin fails:

This doesn't happen with viewer 6.6.10.

The viewer offers the login screen, and I log in.

Then, the full crash:

wine: Unhandled page fault on write access to 0000000000000000 at address 00000001411C4612 (thread 0288), starting debugger...

2023-12-30T18:44:30Z WARNING #LLCoros# llcommon/llcoros.cpp(368) LLCoros::toplevel : Capturing uncaught exception in coroutine LL
Login::Impl::login_
2023-12-30T18:44:30Z INFO #AppInit# newview/llstartup.cpp(2980) LLStartUp::setStartupState : Startup state changing from STA
TE_LOGIN_AUTH_INIT to STATE_LOGIN_CURL_UNSTUCK
2023-12-30T18:44:30Z INFO #LLProcess# llcommon/llprocess.cpp(1023) LLProcess::handle_status : updater process (2452) exited
with code 1
2023-12-30T18:44:30Z INFO #LLLeap# llcommon/llleap.cpp(180) LLLeapImpl::done : updater process (2452) exited with code 1
2023-12-30T18:44:30Z WARNING #LLCoros# llcommon/llcoros.cpp(225) LLCoros::rethrow : Rethrowing exception from coroutine LLLog
in::Impl::login_
Unhandled exception: C++ exception(object = 0000000017789860, type = 0000000142647548, base = 0000000140000000) in 64-bit code (0x00000
07b01329e).

This is what happens in C++ when you de-reference a null pointer.

Trouble is in Login::Impl::login_, which is where the results from the updater are received and dealt with.

Looks like this build doesn't have debug symbols, so the backtrace is just numbers. I have the full log, of course; ask if you want a copy.

Now, in 6.6.10, the updater still crashes, but it doesn't take down the whole viewer. Nor is the CEF plugin crashing.

 

[Looking fro bookstores or newsstands in SL]

There is a very nice bookstore in the Japanese area along the Linden road in Electra. Just go to Electra and walk east.

[Can anyone get the SL viewer to run under Wine on Linux any more? Crashes with uncaught C++ exception during login.]

1 hour ago, Jenna Huntsman said:

can't say I've ran it recently, but I was using the SL Viewer via Wine during PBR testing and didn't have issues beyond the python issue.

Which version? I was using 7.1.1. A few versions back, it worked. Released 6.6.10 works on the same machine. So, somewhere between 6.6.10 and 7.1.1, something was broken.

I can live with LL not supporting a full Linux version, but it ought to as least work under Wine, like most other games.

Ref to 2021 JIRA: https://jira.secondlife.com/browse/BUG-230918

[Can anyone get the SL viewer to run under Wine on Linux any more? Crashes with uncaught C++ exception during login.]

Until recently, it was possible to run the Second Life viewer under Wine on Linux. That's no longer working. I hardly ever bother to run the SL viewer, but I'm working with someone who is trying to set up pathfinding with the SL viewer. They can't find the right menus. and the LL documentation has screenshots of totally different menus from the distant past.

Fortunately, I can still run the Puppetry project viewer! So I don't have to fix LL's code.

What seems to be broken is the "SL Version checker". That hasn't worked under Wine for years. Crashes because it wants some DLL Wine doesn't have. But until recently, that didn't matter. Now, for some reason, when it aborts, LLLogin::loginCoro fails with an unhandled C++ exception during login. Look at the code around line 232 of lllogin.cpp. It looks like if the login response has "wait_for_updater" set, it's forced down a path that requires updater success. I think.

Also,

• media_plugin_cef.dll now fails. This is new. 2023-12-29T22:36:15Z WARNING #Plugin# llplugin/llplugininstance.cpp(106) LLPluginInstance::load : apr_dso_load of C:\Pro
  gram Files\SecondLifeViewer\llplugin\media_plugin_cef.dll failed with error -2146763645 , additional info string: Unknown error

No, I'm not going to file a JIRA. I'll get the same "Linux is not supported" response as last time.

[Obscure question: when does the simulator send EstablishAgentCommunication to the viewer?]

I'm still being held back by that one minute delay before a new region appears. But if I just walk into that region, the missing EstablishAgentCommunication message from the simulator shows up and the region being entered goes live. This looks terrible but doesn't break anything.

Avatar-only region crossings seem to be working fine, including repeatedly going back and forth across corners. So, about the same as other viewers.

Vehicle region crossings work fine for single crossings. A corner crossing failed after about 10 tries. Again, about the same as other viewers. This tells us that region crossing problems are not a C++ viewer code bug, since this fail is occurring with a totally different implementation. You don't really understand a protocol until you have multiple implementations.

Begin long section that will be extremely boring to anyone not involved with the innards of how all this works.

I have a log of this. Nothing jumps out at me as blatantly wrong. It looks normal. For those few who need the details, here they are. The regions involved are Bronlen (SE), Kama Center (SW), Vallone (NW), and Charlesville (NE)

• There's a CrossedRegion event showing a cross from Vallone to Kama Center. (I'm testing in front of my own shop in Vallone.) Viewer answers with CompleteAgentMovement. That hands off control from Vallone sim to Kama Center sim.
• Then, for no obvious reason, there's a duplicate RegionHandshake telling the viewer about Kama Center, which the viewer already knows about and is talking to. Those show up frequently, not just around failed region crossings. Henri Beauchamp has pointed that out. Viewer answers with RegionHandshakeReply as usual, because failure to reply to even duplicate RegionHandshake messages has been known to stall things out. This should be harmless but might indicate some confusion server side.
• A late TerseImprovedObjectUpdate message for the avatar comes in over UDP from the losing region. This is a race condition between the event channel and the UDP messages, which are not synchronized. But that's harmless; we know it's old info. It's logged as an error and ignored.
• Then an ObjectUpdate with the avatar's UUID shows up from the gaining region. This is what makes the avatar object appear in the new region. The avatar is deleted in the old region by the viewer. (This is how things move across regions - create in new, delete in old, with reuse of meshes and textures.)This is totally normal. It's interesting that this happens, because it says that the gaining region did receive the avatar. That may be helpful to the simulator devs, because it tells them how far the avatar movement process got. I didn't expect to see that message.
• The avatar object has a different local ID than the last time it was in that region, which is normal.
• Some ParcelProperties and ParcelOverlay info comes in. This is the info that creates ban lines and the mini-map. That data should be unnecessary, because the gaining region is already live and has all that info. That may be associated with the duplicate RegionHandshake.
• Shortly after this, Kama Center tells the viewer about Bronlen, using an EnableSimulator event. That's redundant; the viewer already knows about all four regions. It looks like everybody at the corner tells the viewer about the other regions at the corner, just to be sure. The important info (IP address and UDP port) seems to match.
• I'm pressing the up arrow key, trying to drive, and the message log shows the appropriate AgentUpdate messages going to Kama Center sim. At this point, as far as the viewer knows, Kama Center sim has both control and possession of the avatar object, so that should work. I'm watching all this from another avatar, in Firestorm. Nothing moves.
• It's one of my motorcycles, with all the paranoid region crossing scripting, and the script is loudly complaining that it's stuck in the half-unsit state, where "seat being sat on by avatar" and "avatar sitting on seat" info do not match. The motorcycle won't respond to keyboard controls.

And we have a classic region corner crossing fail. 

Insights from this are:

• The region crossing gets further than expected. The viewer sees both control and avatar object arriving in the gaining region.
• It fails in Sharpview the same way it does in the C++ viewers. So it really is failing sim-side.
• At the message and event level, nothing is visibly wrong. Things are just stuck.

[2024 - Will LL finally fully acknowledge the Adult side of SL?]

3 hours ago, elleevelyn said:

A person is sailing their boat. meets up in the middle of the ocean with another boat. ...

and if there was a ocean region with the radio stream turned on then would soon become a social centre. Similar to Siren Island in the Blake Sea

Zindra has about 30 regions of open ocean, and almost no one uses them. Nobody knows where the rez zones are. (Mostly at the Port of Kama City, but they're not marked and hard to find.) There's a rez zone at Mullinax, with a sailboat rezzer. There may others I haven't found.

The shores of the Blake Sea have many places for sailors - marinas, boat shops, bars, yacht clubs, and such. Zindra has few. There's the Slow Burn Saloon in Philsen. Many docks, no rez zones, and no object entry, so you can't go there by boat. There used to be an LCC location, but it's gone.

Sailing in Zindra just never got started.

Zindra is not about sex. Zindra is about land-flipping.

 

[What do I replace my Christmas tree with?]

Just let the Belissarian Department of Solid Waste dispose of it.

Saturday is trash day.

(I''ll delete it when the trash is emptied. The bin people might offer a "big trash" script to drop into unwanted items. Put junk next to a BelliBin, and when the bin is emptied, the big trash item would be deleted, too.)

[Objects being returned to Lost and Found for no reason.]

There have been two reports of recent object disappearances in New Babbage. But that place has a complicated permissions structure. Is this on land owned outright from Linden Lab, with no one else having permissions?

[What are you watching today?]

Started the Chinese version of The Three Body Problem, but it's very slow paced.

[Shot down by zero-second security orb on Xmas Eve]

Shot down on Xmas Eve by this, flying over Bellessaria. No warning. I was having a nice, relaxing flight, and was getting close to my own place there.

I thought those zero-second orbs had been cleaned out of Belli. Somehow this one was missed. Found the orb, filed the usual abuse report.

66 kills from one orb! I'm tempted to give the owner a lump of coal. But no. It's usually people who didn't read the instructions, not hostility.

[Driving looking for Christmas Lights?]

The Bellessarian Bureau of Bureaucracy Gazette has some suggested places.

[Time for Mainland 2.0]

How about a new covenant for new mainland regions? Borrow from the Bellessaria covenant, which seems to be working well. At least the Skybox and ban line rules, which are objective.

Then add a few new mainland regions, put them on sale, and see how it works out.

I'd like to see the southeast quadrant of Zindra filled in. There's a hole where four roads dead-end and, if you cam off-sim, you can see the raw edge of the world. The western edge of Bay City has a similar raw edge. Both of those would be high-value real estate if created, and it would be interesting to see how they are used. Auction them off and see what happens.

Land auctions should extend the time to at least one minute after the last bid, to reduce auction sniping.

[MessageLinked with specific link numbers]

Unfortunately, the viewer isn't told about link numbers. The messages from simulator to viewer don't have link numbers. The viewer just numbers them in the order it is told about them. Viewer side link numbers tend to agree with the simulator side link numbers, but there is no guarantee. The viewer doesn't need link numbers; it has a totally different system for identifying child prims and their parents.

Name your child prims. The viewer and simulator both have good info about names. This is mostly a problem when all the child prims are named "Object". See this old article about how to look up by name in a script.

[SecondLife tech myths]

2 hours ago, Ardy Lay said:

I did not use any of the Second Life Viewer's metric displays to evaluate the performance of my network or the performance of the viewer when I deliberately delayed delivery of packets via my network.  I used the same external tools I use when measuring performance of voice and video transports.

The Second Life Viewer's behavior as experienced by the user is all that I cared about.  As data was delayed on the way to the viewer, I, the user, could 'feel' the delays in on-screen updates in the location of objects and avatars.  There are two interpolation mechanisms in the viewer that have some effect on this.  You know about them as you have described them in great detail in the past.  I left them at their defaults.

It's interesting to note that, yes, some packets may arrive too late to be used and may be counted as "lost" by the viewer, and that some message types sent via UDP will be resent if missed, either by loss or excessive delay.  Mostly I was trying to get a feel for how network characteristics affected the end-user experience on Second Life.  I ran all the same tests on several other applications.  Telnet and SSH really sucked to use when delay was introduced.  Voice was okay with some delay except for the time-shift issues that caused participants to overlap and collide when the delay was more than 120ms.  One-way video gives no hoots about delay.  ALL of them sucked when loss was introduced along with the delay, because of the timeouts and additional delays with any retransmissions.  Most voice and video transmissions tested did not loss well.  SRT introduced delays by buffering the video and requesting retransmission of missing data.  It tolerated up to 20% loss unless the bandwidth available introduced so much delay that the retransmitted data arrived too late to be played, limited by the allocated buffer size.  SRT with elastic-buffering just kept increasing the depth of the buffer until it hit a limit and crashed.  HLS, however, just makes the user wait for the data to arrive, no matter how long that might take, at it is downloading files to play sequentially.  

With the Second Life Viewer, which I personally have used for many hours a day since March 16, 2008, loss in-flight ALWAYS sucks when that which is lost is required.  You know the messages...  The ones that result in retransmissions and timeouts, up to and including disconnects.  However, if the round-trip time is reasonable and there is sufficient bandwidth to carry the retransmissions without causing additional loss, some retransmission activity can be tolerated.  What seems to annoy me the most is the loss or delay for "course updates" that result in increased inaccuracy of avatar position and the loss of "full updates" that result in objects, including avatars, sticking around in the viewer when they should be gone and incidents of some objects, and yes, sometimes entire avatars, not appearing when they should be visible.

I, however, wasn't adding the complexity of teleporting and region crossing.  As you have described in excruciating detail, these activities are way more sensitive to network foibles and infidelities, resulting in much anguish over disconnections.  I greatly appreciate the efforts being made to find the causes of these sensitivities and correct or mitigate them. 

That's about what I see. I've also tested introducing delay in the past. See my 5 year old video. Back then, LL people were complaining that region cross failures could not be reproduced, and so they could not fix the problem. So I came up with a way to generate a double region crossing fail every time, by adding a full second of network latency using Linux tools.

The retransmission time calculation in the "reliable UDP" system may need some work. In Sharpview, I start retransmits at 2*RTT and back off from there. I think the LL code has a retransmission time constant, set decades ago, and it may be too large. Something to be looked at after event poller reliability gets fixed. Monty Linden has been looking at the event poller, and finding ancient low-level code that needs work.

Some of this is a viewer architecture issue. The viewer tries to cope with message overload  that would reduce the frame rate too much. Some of that is done by dropping some messages and letting the retransmit system send them again. That's why, when frame rate goes down, "packet loss" goes up. That's the viewer, not the network. In a mostly single thread viewer, the question is always how much time can you steal from the rendering thread. That has to be limited. Remember, when SL was designed, nobody had multiple CPUs on consumer desktops. Today, everybody has lots of them, even on mobile.

In Sharpview, incoming messages are queued and handled in a separate thread. Received packets are never dropped. One statistics graph I display shows the queue length. It's usually length 0, or, occasionally, 1, indicating that updates to the world are caught up. If that queue length starts to climb, something is wrong. Same problem, different approach. Scales to machines with multiple CPUs, which is the point.

There's a standards group working on metaverse standards. Linden Lab is not a member. But everyone from Autodesk to W3C is. Their big push for 2023 was on glTF and USD standardization for metaverse content. There's a networking subgroup, and I've been sending in some comments.

[Somebody finally gets a mesh lowest LOD right.]

Ouch. I seem to have upset some people. Sorry.

My point here is that LOD transitions in SL tend not to be invisible. There's a lot of flickering in the distance in SL. I'm looking at how to fix that. Most game LOD systems do a cross-fade, rather than a cut, on an LOD switch. SL viewers do not cross-fade, but they could. I'm looking at whether it's worth the trouble to implement. The illusion only holds up if the different LODs match well.

[Somebody finally gets a mesh lowest LOD right.]

Tree at center is at highest LOD. This is clipped out of a long shot.

 

Tree at center is at lowest LOD. This is a transition from a full 3D model to a 2D impostor with just two flat surfaces.

Note slight blurring. That's the only visual difference. The impostor and 3D model line up properly.

Now this is level of detail done right. It's standard in video games, but not that common in SL.
If we could do the transition with a fade out of one and a fade in of the other, nobody would notice.

(This is from some Sharpview testing. Location is in Electra, in Heterocera.)

[SecondLife tech myths]

4 hours ago, Ardy Lay said:

SL works on slow networks as long as there is no packet loss.  It's an awkward balance but I have done it.  And by "slow networks" I don't mean garbage multi-hop wireless links.  I did say "no packet loss."  This must include UDP traffic. 

"Packet loss" in SL is strange. What the viewers report as "packet loss" includes packets dropped by the viewer because the viewer is falling behind. Also, "ping time" as reported by the viewer reflects time until a packet is processed along with the next display frame. This is why slow frame rates result in higher reported ping times.

If you slow the network down, it may reduce the packet loss rate in the viewer, because the viewer has less packets to process per frame.

Latency is a separate issue. Excessive delay will break some things. A round trip time of 1 second will break double region crossings with vehicles every time.

UDP packets are (mostly) retransmitted if lost, by SL's "reliable UDP" system. There are a limited number of retries. There are also "unreliable" UDP messages, mostly the ones that indicate something moved. You want the last one of those, not one from the past, so retransmitting where the avatar was ten frames ago is not useful.

It's recently been discovered that the event poller, which is a slow TCP poll, is much less reliable than it should be. Monty Linden is working on this. Henri Beauchamp and I have some workarounds.

If you dig deeply enough into this, some unexplained SL behavior turns out to be low level bugs.

[2024 - Will LL finally fully acknowledge the Adult side of SL?]

3 hours ago, Scylla Rhiadra said:

But the key point, surely, is that it will only be able to spit out the kind of information one can already find online. What it's NOT going to be able to do is hold a noob by the hand (so to speak) and lead them through a process. "Can you show me a screenshot of your inventory" is not going to be an option here.

That's the basic problem with large language models at this point. They're good at general information, if it's in the training set. Coupling them up to the "real world" (virtual or otherwise) for context is not yet working well. It's not impossible, but it's a research topic, not just buying a ChatGPT account. Lots of interesting research going on in game dev AI. Give it a year.

[2024 - Will LL finally fully acknowledge the Adult side of SL?]

19 minutes ago, Zalificent Corvinus said:

Does this mean LL have figured out the unpaid mentors don't really work as well as they hoped?

Is the mentor program over? I haven't seen a mentor at WelcomeHub over three visits in the last two weeks.

[New Business Model]

I've seen a stable where each paddock was a separate parcel. And parking spaces with parking meters! Like rent boxes, but with time limits in hours. But usually, tiny parcels are for some ad scheme.

[Tried to cash out Lindens]

Tilia is a regulated money transfer service. They don't get to screw up without consequences to them.

The form for filing a complaint with the state of California is here.