animats

A missing RegionHandshake message means a region won't appear in the viewer. So one can see how a fix like that might go in. Worse, from a sim-side perspective, if a viewer fails to send RegionHandshakeReply to the viewer (an early version of Sharpview did that), it leaves the avatar sim-side in a bad state. On the Other Simulator, that lasts until sim restart, and it takes two logins to clear. SL seems to time out after a while. Sims really need that RegionHandshakeReply. If there's some way a RegionHandshakeReply could be lost or not be processed sim-side, that might justify sending extra RegionHandshake messages to get things unstuck. There's still that 10 seconds / 1 minute stall around EstablishAgentCommunication. That's the next handshake that has to complete to get a region talking to the viewer. Amusingly, entering the region with the avatar in Sharpview will consistently generate an EstablishAgentCommunication message from the simulator quickly. I do this all the time at the Morris corner on the beta grid to get all four regions to go live quickly. Looks awful, but consistently works for testing. So, whatever isn't sending EstablishAgentCommunication is ready to send it.

Yes. Some changes will require re-generation. That's true of BOM now.

You can still have rigged mesh doing its thing. The GPU does the work on rigged mesh.

From my perspective with Sharpview, SL non-avatar content complexity is a manageable problem. Land impact keeps things from getting so bad the GPU can't handle them. Avatars, though... To solve this, we need some kind of mesh baking that happens on a clothing change. Like bakes on mesh, but operating on the meshes themselves, not just textures. Simplify stacked layers. Combine meshes. Discard triangles that can't possibly appear. Mesh reduce. Bake down to a one-piece animesh. There are tools for this, but not for SL. Anyone tried Mesh Baker?

There are resources on line regarding intellectual property for game developers. Use Google. This litigation in the "quiet luxury" fashion sector is worth following. The question is whether you can make a duplicate of some overpriced branded item minus the logo. On the other hand, there isn't that much originality in fashion. If you see some novel runway design today, there's probably something like it somewhere in the archives of the Fashion Institute of Technology. It's people who slavishly duplicate the details of branded fashion that get in trouble. In practice, companies seem to pursue things that copy their branding rather than their actual design. Trademark protection is strong and infringement is well-defined. That's a bit much. Do something original, or transformative, or something.

That's not too far off. What we can't see as users is the sim to sim traffic. Looking at viewer traffic alone yields a blind-men-and-the elephant problem. When the vehicle arrives in Sim C, it should be carrying with it enough information to know what avatars are supposed to be on board. It's not clear if avatars are pushed (Sim B sends them to sim C) or pulled (Sim C asks sim B to send them over.) If they are pushed, maybe a back-up pull mechanism is needed. Sim C to Sim B - I have the vehicle, send me Vehicular Idiot. Sim B might have to wait for Vehicular Idiot to fully arrive from A before starting the transfer. I've been talking to the Other Simulator devs about how they do this, and doing some testing over there. They have set up some test regions. Maybe we can get this fixed for the Other Simulator, where we can see all the logs.

I've added more logging in Sharpview, and I can now see region crossing failures in some detail. This is all about double region crossings - single crossings are working fine. Double region crossings are working most of the time. I'm seeing two failure modes so far. First, there's the simulator explicitly kicking an avatar off a vehicle. This I log as: 19:51:34 [WARN] Region crossing: agent object parenting change from Some(264974690) to None in region [Bronlen] at Place { position_relative: <1.0641974,254.99876,37.0042>, rotation_relative: <-0,-0,-0.97578573,0.21872917> } This says that an ObjectUpdate came in for the the avatar with no parent (local id = 0), when the avatar previously had a parent. This is an unsit - the avatar was sitting, and now it's not. A simulator initiated this. TPVs can look for this and log it. A normal "stand" would also generate this message, but that's not what's happening here. This does not happen, even temporarily, in successful region crossings. So this is an explicit wrong message from a sim. This is something LL could check for simulator-side. If an avatar shows this kind of unsit, and it's on top of a region corner, the recent history of the simulator should show something wrong. So, knowing this, the bug is easier to find. I have more logging info if anyone needs it. Second, there's another kind of error. 04:42:57 [WARN] Region crossing out of sync: agent object is in (462592,306944) but control is in (462848,306688) 04:42:57 [WARN] Region crossing out of sync: agent object is in (462592,306944) but control is in (462848,306688) 04:42:57 [WARN] Region crossing out of sync: agent object is in (462592,306944) but control is in (462848,306688) 04:42:58 [WARN] Region crossing out of sync: agent object is in (462592,306944) but control is in (462848,306688) 04:42:58 [WARN] Region crossing out of sync: agent object is in (462592,306944) but control is in (462848,306688) There are three moving parts in a vehicle region crossing, as seen by the viewer. (Sim side has more moving parts, but we can't see them in a viewer.) The moving parts are the avatar object (the "main agent"), which is what gets drawn on screen, the vehicle (which is itself an object), and control (who's getting the keyboard and mouse input.). Those all change regions separately, and not all at once. The error above shows that control was passed to a new region but the avatar object didn't make the trip. It's normal to see that once or twice in a region crossing, but this time it got stuck. This is also something both a viewer and the simulator can detect as an error. If that out of sync condition persists, something has gone wrong. This is not a bug fix. This is not even the beginning of a bug fix. But it is, perhaps, the end of the beginning of looking for something concrete to fix. This problem is moving from mysterious and un-knowable toward becoming understandable. Henri was right. We can now see something explicitly going wrong, not just an absence of activity.

The Belessarian Bureau of Bureaucracy has a room like that. They even have a machine for getting a number ticket for your queue position. Stand behind the line until your number is called. It never is. Crack Den has vehicle registration, and it's rather expensive, because they treat it as renting prims.

Yes. If an avatar sits on a phantom object, the avatar also becomes phantom and can go through things.

Yes. The comment I usually make on this is to set up a little office in Second Life and do interviews there. Rent a space in someplace popular for a month. London City has space. Put in some office furniture and informational displays, put up a sign with office hours, and see who wanders in. You can probably do the whole job for under US$50. Maybe try four locations for a week each over the course of a month - a new user region such as London City, a shopping region, a roleplay region, and maybe an adult region. This isn't one homogeneous game. It's an area about the size of Los Angeles, and most activity is local. If you have an in-world presence, people will take you seriously. Go to busy clubs and and stores, and hand out invites to be interviewed. Offer people L$100 just to show up. That's about US$0.40, and that's enough to get people in the door. Give them a T-shirt if they do the interview. (How to make custom T-shirts.) These forums are not read by many users. Mostly creators, developers, and landlords are on here. This person seems to want consumers, so they need to go to shopping events.

If you get this figured out well, please post details, and a tutorial. Many people need to do this, and few know how.

Users should not need to know this stuff. It should Just Work. I'm impressed with how well the single-thread viewers cope with overload. It's technology from the early 2000s, before everybody had lots of CPUs, but it's good technology from that era. The people who architected this thing were really good. You can build the SL viewer from source with Tracy tracing enabled, if you want. It's a build option. Then you can run the Tracy profiler while the viewer is running and capture this data. Firestorm can also be built that way.

A useful way to think about the problem is this. The viewer has four main jobs that use substantial time. Drawing the frames Handling movement interpolation between frames. Receiving updates from the network and acting on them. Fetching assets. Classic SL viewers do all this in one thread. #4 has been moved out to other threads in some viewers. There's a "coroutine" system, or what would today be called "async", but it only uses one CPU. So the normal situation is one compute bound thread, with 1), 2), and 3) contending for time. This works amazingly well, considering the constraints. There are lots of little algorithms in the classic viewers trying to load-balance those tasks. In Sharpview, all four of those tasks are done in different threads, so we can get multiple CPUs working on the problem. It then becomes much clearer who's falling behind. This isn't magic. It requires substantial inter-thread coordination, and introduces new problems. There are lock contention problems, where two threads need to manipulate some shared item. The goal is to prevent the drawing thread from waiting for the other threads. That's not the only problem. I've had troubles with the movement thread falling behind. This shows up as high frame rate but jittery movement. Enough is now working in Sharpview that I can start to work on such performance issues. Eventually, if and when LL switches to Vulkan, the C++ viewers might, someday, be updated to work that way. It won't be easy. Vulkan lets you change the contents of the GPU while rendering is in progress. All the levels above Vulkan have to cooperate to make that useful. Graphics systems which have been through that conversion took years of work by large teams to do it. May not be feasible as a retrofit. The rendering stack I use (Rend3/WGPU/Vulkan) just went through that, and it took most of a year. Converting Unity over was reportedly a huge effort. I wonder how the mobile viewer does it.

There are tools for finding out what's really going on. Putting a microscope on a slow frame in Sharpview's rendering library, Rend3/WGPU. This is a tool called Tracy. You can record several minutes of activity, and then zoom in on the log down to the nanosecond level. The line at the top shows how long each frame took. You can pan and zoom and see in detail what happened in each frame. You can build the SL viewer with this level of tracing enabled. It will slow down by a few percent. Unless you're doing viewer development, you probably don't want this level of detail. The point here is that performance issues are not forever mysterious and un-knowable. There are ways to find out.

Yes, the switches and signals need work. In practice, everybody uses phantom trains that pass through each other. Solid trains do work, but they are rare. There's no good way to detect train presence for an entire block. Either you need active responses from all trains (the automatic trains on Heterocera do this, and there's a control center with a map in the tower at Cretopia Station), or you need detectors in each section of track, reporting to a trackside controller at the interlockings at both ends. To make it work realistically, each siding would give you a green signal for exit only when the track was empty all the way to the next siding, and one branch at the next siding was empty. Then you could drive solid trains without conflicts. Sometimes, though, you'd be stuck "in the hole" waiting for someone else to free up the track section ahead. For a real world view, see "Rail Cow Girl" on Youtube. Long cab videos from a train driver in Norway. Norway's rail system is almost all single track with occasional passing sidings, and you can see how a real railroad like that works. Opposing trains meet at sidings; the first one goes on the side track, the other one goes past, and the side-tracked train comes out. It's do-able, and would appeal to the tiny number of people who read Model Railroader. For now, I'd settle for someone making a track inspection vehicle that finds places where the guides are out of spec or switches are broken, so you could reliably drive phantom trains without derailments at broken switches.

That is just precious.

If it can be done efficiently and securely with a script, it should be done that way. llRezObjectWithParams was needed because the overhead of starting a script for each bullet was slowing things down. There may be a need for built-in damage tallying, because that has to happen on each hit. Beyond that, LSL scripts should do the work. The main security problem is making sure all players are using the same script. That can be done through the experience system. Different combat areas will want different scripts, of course. Are there other system functions that need to be optionally locked out by an experience or parcel? Parcels can already lock out "fly". How about "click to teleport?" Camming? Other than first person viewpoint? Wireframe mode? Player locations on the map? It's really hard to change all the simulators and viewers, but it's easy to change scripts. The system should have the bare-bones features upon which combat systems can be built, but not, itself, be a shooter game.

My 2019 comment on how to drive in SL. I tell people to head for Robin Loop and try cars from the many free car rezzers there. Once you can drive smoothly around Robin Loop without problems, you're good for any road in SL.

Ah, I'm at Wine 8.1, the default shipped with Ubuntu 22.04 LTS. Time to try newer Wine versions. Thanks.

Of the LL viewer, or of Wine? Please give versions of everything. Thanks.

No, please don't do that. You'll have a sensor running all the time for each tile, which eats up sim resources. If you want to see glow when walking on it, visit Oxymoron, the Graves store. They seem to be detecting collision starts, and triggering something, probably a simple animation, to make the tile glow and then fade out.

That's what I was hoping to find in all those logs - some message missing or wrong. But it doesn't look that way. A failed double region crossing looks like one successful crossing followed by the second crossing not starting at all. Yes. This is going to require tests in quiet sims on the beta grid with LL's internal sim to sim logging turned on. But we know what to look for now - viewer sees first crossing went fine, second crossing never started.

Huh. What are you thinking is missing or mis-timed? For anyone who is really, really bored, and knows how to read viewer logs, here's Sharpview's log of a few successful double region crossings followed by a failed crossing. Formats are different, but it's not that different from what you'd get in an LL or TPV log. Notes: Lines beginning "From:" are traffic from sim to viewer. "Message" means it was a UDP message. Just the message type is printed, because many of those are big. "Event" means it came in via an event message poller. The two numbers in parentheses are the location of the sim in-world that sent the message. Lines beginning "To:" are UDP traffic from viewer to sim. All the data is displayed. Look for CrossedRegion to see activity around region crossings. In the vicinity of those events, the other activity that makes a region crossing go will show up. Also look for RegionHandshake and EnableSimulator messages. All four regions at the corner have been live for minutes at this point, so there's no good reason for those to have been sent to the viewer. This is a viewer under development, and you'll see messages about message processing not yet implemented. Yes, I know this is all terminally boring, but it's how stuff gets fixed. (I'm still getting over some non-COVID flu-like virus, I'm not going anywhere, and it's rained a lot lately.)

I don't think we'll see this in the viewer traffic. This looks like a sim to sim problem, which we, as users, cannot see.

OK, more boring stuff to finish out the year. Here's an annotated log of several successful double region crossings, and one failed one. This is from Sharpview. ====> Beginning of successful double region crossing <==== ====> Thes are all at the same corner. ====> [Bronlen] (462848,306688) ====> [Charlesville] (462848,306944) ====> [Vallone] (462592,306944) ====> [Kama Center] (462592,306688) ====> This first one is from Vallone to Bronlen via Charlesville. 02:49:40 [WARN] From (462592,306944), event CrossedRegion: MsgServerEvent { region_handle: (462592,306944), event: CrossedRegion(EventCrossedRegion { look_at: <0.392944,-0.919562,0>, position: <1.00946,0.236431,35.6884>, socket_addr: 34.222.128.86:12035, seed_capability: "https://simhost-05f87cd2d2a7b0b7a.agni.secondlife.io:12043/cap/be1edc3a-5ba8-4b50-7ffc-b75380dcec9c", region_box: RegionBox { region_handle: (462848,306944), size: [256, 256] }, session_id: a0f40a7c-a5f3-11ee-9453-0242ac110003, agent_id: b5fde908-eebf-4505-b686-ca74c3f91979 }) } 02:49:40 [WARN] CrossedRegion: leaving region (462592,306944), entering live region [Charlesville] (462848,306944) 02:49:40 [WARN] CrossedRegion from (462592,306944) to (462848,306944), distance <256,0,0> 02:49:40 [WARN] CrossedRegion from [Vallone] ====> Vallone sim has handed over control of the avatar to Charlesville. Fine. ====> These errors are some late object updates straggling in from Vallone. That seems to be normal. Those are ignored. 02:49:40 [ERROR] Main agent object is in region (462592,306944) but current CrossedRegion region is (462848,306944) 02:49:40 [ERROR] Main agent object is in region (462592,306944) but current CrossedRegion region is (462848,306944) 02:49:40 [ERROR] Main agent object is in region (462592,306944) but current CrossedRegion region is (462848,306944) 02:49:40 [ERROR] Main agent object is in region (462592,306944) but current CrossedRegion region is (462848,306944) 02:49:40 [ERROR] Main agent object is in region (462592,306944) but current CrossedRegion region is (462848,306944) ====> Here are some seemingly bogus EnableSimulator messages. This is Charlesville telling the viewer to start talking to Bronlen and Kama Center. ====> The viewer has been talking to all four regions for several minutes at this point, so this is redundant. Harmless, though. 02:49:41 [ERROR] Event error: Duplicate add of region (462848,306688), event MsgServerEvent { region_handle: (462848,306944), event: EnableSimulator(EventEnableSimulator { region_box: RegionBox { region_handle: (462848,306688), size: [256, 256] }, socket_addr: 35.87.129.151:13013 }) } 02:49:41 [ERROR] Event error: Duplicate add of region (462592,306688), event MsgServerEvent { region_handle: (462848,306944), event: EnableSimulator(EventEnableSimulator { region_box: RegionBox { region_handle: (462592,306688), size: [256, 256] }, socket_addr: 34.215.138.142:13007 }) } 02:49:42 [WARN] From (462848,306944), event CrossedRegion: MsgServerEvent { region_handle: (462848,306944), event: CrossedRegion(EventCrossedRegion { look_at: <-0.332161,0.943223,0>, position: <4.8429,254.94,35.6402>, socket_addr: 35.87.129.151:13013, seed_capability: "https://simhost-043e6020eb4baf499.agni.secondlife.io:12043/cap/60ea19c3-0600-310b-3f10-56c9e008f05a", region_box: RegionBox { region_handle: (462848,306688), size: [256, 256] }, session_id: a0f40a7c-a5f3-11ee-9453-0242ac110003, agent_id: b5fde908-eebf-4505-b686-ca74c3f91979 }) } 02:49:42 [WARN] CrossedRegion: leaving region (462848,306944), entering live region [Bronlen] (462848,306688) 02:49:42 [WARN] CrossedRegion from (462848,306944) to (462848,306688), distance <0,-256,0> 02:49:42 [WARN] CrossedRegion from [Charlesville] 02:49:42 [ERROR] Main agent object is in region (462848,306944) but current CrossedRegion region is (462848,306688) 02:49:43 [ERROR] Event error: Duplicate add of region (462592,306688), event MsgServerEvent { region_handle: (462848,306688), event: EnableSimulator(EventEnableSimulator { region_box: RegionBox { region_handle: (462592,306688), size: [256, 256] }, socket_addr: 34.215.138.142:13007 }) } =====> Here's another successful double region crossing - Bronlen to Vallone via Charlesville, the reverse of the previous move. =====> This is a motorcycle with avatar going forwards and backwards right across the region corner. 02:49:58 [WARN] From (462848,306688), event CrossedRegion: MsgServerEvent { region_handle: (462848,306688), event: CrossedRegion(EventCrossedRegion { look_at: <-0.504536,0.863391,0>, position: <0.915858,1.00684,35.6747>, socket_addr: 34.222.128.86:12035, seed_capability: "https://simhost-05f87cd2d2a7b0b7a.agni.secondlife.io:12043/cap/be1edc3a-5ba8-4b50-7ffc-b75380dcec9c", region_box: RegionBox { region_handle: (462848,306944), size: [256, 256] }, session_id: a0f40a7c-a5f3-11ee-9453-0242ac110003, agent_id: b5fde908-eebf-4505-b686-ca74c3f91979 }) } 02:49:58 [WARN] CrossedRegion: leaving region (462848,306688), entering live region [Charlesville] (462848,306944) 02:49:58 [WARN] CrossedRegion from (462848,306688) to (462848,306944), distance <0,256,0> 02:49:58 [WARN] CrossedRegion from [Bronlen] 02:49:58 [ERROR] Main agent object is in region (462848,306688) but current CrossedRegion region is (462848,306944) 02:49:59 [WARN] From (462848,306944), event CrossedRegion: MsgServerEvent { region_handle: (462848,306944), event: CrossedRegion(EventCrossedRegion { look_at: <-0.504536,0.863391,0>, position: <254.984,2.26188,35.6209>, socket_addr: 34.212.163.17:13019, seed_capability: "https://simhost-04148d0076d95659a.agni.secondlife.io:12043/cap/2de81e9e-b7a6-53aa-66c8-f6bd753aa6c2", region_box: RegionBox { region_handle: (462592,306944), size: [256, 256] }, session_id: a0f40a7c-a5f3-11ee-9453-0242ac110003, agent_id: b5fde908-eebf-4505-b686-ca74c3f91979 }) } 02:49:59 [WARN] CrossedRegion: leaving region (462848,306944), entering live region [Vallone] (462592,306944) 02:49:59 [WARN] CrossedRegion from (462848,306944) to (462592,306944), distance <-256,0,0> 02:49:59 [WARN] CrossedRegion from [Charlesville] 02:49:59 [ERROR] Main agent object is in region (462848,306944) but current CrossedRegion region is (462592,306944) 02:49:59 [ERROR] Main agent object is in region (462848,306944) but current CrossedRegion region is (462592,306944) 02:49:59 [ERROR] Main agent object is in region (462848,306944) but current CrossedRegion region is (462592,306944) 02:49:59 [ERROR] Main agent object is in region (462848,306944) but current CrossedRegion region is (462592,306944) ====> And another successful double crossing, Vallone to Bronlen via Charlesville.==== 02:50:21 [WARN] From (462592,306944), event CrossedRegion: MsgServerEvent { region_handle: (462592,306944), event: CrossedRegion(EventCrossedRegion { look_at: <-0.0494177,0.998778,0>, position: <1.00549,0,35.6919>, socket_addr: 34.222.128.86:12035, seed_capability: "https://simhost-05f87cd2d2a7b0b7a.agni.secondlife.io:12043/cap/be1edc3a-5ba8-4b50-7ffc-b75380dcec9c", region_box: RegionBox { region_handle: (462848,306944), size: [256, 256] }, session_id: a0f40a7c-a5f3-11ee-9453-0242ac110003, agent_id: b5fde908-eebf-4505-b686-ca74c3f91979 }) } 02:50:21 [WARN] CrossedRegion: leaving region (462592,306944), entering live region [Charlesville] (462848,306944) 02:50:21 [WARN] CrossedRegion from (462592,306944) to (462848,306944), distance <256,0,0> 02:50:21 [WARN] CrossedRegion from [Vallone] 02:50:21 [ERROR] Main agent object is in region (462592,306944) but current CrossedRegion region is (462848,306944) ... 02:50:22 [ERROR] Main agent object is in region (462592,306944) but current CrossedRegion region is (462848,306944) 02:50:28 [WARN] From (462848,306944), event CrossedRegion: MsgServerEvent { region_handle: (462848,306944), event: CrossedRegion(EventCrossedRegion { look_at: <-0.0494177,0.998778,0>, position: <2.10215,254.998,35.6366>, socket_addr: 35.87.129.151:13013, seed_capability: "https://simhost-043e6020eb4baf499.agni.secondlife.io:12043/cap/60ea19c3-0600-310b-3f10-56c9e008f05a", region_box: RegionBox { region_handle: (462848,306688), size: [256, 256] }, session_id: a0f40a7c-a5f3-11ee-9453-0242ac110003, agent_id: b5fde908-eebf-4505-b686-ca74c3f91979 }) } 02:50:28 [WARN] CrossedRegion: leaving region (462848,306944), entering live region [Bronlen] (462848,306688) 02:50:28 [WARN] CrossedRegion from (462848,306944) to (462848,306688), distance <0,-256,0> 02:50:28 [WARN] CrossedRegion from [Charlesville] 02:50:28 [ERROR] Main agent object is in region (462848,306944) but current CrossedRegion region is (462848,306688) ... 02:50:29 [ERROR] Main agent object is in region (462848,306944) but current CrossedRegion region is (462848,306688) ====> And another successful double region crossing, back from Bronlen to Vallone via Charlesville 02:51:08 [WARN] From (462848,306688), event CrossedRegion: MsgServerEvent { region_handle: (462848,306688), event: CrossedRegion(EventCrossedRegion { look_at: <-0.744412,-0.667721,0>, position: <0,1.00766,35.6432>, socket_addr: 34.222.128.86:12035, seed_capability: "https://simhost-05f87cd2d2a7b0b7a.agni.secondlife.io:12043/cap/be1edc3a-5ba8-4b50-7ffc-b75380dcec9c", region_box: RegionBox { region_handle: (462848,306944), size: [256, 256] }, session_id: a0f40a7c-a5f3-11ee-9453-0242ac110003, agent_id: b5fde908-eebf-4505-b686-ca74c3f91979 }) } 02:51:08 [WARN] CrossedRegion: leaving region (462848,306688), entering live region [Charlesville] (462848,306944) 02:51:08 [WARN] CrossedRegion from (462848,306688) to (462848,306944), distance <0,256,0> 02:51:08 [WARN] CrossedRegion from [Bronlen] 02:51:08 [ERROR] Main agent object is in region (462848,306688) but current CrossedRegion region is (462848,306944) 02:51:08 [ERROR] Main agent object is in region (462848,306688) but current CrossedRegion region is (462848,306944) 02:51:08 [ERROR] Main agent object is in region (462848,306688) but current CrossedRegion region is (462848,306944) 02:51:14 [WARN] From (462848,306944), event CrossedRegion: MsgServerEvent { region_handle: (462848,306944), event: CrossedRegion(EventCrossedRegion { look_at: <-0.744412,-0.667721,0>, position: <254.998,2.55342,35.6153>, socket_addr: 34.212.163.17:13019, seed_capability: "https://simhost-04148d0076d95659a.agni.secondlife.io:12043/cap/2de81e9e-b7a6-53aa-66c8-f6bd753aa6c2", region_box: RegionBox { region_handle: (462592,306944), size: [256, 256] }, session_id: a0f40a7c-a5f3-11ee-9453-0242ac110003, agent_id: b5fde908-eebf-4505-b686-ca74c3f91979 }) } 02:51:14 [WARN] CrossedRegion: leaving region (462848,306944), entering live region [Vallone] (462592,306944) 02:51:14 [WARN] CrossedRegion from (462848,306944) to (462592,306944), distance <-256,0,0> 02:51:14 [WARN] CrossedRegion from [Charlesville] 02:51:14 [ERROR] Main agent object is in region (462848,306944) but current CrossedRegion region is (462592,306944) 02:51:14 [ERROR] Main agent object is in region (462848,306944) but current CrossedRegion region is (462592,306944) ====> Beginning of a failed double region crossing <==== ====> This time, though, we start in Vallone and go through Kama Center, rather than Charlesville. Stuck in Kama Center. Bronlen not talking. <==== ====> Many unnecessary EnableSimulator messages. ====> Need to check if the server is trying to change the IP address or UDP port. ====> These messages are from Vallone, telling the viewer about Charlesville and Kama Center, which are neighbors of Vallone; they share an edge. 02:52:15 [ERROR] Event error: Duplicate add of region (462848,306944), event MsgServerEvent { region_handle: (462592,306944), event: EnableSimulator(EventEnableSimulator { region_box: RegionBox { region_handle: (462848,306944), size: [256, 256] }, socket_addr: 34.222.128.86:12035 }) } 02:52:15 [ERROR] Event error: Duplicate add of region (462592,306688), event MsgServerEvent { region_handle: (462592,306944), event: EnableSimulator(EventEnableSimulator { region_box: RegionBox { region_handle: (462592,306688), size: [256, 256] }, socket_addr: 34.215.138.142:13007 }) } 02:52:20 [WARN] From (462592,306944), event CrossedRegion: MsgServerEvent { region_handle: (462592,306944), event: CrossedRegion(EventCrossedRegion { look_at: <0.0256889,0.99967,0>, position: <255,254.989,35.6911>, socket_addr: 34.215.138.142:13007, seed_capability: "https://simhost-0e6e8ee1f4104630e.agni.secondlife.io:12043/cap/6ec6de3f-73fd-03a5-fe35-f73f1cc09a23", region_box: RegionBox { region_handle: (462592,306688), size: [256, 256] }, session_id: a0f40a7c-a5f3-11ee-9453-0242ac110003, agent_id: b5fde908-eebf-4505-b686-ca74c3f91979 }) } 02:52:20 [WARN] CrossedRegion: leaving region (462592,306944), entering live region [Kama Center] (462592,306688) 02:52:20 [WARN] CrossedRegion from (462592,306944) to (462592,306688), distance <0,-256,0> 02:52:20 [WARN] CrossedRegion from [Vallone] 02:52:21 [ERROR] Main agent object is in region (462592,306944) but current CrossedRegion region is (462592,306688) 02:52:21 [ERROR] Main agent object is in region (462592,306944) but current CrossedRegion region is (462592,306688) 02:52:21 [ERROR] Main agent object is in region (462592,306944) but current CrossedRegion region is (462592,306688) ====> Got to Kama Center OK. Then stuck. ====> Now this is Kama Center telling the viewer about its neighbors, Bronlen and Vallone. ====> Same UDP port as previously used for Bronlen, so the UDP connection is not being restarted. 02:52:22 [ERROR] Event error: Duplicate add of region (462848,306688), event MsgServerEvent { region_handle: (462592,306688), event: EnableSimulator(EventEnableSimulator { region_box: RegionBox { region_handle: (462848,306688), size: [256, 256] }, socket_addr: 35.87.129.151:13013 }) } 02:52:22 [ERROR] Event error: Duplicate add of region (462848,306944), event MsgServerEvent { region_handle: (462592,306688), event: EnableSimulator(EventEnableSimulator { region_box: RegionBox { region_handle: (462848,306944), size: [256, 256] }, socket_addr: 34.222.128.86:12035 }) } ====> First crossing was successful, second one never started. The main thing to note about this is that nothing unusual happened. It just stopped working. Notes: The second crossing at a region corner just never happened, according to the info the viewer sees. All the messages for the first crossing are correct and on the happy path. There's just nothing about a second crossing. To the viewer, it looks like the avatar just never left the second region for the third. There's some "noise". There are unnecessary EnableSimulator events. Sometimes those show up on successful crossings. Sometimes they don't show up. There are also unnecessary RegionHandshake messages. It's not in this log, but there are also unnecessary KillObject messages for objects leaving a sim. Sometimes those are present, sometimes not. Viewers discard all this junk routinely. Henri Beauchamp reports seeing this stuff, too. The "noise" suggests imperfect synchronization between neighboring simulators. We can't tell from the outside, but this may be a symptom of some inter-simulator communication issue. That's speculative. However, it's something concrete that the simulator-side devs could look at. When we all looked hard at EstablishAgentCommunication problems, that turned up things that needed to be fixed. This may be similar. I have a vague suspicion that there may be some kind of low-level inter-simulator timeout or reset which causes some loss of shared state, and those excess messages are side effects of recovery from that. But that's speculation on my part. Sharpview, Firestorm, and Cool VL all log similar message patterns. I've looked at Firestorm logs in the past, and Henri Beauchamp has looked at Cool VL logs. Nothing seems to be wrong. Successful crossings and failed crossings look similar right up to where progress stops. Viewer <-> Sim protocol for all this seems sound. The lockstep handoff for ChangedRegion/CompleteAgentMovement works fine. The state change for RegionHandshake/RegionHandshakeReply works fine. The system by which objects move from region to region works fine. There's a brief moment of confusion because control handoff (who has the keyboard and mouse?) and object movement (where is the avatar?) are not synchronized, but those fall into sync in a second or so.The architects of these parts did a good job. I was hoping to find something clearly wrong that could be fixed or worked around viewer side. Or even just see different behavior in Sharpview than what the C++ viewers provide, which would give us some ideas to follow up. Nope. What's probably needed to fix this is to find four regions on the beta grid where there's a clear corner and very little activity. Then log everything the simulators do while an avatar on a vehicle is moved back and forth across the corner. Then look over the logs in great detail. Somewhere in the sim to sim traffic just before a stall, there will be something of interest. It's not simulator to viewer.