Vehicles vs. sim crossings, why it's so awful

[animats]

Another day, another 100 region crossings without a crash. Looking good. Whatever was broken in Timescape isn't broken today.

Biggest remaining hassle is bad road region crossings.

Rather than fixing this by endless mole work, that could be fixed sim-side. Have each sim duplicate all locked non-phantom objects from neighboring sims within a small distance, say 8 meters. Locked, because locked objects don't move. Roads and buildings are usually locked. Little stuff usually isn't, and moving objects can't be, so the overhead isn't too bad. Duplicates would be transparent, physics model only. Sync objects only when "locked" status changes, or on sim restart.

This would fix all sinking, tripping, bouncing, falling through floors, sideways jerk, and going through walls problems at region crossings grid wide.

Edited March 9, 2018 by animats
More about Timescape.

[Callum Meriman]

5 hours ago, animats said:

Rather than fixing this by endless mole work, that could be fixed sim-side. Have each sim duplicate all locked non-phantom objects from neighboring sims within a small distance, say 8 meters. Locked, because locked objects don't move. Roads and buildings are usually locked. Little stuff usually isn't, and moving objects can't be, so the overhead isn't too bad. Duplicates would be transparent, physics model only.

That's a complex one. Maybe doable, but what is the cost of dev time vs overseeing a mole team?

Looking at an automated solution, maybe if you restrict that to items actually on protected land. Otherwise, if it's the whole 1024M of sim border... there would be a question of "who pays". And if they do take it from the free pool what if it's a huge object that soaks up the free pool and stops vehicles passing through?

Consider it could also be a grief vector, someone locking 2000LI along a mainland sim border to stop all vehicles entering (as the free pool is burnt)

In one of my regions I have a 512LI pirate ship locked in the 10M range. In another I I have a 196LI lighthouse in that area.

Edited March 10, 2018 by Callum Meriman

[animats]

53 minutes ago, Callum Meriman said:

That's a complex one. Maybe doable, but what is the cost of dev time vs overseeing a mole team?

Looking at an automated solution, maybe if you restrict that to items actually on protected land. Otherwise, if it's the whole 1024M of sim border... there would be a question of "who pays". And if they do take it from the free pool what if it's a huge object that soaks up the free pool and stops vehicles passing through?

Consider it could also be a grief vector, someone locking 2000LI along a mainland sim border to stop all vehicles entering (as the free pool is burnt)

In one of my regions I have a 512LI pirate ship locked in the 10M range. In another I I have a 196LI lighthouse in that area.

You have to have unused object quota on your property in one sim to lock them. All this does is mirror them in adjacent sims. 

Mirrored objects don't need to count against the existing quotas. They'd be physics geometry only - they don't have color or texture, they don't render, they don't have scripts, and they're not shipped to viewers. So most of the weight of an object is not mirrored.

[animats]

Where did the textures go?

Now here's a strange bug at a double region cross. The textured objects on the bike didn't make it through a double region crossing. I've had trouble at this region crossing before. This bike is able to make it across. But twice so far, this - only the wheels show up. The bike still drives fine. 

I'm up to about 500 region crossings since the last half-unsit. Testing is going well.

When I went back across the same region crossings, the textured parts came back.

Incidentally, there's some nice abandoned waterfront property with road access on the right there.

Edited March 10, 2018 by animats

[Prokofy Neva]

On 3/9/2018 at 8:32 PM, Callum Meriman said:

That's a complex one. Maybe doable, but what is the cost of dev time vs overseeing a mole team?

Looking at an automated solution, maybe if you restrict that to items actually on protected land. Otherwise, if it's the whole 1024M of sim border... there would be a question of "who pays". And if they do take it from the free pool what if it's a huge object that soaks up the free pool and stops vehicles passing through?

Consider it could also be a grief vector, someone locking 2000LI along a mainland sim border to stop all vehicles entering (as the free pool is burnt)

In one of my regions I have a 512LI pirate ship locked in the 10M range. In another I I have a 196LI lighthouse in that area.

The public should not pay. 

The sim buffers should remain as is.

No one needs this.

Other people's objects should not be duplicated for this wild idea.

The sim seams are good enough.

 

[Callum Meriman]

On 3/10/2018 at 3:34 PM, animats said:

Where did the textures go?

Now here's a strange bug at a double region cross. The textured objects on the bike didn't make it through a double region crossing. I've had trouble at this region crossing before. This bike is able to make it across. But twice so far, this - only the wheels show up. The bike still drives fine. 

heh, yes, I see that extremely rarely on my boats and planes too, it catches up eventually. I have a sneaking suspicion it's viewer side, race condition or dropped packet around updates, testable with right clicking bringing it back if so (makes a mental note to try next time). Very rare edge case though.

Edited March 12, 2018 by Callum Meriman
Clarified on other vehicles, not the bike

[animats]

OK, who wants to test a bike? I need a few more testers of the bike that survives region crossings. IM me and I'll send you a bike.

I'm going to give a trophy to the tester who racks up the longest road trip.

(This bike version doesn't fill up your chat window with debug info, like the previous models. Logging to a server now.)

 

 

[animats]

A few people now have test bikes. I sent one to Klytyna.  Since she's been so critical, I'll get a tough test.

Yesterday, I rode all the way around Heterocera's central bay. I got lost and had to drive on railroad tracks over the mountains at a few points, but made it all the way around without problems. That long bridge with no railings across the mouth of the bay is scary. 21km, 54 region crossings, 50 minutes. A slow trip. Once I know the route better, I should be able to do that faster.

[animats]

Testing is going well.  Results so far:

• These bikes have been into 264 different regions.
• No half-unsits have been logged.
• Problams logged so far are road-related. There's a sim crossing in a tunnel on a mountain road in Heterocera , inside the Great Wall, where you can fall through to a lower level. That's unusual for Heterocera, where the roads are usually solid. Kama City intersections are still awful, but passable.
• Need a better-handling bike design for this technology.
• Camera control needs attention. There's a workaround in the bike to restore the camera position after each sim cross, but that's not enough.
• My testers aren't riding aggressively enough. They're not pushing the limits.

I need more testers. IM me if you want a bike. (I'll be gone Thurs-Sunday.)

[animats]

Finally got a region crossing failure with my fixes enabled, after multiple testers had visited 355 different regions and covered 150km.

Classic avatar-stuck-in-midair after a region crossing fail

Here's the log. I have these bikes logging debug info to a server, as I mentioned previously.

+-------------------------------------------------------------------+----------+-------------+
| serial | eventtype  | msg                                                               | auxval   | region_name |
+--------+------------+-------------------------------------------------------------------+----------+-------------+
|      0 | STARTUP    | animats Resident/Joe Magarac                                      |        0 | Neumoegen   |
|      1 | SITTER     | on prim #1 :animats Resident distance to seat                     | 0.737695 | Neumoegen   |
|      2 | RIDERCOUNT |                                                                   |        1 | Neumoegen   |
|      3 | PERMS      | Got permissions                                                   |        0 | Neumoegen   |
|      4 | CROSSSPEED |                                                                   |  8.83962 | Burns       |
|      5 | CROSSEND   | Region crossing complete in                                       | 0.048092 | Burns       |
|      6 | CROSSSPEED |                                                                   |  19.5944 | Electra     |
|      7 | CROSSEND   | Region crossing complete in                                       | 0.045059 | Electra     |
|      8 | CROSSSPEED |                                                                   |  10.4254 | Grote       |
|      9 | CROSSEND   | Region crossing complete in                                       | 0.131538 | Grote       |
...
|    323 | CROSSSPEED |                                                                   |  16.5563 | Zephyr      |
|    324 | CROSSEND   | Region crossing complete in                                       | 0.189453 | Zephyr      |
|    325 | CROSSSPEED |                                                                   |  15.9909 | Grote       |
|    326 | CROSSEND   | Region crossing complete in                                       | 0.101074 | Grote       |
|    327 | CROSSSPEED |                                                                   |   9.0799 | Electra     |
|    328 | CROSSEND   | Region crossing complete in                                       | 0.091064 | Electra     |
|    329 | CROSSSPEED |                                                                   |  21.9215 | Burns       |
|    330 | CROSSEND   | Region crossing complete in                                       | 0.066406 | Burns       |
|    331 | CROSSSPEED |                                                                   |  16.3735 | Neumoegen   |
|    332 | CROSSSLOW  | Slow region crossing                                              |        0 | Neumoegen   |
|    333 | CROSSSLOW  | Slow region crossing                                              |        0 | Neumoegen   |
|    334 | CROSSSLOW  | Slow region crossing                                              |        0 | Neumoegen   |
|    335 | CROSSFAIL  | Crossing is taking too long. Probably stuck. Try teleporting out. |  10.0271 | Neumoegen   |
|    336 | SHUTDOWN   | Distance traveled (km)                                            |  27.7028 | Neumoegen   |
|    337 | RIDERCOUNT |                                                                   |        0 | Neumoegen   |
|    338 | SHUTDOWN   | Distance traveled (km)                                            |  27.7028 | Neumoegen   |
+--------+------------+-------------------------------------------------------------------+----------+-------------+

All the way around Heterocera without a problem. 27.7 km.  Then, 50 meters from my starting point, a region crossing stalls and I'm up in the air. Not a double region crossing ,just an ordinary one. Neither sim was overloaded. No road problems. Not going very fast.  Location was Neumogen (4,72), crossing from Burns. Hitting that spot again resulting in a smooth region crossing.

The logging and error messages tell us that the avatar became unlinked and separated from the bike, which happens very briefly at every sim crossing, but didn't reconnect as they usually do. The bike detects this, logs the fault, and shuts down. (That's important, because vehicles tend to run away in this situation, causing trouble on the ground and throwing the partially-linked avatar around. With this, worst case is stuck in the air.) Exactly what went wrong isn't clear. A lost packet could do it. One of the Lindens is working on a packet retransmission fix that might fix this sort of thing.

Three teleports and a sit/unsit/sit got me back on the bike OK. Didn't have to log out, didn't lose the bike. That's what I previously automated with my "Seat Belt" attachment. I'm not using that right now. It worked, but all those teleports are a pain, and not everyone is running an RLV-capable viewer. I'd rather not go that route, but it's a plan B. Maybe an extra-cost option for those who want it.

[Zequ Quartz]

Couldn't the next server on your movement vector be sent a early package of all your details to properly rez you once you enter it?  You've just entered a sim at a certain speed/direction and depending on your course in the sim, you will be entering the next sim at a certain point.  A few seconds before exiting the current sim, it sends the next sim on the vector the avatar object.

[animats]

3 hours ago, Zequ Quartz said:

Couldn't the next server on your movement vector be sent a early package of all your details to properly rez you once you enter it?  You've just entered a sim at a certain speed/direction and depending on your course in the sim, you will be entering the next sim at a certain point.  A few seconds before exiting the current sim, it sends the next sim on the vector the avatar object.

You asked this before in February 2017. Klytyna answered you.

What you suggest is what the Linden Labs patent on region crossings suggests they originally wanted to do. As the avatar approaches a sim boundary, a "ghost" was to be created in the new sim, and moved in synchronism with the avatar in the old sim. The state of the avatar was to be copied to the "ghost", and then the ghost would become the primary avatar, while the old primary avatar becomes the ghost. Once the avatar moved away from the sim boundary, the dead ghost was to be deleted.

They didn't really implement all that. The approach in the patent would require a lot more messages back and forth between adjacent sims to keep avatar and ghost in sync. The "saggng through the ground" problem might have been dealt with that way, with support coming from both primary avatar and ghost, and the stall at sim crossings might have been eliminated by that softer approach. Good idea, but harder to do.

What SL really does is simpler. Each region is an island. A sim crossing is a teleport.  When you go off the edge, you're teleported to the next island, without the blackout and  "whoosh" sound. The viewer talks to multiple sims at the same time and creates the illusion that the islands are a seamless world.

This works quite well for walking across a sim boundary. Faster movement, vehicles, and trying to do anything physical across a sim boundary all run into the limits of that approach.

[animats]

Here's another region cross failure like the one from March 21st:

The avatar didn't do the region cross at all.

Here, the bike crossed a single region boundary, and our script made it wait for the avatar to catch up. But the avatar didn't make the region crossing at all. It just sat there. It's not sitting, and it doesn't have controls (arrow keys won't move it) The script detected that the avatar wasn't present, waited 10 seconds, then logged a fault and turned off the bike. Same as last time.

The aftermath is stable and safe. The bike just sits there, and the avatar just sits there. It's not that hell of being slung around wildly in midair while the bike runs away until it hits something. That induces user panic. This is just seriously annoying.

As before, a teleport into the same sim as the bike, near the bike, results in the avatar being re-seated.  Getting off and back on restarts the bike, and then everything works. I have to do the teleport manually; this bike doesn't have all the RLV stuff to do it automatically. I'm trying to do this without using the big RLV hammer.

 It's a well-defined failure mode. Everything is completely normal going into the region cross, and then the avatar just doesn't get moved to the new sim. It seems to be unrelated to speed, a double region crossing, bad road construction, avatar complexity or sim lag. Packet loss on the network is a possibility. Automatic recovery is possible, but clunky.

One of the Lindens is working on speeding up message retransmission on errors. This may avoid some problems where a message is lost, then retransmitted about 10 seconds later. Often, that's too late. I look forward to trying that. Coming soon to the beta grid, I'm told.

Grid-wide experience permissions are coming, which means the ability to do script-controlled teleports and sits if the avatar allows it. That will provide the functions needed to auto-recover from this failure, without RLV. It will be better if this failure mode can be eliminated entirely by sim fixes, but having a plan B can't hurt.

As I've said before, sim crossing failures come from multiple sources. We fix one, trouble happens less often, and the next most common bug appears with more clarity. Onward.

Meanwhile, riding around Heterocera in spring is quite nice. We're putting this technology into more bikes, too. This new one handles better, carries a passenger, and when it makes a region crossing, it waits for all the avatars to catch up. This avoids passengers being left behind at region crossings.

Coming soon, a free bike rezzer for our old blue-stripe model. Commercial versions to follow.

Edited March 29, 2018 by animats
Left in phrase moved elsewhere.

[animats]

A bit more progress late this evening. I've been getting more region crossing failures lately after some minor script changes. I've been trying to understand why. The timing seems very touchy.

The bike script has been set to give up after a region crossing stalls for 10 seconds. This is the delay between "vehicle arrives in new sim" and "avatar arrives", something which is normally about 40 to 80 milliseconds. It then turns off the bike, which includes turning off physics and vehicle mode. I tried setting the time limit to 75 seconds. Then I added 1 second of network lag to force errors. Sometimes, success occurs just after 10 seconds. Longest delay was 59.6 seconds.  I've forced this five times now. Eventually. things unstuck without a half-unsit every time, and bike and driver were able to drive off.

During this hold, the bike script is holding the bike locked in place, with physics off, so the situation is frozen. The bike script finally sees the avatar show up in the new sim, turns physics back on, and you drive off. Without that lock, things usually go downhill during the stuck period, with both vehicle and avatar moving around all over the place, so the retry comes too late to complete the job. So a permanent fix has at least two parts - timely retry, and freezing vehicles after a sim crossing until the avatars catch up. The freeze will usually be very short, but when things go wrong, it can take a while. 

This hints strongly at excessive retry timer delay being a big part of the problem. That it eventually gets unstuck indicates that a retry worked.  SL is supposed to be a "consistent eventually" system - sims and viewers all get a little out of sync, but are supposed to get themselves back in sync.  It's normal to have short out of sync periods, but not 10-60 seconds.

Simon Linden said at the last Simulator User Group meeting that he's working on the retransmit logic for UDP datagrams, and that the  retry timers are way too long, on the order of 10 seconds. That would explain where the 10 second delay is coming from. Why one took a full minute I don't know yet. More tomorrow.

One more notch closer to understanding and fixing the problem.

[animats]

At the request of Linden Labs, I've reproduced the problem on the beta grid.

A forced half-unsit on the beta grid

This is a boring but useful tech demo to reproduce the double region crossing problem on Aditi, made at LL's request. 1000ms of network delay and a double region cross, and you're dead. Even in an empty sim on the beta grid. You can generate this failure whenever you need to. Now LL can fix it.  Related JIRA.

I left the test setup on "Animesh1" on Aditi, along with some bikes, if anyone wants to try it. It should stay there for 24 hours before it autoreturns. The bikes are stock; low-end bikes from Marketplace, not using my code.

While you can get this failure at a single region crossing, it's rare. It's common at double region crossings. It's hard to reproduce in normal SL use because, once you've made a region crossing without a problem, you can usually repeat the same crossing again without a problem. That's probably because the sim has cached info that it would otherwise have to transfer across the network. This makes debugging very time consuming. But add one second of network delay, and the problem becomes repeatable. Every time you see the network lag indicator in the SL viewer turn red, you have about that much network delay. Momentary network traffic peaks can easily create this problem. It doesn't even require packet loss, just delay.

[animats]

So I teamed up with one of SL's better bike builders, Rachel Stardust, and we now have a much better bike.

Smooth Rider

This bike is a good bike tuned for road use, and it has all the region crossing code. We routinely drive this all the way around SL continents without much trouble. 40-50MPH is no problem, and you can go faster on long straight sections of good road. Rachel went through Bay City at 140MPH without a problem. Going on sale soon.

We've done a lot of testing. 812 kilometers of it. Our bikes have been to 972 different regions. I've driven around every SL continent. Sometimes fast, sometimes slow. 25MPH for sightseeing; 50-60 on the long straight sections. Just like real life.

We've been able to fix or work around everything but Second Life BUG 214653, the bug discussed above. About once per hour of travel, depending on network conditions, that bug hits and causes a half-unsit. The bike stops, and if you log out and back in, it's there waiting for you. You can get back on and drive away. We can't fix that one, because it's sim-side, but we can contain the damage. I'm beating hard on several of the Lindens to fix this bug.

Many of the fixes here were discovered in the distant past of SL and forgotten. Locking a vehicle at a region crossing until the avatars catch up was known in 2012.  Myuki Mills and I both discovered it independently. Myuki builds some very nice bikes. with most of the fixes we have. We've talked, and now we both know how the other does it. He has a nice feature I didn't have; no sinking at region crossings. We solved the sinking problem by going to balloon mode (global hover height) briefly during a region crossing. Bad roads and bridges are now far less of a problem.

Now it's simple.  Just get on and ride. You get where you're going without serious trouble. And you can go pretty fast. We're going to have some fun with this. Stay tuned.

 

Edited April 6, 2018 by animats
Cleanup

[Fionalein]

Oh, wait, Mainland Rally Myuki? Which I was not able to finish 2017 no matter how often I tried because sooner or later the vehicles crashed on a sim crossing? *nods* yeah Myuki must have the perfect solution =^.^=

[animats]

4 hours ago, Fionalein said:

Oh, wait, Mainland Rally Myuki? Which I was not able to finish 2017 no matter how often I tried because sooner or later the vehicles crashed on a sim crossing? *nods* yeah Myuki must have the perfect solution =^.^=

He's made improvements since then. I just circled Heterocera on one of his latest bikes without significant problems.

[Fionalein]

1 minute ago, animats said:

He's made improvements since then. I just circled Heterocera on one of his latest bikes without significant problems.

That's good to hear.

[animats]

Both ours and his now do far better than that. I take each new version out for a test run around an SL continent, and usually complete the loop.

Progress marches on!

[Rachel1206]

 

This truly brings back the joy of continent riding, exploring and being on the roads  for hours. Be it relaxed cruising at 25-35 mph along coastal lines, fast drives on long country roads going 55-75 mph or speeding down town center with 100-140 mph .

Animats excellent solution and programming has lifted vehicle driving to a new level.

 

Updates & Fixes ASAP

Now we need updates and fixes implemented:

• Viewer updates - in near future will further improve the vehicle experience on SIM crossings for any kind of vehicle
• LL server updates - hopeful within months will make the crossings nearly perfect.

~

Add viewer: The texture load & caching really need to be fixed viewer side - the slow and faulty load of EXISTING CACHED textures is killing any improvements.

Add server: Attachments lost on crossings - from feet to clothes and HUDs stopping working after some crossings should not happen. We are not talking about heavy script and HUD load here, but standard Maitreya Mesh Body where feet, hands often go missing after crossings - AO HUD Vista can stop working, where only a redeliver is the solution.

The golden rule of vehicle driving/sailing/flying is of course minimal script impact and no or only required HUD loaded, but better handling of attachments by server on crossings would allow non problematic crossings wearing basic attachments needed for a modern mesh avatar year 2018.

 

Edited April 8, 2018 by Rachel1206

[animats]

We've now driven 1000km with our test bikes. Results so far:

• 1000.63 km driven.
• 68.42 driving hours.
• 5234 region crossings.
• 25 clear "half unsit" region crossing failures.
• 64 trips with "trouble" (see below).
• Fastest trip over 10km: 54.59km/h. (Rachel Stardust)

Leader board - Successful trips over 10km.

+---------------------+--------------------+----------+-------------+
| driver_name         | speed              | distance | trip_status |
+---------------------+--------------------+----------+-------------+
| Rachel1206 Resident |  54.59476223527404 |  15.6657 | OK          |
| animats Resident    |  50.04449913300664 |   11.538 | OK          |
| Rachel1206 Resident | 42.340414923148096 |  10.9615 | OK          |
| Rachel1206 Resident | 41.240210008481924 |  11.7649 | OK          |
| animats Resident    |  40.81029328697811 |  33.3964 | OK          |
+---------------------+--------------------+----------+-------------+

Distance is km, speed is km/h.

We're ready to start planning some road rallies.

(Getting caught in a ban line, running off the road and logging out, browser crashes,  getting disconnected, or simply logging out without getting off the bike first  all count as "trouble". A successful trip is when the driver had no logged problems and parked the bike at the end.)

Our free demo bike rezzer in Kama City.

Go take a test drive. Go anywhere in Zindra.  No time limit; the bike will disappear when you get off.

Edited April 10, 2018 by animats

[lucagrabacr]

Nice to see you trying to improve region crossing on the viewer-side of things still I'm a regular sim crosser so any improvement on that end is totally welcome

[Donovan Michalski]

On 4/7/2018 at 2:34 PM, Fionalein said:

Oh, wait, Mainland Rally Myuki? Which I was not able to finish 2017 no matter how often I tried because sooner or later the vehicles crashed on a sim crossing? *nods* yeah Myuki must have the perfect solution =^.^=

I completed the rally close to 30 times with a best time of 12 minutes, 21 seconds.  Even got the "rare" Myata prize.  (which is why I kept running it, besides entertainment value)

So it wasn't perfect, but it was pretty darn good.  Still my favorite car for mainland driving.

[Fionalein]

2 minutes ago, Donovan Michalski said:

I completed the rally close to 30 times with a best time of 12 minutes, 21 seconds.  Even got the "rare" Myata prize.  (which is why I kept running it, besides entertainment value)

So it wasn't perfect, but it was pretty darn good.  Still my favorite car for mainland driving.

That's interesting, I also asume you didn't try a 1000 times, right? I tried several dozen times at different times of days, picking low lag avatars...nothing helped.