Merrily we bump along

[animats]

Or, why there's bogus movement at region crossings.

First, this has nothing to do with region crossing failures. This is all about how it looks in the viewer on successful region crosses.

To recap: whenever a vehicle crosses a region boundary, there's a delay of about a second, sometimes longer. During this delay data about the vehicle, the avatars involved, the attachments and scripts involved, and some other data is packaged up by the sim server of the region being exited, transmitted over the network to the server handling the new region, unpackaged, loaded into the new sim state, and started up. During this period, the vehicle, avatars, etc. do not move at all.

The viewer tries to hide this pause with some fakery, trying to guess where the vehicle/avatar is going. This works well for walking, OK for slow boats, and badly for fast ground vehicles. Sometimes the guess is so bad that vehicles roll over, go underground, or fly into the air, only to snap back to their proper position as the new sim takes over control.

This effect is worse than it should be. Here is why.

As a test, I built a version of Firestorm which logs every object update message (sim to viewer) to whatever the avatar is sitting on. Then I made a few trial runs and graphed the results.

I'm on a boat. This is the velocity in Z (up and down) the sim sent the viewer, in meters per second. It's almost zero all the time. The big moves at the beginning are from getting the boat into the water. At a region crossing, the last velocity value before the region crossing is used to project, in a straight line, where the boat will go. No problem with that here. Smooth sailing.

 

I'm in a car. This is a trip around Robin Loop in Heterocera, starting and ending in Burns. Mostly good road.

On this little trip, using Firestorm with standard settings, the vehicle went underground several times at region crossings, and into the air once. Once you can see the Z velocity data, it's clear why. SL ground vehicles bump along as the physics engine slides them along the terrain. (Rolling is an illusion; they really slide.) Just as in real life, there's some bouncing up and down as this happens, and it's worse at joints between pavement sections. Nothing wrong with that. The physics engine in the sim process operates 45 times per second, and those little bumps get corrected very quickly.

Until a region crossing comes along. Then, the last velocity values (and angular rate values, not shown here) are sent to the the viewer, followed by silence while the region crossing takes place. The viewer tries to extrapolate those, for a second or so. Or, 45x as long as those values were supposed to be valid.

That extrapolation amplifies every tiny bump right before a region crossing into a huge move. Only the very last velocity value before the region crossing gets extrapolated. Any bumpiness in that last value turns into a huge bogus move, amplified by about 50x. As you can see from the chart, there are some big bumps in there. Some of this is imperfect road. Most of it is just the random result of the physics engine keeping the vehicle in contact with the road, which it does by applying a correction every 1/45 second.

That is why there is large bogus motion at region crossings.

Previous thinking was that there was some kind of bug involving the last object update before a region crossing. That's not it. It's a straightforward result of how the system works. Once you can see the data above, with  those spikes, it's clear what's happening. If one of those spikes happens to be just before a region crossing, you're going flying.

So, what to do about this? My previous try at this led to "stop", or "don't extrapolate" mode in Firestorm. Drivers liked it because they didn't lose control trying to steer out of bogus motion. Sailors hated it because that sudden stop broke immersion. (Also, wakes and similar visual effects don't play well with sudden stops.) The "stop" fix stopped working in Firestorm 6.3.x, because LL put in a fix of their own, which still extrapolates, but not as much. That improved things for LL viewer users, and spread the misery around between drivers and sailors for Firestorm users.

For the next round, I'm looking at some kind of filtering to smooth out the noise, plus limiting the amount of extrapolation based on some measure of how noisy the data is. I'm trying some statistical tests on the logged data. More on this later. There's no perfect solution to this, but it should be possible to clamp bogus movement to something that doesn't make SL look broken.

Edited March 22, 2020 by animats

[Odaks]

Thanks @animats. That was very nicely presented and has definitely caught my interest. It'll be good to see where you can go with this.

[animats]

Related Firestorm JIRA proposal.

More technical than the above discussion.

[animats]

Smooth crossings.

Works for both a fast motorcycle and a slow sailboat, with no user adjustments needed. Needs to be tested in more situations. So far, so good.

I've  built a modified version of Firestorm for my own use with a new movement prediction limit system. Boats behave like they do in the LL viewer. The boat moves forward smoothly for one second (that's LL's current limit, not mine), then stops, then rubber-bands back a few meters. So that doesn't change.

Ground vehicles on bumpy ground, though, get hard-stopped before bogus motion gets out of hand. No flying into the air or diving into the ground has been seen yet.

Movement after a region crossing is stopped after an error predictor calculates that the error will exceed set limits. Currently, I have those set to a position error of 0.25m and an rotation error of 20°. During a region crossing, the viewer moves the vehicle and avatars forward in the direction they were going over the last 5-10 seconds. If the previous movement was very smooth, as when sailing, prediction is allowed for up to 1 second. On roads, it's about 1/10 second. After the region crossing completes, there's the usual snap-back

Camera control hasn't changed, so you still get the camera continuing to move while the vehicle is stopped. Anyone want to look at that?

Code here, for viewer developers. Licensed Firestorm/LGPL. I will be talking to the Firestorm developers about putting this in.

There's no perfect solution to this, but this one is better than what we have now.

Edited April 3, 2020 by animats
Typo.

[animats]

So far, so good. I've tried an assortment of vehicles; boats, cars, motorcycles, helicopters.

The sakura are so close! On the part of Robin Loop from Electra to Neumogen on Heterocera, there are many Japanese parcel owners. They've all put Japanese cherry blossom trees along the road. It's a beautiful drive. So I drove it in a classic car, using my viewer fix in Firestorm.

Driving with this fix makes high-speed region crossings feel like a moment in a muddy spot. You slow down and maybe stop for a moment, then push through, going straight the whole time. No need to take your finger off the  up arrow key. Low speed region crossings are very smooth.

Some sailors were unhappy with the previous "stop" fix. If that was a problem for you, please IM me. I'd like to go for a ride on your boat while recording video. Thanks.

Edited April 3, 2020 by animats

[Jackson Redstar]

the AKK horses i remember had a fuction if you was running it would slow to a to walk at region crosses then resume vrunning

[animats]

Did a Drivers of SL drive today. Boat, helicopter, motorcycle, heavy truck.

Larson truck. Old, rough ride, rocks from side to side. Good test.

This truck is always rolling a little, and, as explained above, that gets amplified at a region crossing. The new limiting works. At every region crossing, the roll stops at about 20 degrees.  That's the idea; there's some bogus motion, but it's limited.

Slow boat. Sailing with an SL yacht club. Absolutely no going underwater, going into the air, rolling over, or other totally bogus motion. We're out on a complex boat that takes about five seconds to cross regions. The first second is extrapolated, then there's a stop for the rest of the region crossing.

The viewer can hide a quarter-second region crossing, as when an avatar walks across a region bound in a non-busy sim. It can almost hide a half-second. It can make a full second look better. Beyond that, you have to accept a stop. There's only so much the viewer can cover up. Maybe the region crossing delays will get shorter after "cloud uplift". AWS offers very fast inter-computer links within their data centers. We've done what we can viewer-side.

 

 

 

[animats]

Patch to Firestorm submitted, ref https://jira.firestormviewer.org/browse/FIRE-29021

[animats]

Patch accepted and in the queue for the next Firestorm beta.