Vehicles vs. sim crossings, why it's so awful

[Donovan Michalski]

1 minute ago, Fionalein said:

That's interesting, I also asume you didn't try a 1000 times, right? I tried several dozen times at different times of days, picking low lag avatars...nothing helped.

I didn't keep track of the total number of attempts, but I'd estimate at least 50.  

[Fionalein]

Just now, Donovan Michalski said:

I didn't keep track of the total number of attempts, but I'd estimate at least 50.  

So you succeeded at least each second attempt which is far better than me =^.^=

[Donovan Michalski]

Sounds about right.

I only just stumbled across this thread, so it'll be interesting to see how it all ends up.

I will say that high speed competitive driving across mainland roads as it stands right now (at least based on my experience with the rally last summer) is a mind-bending experience at its best.  Watching the car sail off into the sky for a few moments as you wait for it to return to the road...or driving at speeds faster than the road can rez and just going by the shape of the land underneath...yeah, there's gotta be a better way.   

[Fionalein]

What I don't get is: why can boats and planes do it? I have no problem with boats or planes crossing whole sims in less than 10 seconds, my 198 LI monster sailship doing an uncontrollable 27 knots ... but cars and bikes... poof and gone, and you sit in the <0,0,0> corner of the sim.

[Donovan Michalski]

I just wanted to mention that disabling the velocity interpolation was a great tip, and that alone significantly improves the experience of driving mainland roads at speed.  Highly recommend it.

[animats]

Parts of SL are broken today. I just had extreme half-unsits at the Eichorn Cove / Corchalo crossing. Twice, about half an hour apart. Much worse than usual. Driver avatar ends up in air. Passenger ends up on ground. Bike disappears, and does not show up in Lost+Found later. During the region cross, the bike motor sound climbs in pitch for several seconds.  (There's nothing in the script to do that. There's not even an LSL feature to do that). Logging information from the bike (our bikes log region crossing data to a server outside SL) shows everything normal up to when the bike disappears.  Teleporting out didn't work. Logging out and back in worked for me, but the passenger tried that and seemed to be logged in twice. Their copy of Firestorm lost its settings.

On top of this, routine stuff is failing. Stalls for five seconds when flying. Diagonally adjacent regions disappearing. Mostly in Kama City; Robin Loop in Heterocera was OK, although a bit sluggish.

https://status.secondlifegrid.net/ shows problems today, and the problems may be worse than the status page indicates.

[animats]

The Smooth Rider

All the technology I've been talking about here has now been put into a bike you can buy. It's for sale at Cute but Fast at the 2RAW EXTREME racetrack sim. It's priced low to get the technology out there. We've developed workarounds for most of the region-crossing bugs in SL. It's a good bike for long trips. You can usually ride for an hour or more without trouble. We tell people "Just get on and ride". Don't worry about region crossings. Don't bother slowing down. Go tour SL. I've circled all the major continents myself, over 500km of driving in SL.

We've now done all we can with vehicle scripts. The sim side still needs fixes. We're beating on LL to fix some specific bugs. The big one is BUG-214653,  the "half-unsit" problem. We can make it less likely through scripts, and less damaging when it happens, but sometimes, possibly due to a lost packet that doesn't get retransmitted, it fails hard. We've created a test case which can trigger that bug reliably, and at LL's request, even duplicated the test case on the beta grid. Now they have no excuse for not fixing it, and a Linden has been assigned to work on it. Whenever you get a half-unsit in any vehicle, go to that bug in the JIRA, make a comment, and upvote it. We need to keep the pressure on LL.

Meanwhile, if you do get into that situation, you have to log out and log back in again. Our bike will detect the problem, stop, and park, not run away. So it's waiting for you when you get back.

More could be done in the viewer. Our fix for bogus movement at region crossings is in the development version of Firestorm. Camera control needs some attention, too. There are moments when the coordinates relative to one region seem to being used with the location of another region, resulting in region-sized jumps and snap-backs in the viewer.

(I have some respect for LL's problems with these longstanding bugs. Linden Labs isn't big, and, 15 years on, they have heavy technical debt. It takes substantial developer time by senior developers to fix something like this properly. Sometimes you have to do it in stages - put in logging to detect when it happens and capture data, wait for it to recur, analyze the data, and perhaps add more logging to focus on the trouble spot.

That doesn't fit well with modern "agile" methodology, which favors quick changes and shipping a product which sort of works, knowing it will be replaced by a new version soon. "Agile" works on the Web, which doesn't have much persistent state or time constraints. It is a poor fit to LL's big. persistent, multi-user synchronized world.)

[Donovan Michalski]

Nice job with the bike, picked up one this morning.

Any plans for a car?   

[Rachel1206]

1 hour ago, Donovan Michalski said:

Nice job with the bike, picked up one this morning.

Any plans for a car?   

I am working on one - old classic European roadster.

[Rachel1206]

The joy of test driving... ban lines on a public road owned by Governor Linden - both sides of the road with ban lines.

And many likes this without making any sense. Now the place I passed, is public on the other side, which makes it even weirder.

And lot of ban lines on empty land and like above killing  a vehicle while no scripts allowed. And if you are really lucky, you are kicked with zero seconds warning, and you never entered the parcel/land area...

 

Edited April 26, 2018 by Rachel1206

[Rachel1206]

Test driver life is hard

Ticket:

RS Wasp v1.0,
Owned by  Rachel1206 Resident
279.325043 KpH ( 175 MPH )
Fine L$249

Amida, Sansara

 

[Donovan Michalski]

22 hours ago, Rachel1206 said:

 

Ticket:

RS Wasp v1.0,
Owned by  Rachel1206 Resident
279.325043 KpH ( 175 MPH )
Fine L$249

Amida, Sansara

 

Had to give it a try myself.  

[16:11] YD Speed Camera: 
##Donovan Michalski -Aura2-, 
Owned by  Donovan Michalski
280.931458 KpH
Fine L$250

 

 

[JoyofRLC Acker]

On 1/25/2018 at 7:00 AM, Theresa Tennyson said:

I would like to suggest an additional optional mode where instead of stopping at the region border, the vehicle is drawn continuing on a straight line at the constant speed and direction that it hit the border. One of the reasons for the "Mr. Toad's Wild Ride" region crossing condition is that the extrapolation assumes that it not only will continue moving, but will continue turning, rising, falling, accelerating, etc. If you hit a region border moving straight (which is the standard condition in a boat or airplane, especially when the pilot is familiar with how the process works) border crossings aren't usually very noticeable. It's ground vehicles that have more issues because roads often turn or rise right before a crossing. Personally it sounds like I'd find your current change just as annoying in its own way than the current conditions in many situations.

 

Coming late to this party and playing catch up through the thread ....

Boats also prone to this due this, when sailing a "straight course" there can still be acceleration effects due to waves, wind shifts / gusts etc and they can be over-extrapolated in similar fashion.   To me the idea of constant velocity (rather than zero) while waiting for the update makes sense.   Maybe with some kind of cap to cover situation where a network issue produces an extended delay (beyond bad hando ff).   Maybe (if not too complex to code) if the boat reaches say mid sim before getting an update it should come to a fairly rapid stop.

[animats]

10 hours ago, JoyofRLC Acker said:

Boats also prone to this due this, when sailing a "straight course" there can still be acceleration effects due to waves, wind shifts / gusts etc and they can be over-extrapolated in similar fashion.   To me the idea of constant velocity (rather than zero) while waiting for the update makes sense.

I was going to look at that again, but I'm waiting for the Firestorm developers to digest the latest round of changes from LL. Right now, building Firestorm from open source produces a broken Firestorm that crashes a lot. The "Official Firestorm" has closed source parts in the media, voice, physics and JPEG compression areas. There are open source alternatives for all of those in the viewers that will talk to OpenSim, but the "Official Firestorm" developers don't use them much and they don't get debugged properly. So that's on hold.

The last velocity to come from the sim before the sim crossing delay seems to be bogus some of the time. Extrapolating that vector is what causes this mess. That needs to be looked at, with debug code that shows what the sim is sending the viewer. Positions from the sim are usually good; turn off Developer->Network->Velocity interpolation to see only positions the sim sends. Velocities and accelerations, not so much. That needs a dumper, maybe metadata display, and a close look.

For slow boats, it may be possible to hide this with scripting. Capture the last 2 or 3 positions  and compute a velocity (speed only). Use that to pick an animation speed to drive the boat forward during the region crossing. Have the animation move the board forward for about 1-2 seconds, then slowly backwards for 5-10 seconds. The idea is to have the animation handle the forward motion during the sim crossing. Then when real motion takes over, the boat is a few meters ahead of where it should be. So over the next few seconds, you slowly take back that extra motion. Time region crossings and average the last few to get an estimate of region crossing time. Maybe throw in a bit of wave effect to gloss over the speed changes. Give the user the illusion of seamless motion.

[rhet0rica]

A note of caution: I've been playing around with the 'Velocity Interpolate Objects' developer setting, and it looks like disabling it isn't casualty-free—it kills animation of the prim omega attribute, which is used, among other things, to make wheels spin without spending script time or generating prim rotation updates from sim to client on a continuous basis. If there's any way you could hack around that, at least for rotation on child prims and avatar attachments, that would mitigate most (albeit not quite all; any prim can have an omega) of the side effects of killing the movement prediction.

As for the boats issue, it sounds like one of the ideas in your very first post might resolve the issue in a satisfactory manner: use a low-pass filter. The boats that were previously gliding across sim boundaries without extrapolation problems move quite slowly, so using a threshold to decide when to disable your patch (ideally adjustable through a debug setting) should accommodate them without affecting high-speed bikes. Extrapolation is intrinsically less trustworthy at larger scales, so adding a maximum velocity to extrapolate to is an entirely reasonable restriction from a numerical methods standpoint, and shouldn't be regarded as a kludge at all.

[animats]

On 5/6/2018 at 4:07 PM, rhet0rica said:

A note of caution: I've been playing around with the 'Velocity Interpolate Objects' developer setting, and it looks like disabling it isn't casualty-free—it kills animation of the prim omega attribute, which is used, among other things, to make wheels spin without spending script time or generating prim rotation updates from sim to client on a continuous basis. If there's any way you could hack around that, at least for rotation on child prims and avatar attachments, that would mitigate most (albeit not quite all; any prim can have an omega) of the side effects of killing the movement prediction.

As for the boats issue, it sounds like one of the ideas in your very first post might resolve the issue in a satisfactory manner: use a low-pass filter. The boats that were previously gliding across sim boundaries without extrapolation problems move quite slowly, so using a threshold to decide when to disable your patch (ideally adjustable through a debug setting) should accommodate them without affecting high-speed bikes. Extrapolation is intrinsically less trustworthy at larger scales, so adding a maximum velocity to extrapolate to is an entirely reasonable restriction from a numerical methods standpoint, and shouldn't be regarded as a kludge at all.

I have a basic fix going into Firestorm that just stops velocity interpolation when you're outside the sim boundaries. (If you build Firestorm from source, you'll get it, but it's not in the release version yet.)That stops most of the bogus motion, but instead you get frozen during the region crossing. So that's been put on a debug switch, so the sailboat crowd can disable it. The viewer fix doesn't kill prim omega, the way turning off Velocity Interpolate Objects does.

I also think there's a bug where the wrong region is briefly applied to an object location update, resulting in the object being yanked one region out of position for one frame. That needs logging to catch.

I want to go back to this, but Firestorm code is in a bad state right now while the developers digest the new LL fixes in texture handling. They have to resync with the LL viewer. So I'm holding off.

LL is working the sim crossing problem. It comes up in every sim user group meeting now, with Lindens reporting on fixing something associated with it. There are a lot of bugs in this area, and they have to be fixed individually. It's not one big problem; it's about five to ten medium sized ones. If LL can fix the half-unsit problem, at least you don't have to log out and back in. Everything else can be worked around with viewer fixes and scripts. So I'm trying to keep them focused on half-unsits. I want people to be able to go places in vehicles and reliably get there.  That's the point of all this.

[animats]

For the past few weeks, I've been seeing a new form of the half-unsit bug:

• Single region crossing with a vehicle at slow speed, nowhere near a region corner.
• Rider gets disconnected from vehicle and ends up in space. Sometimes moving.
• Vehicle disappears, and is never seen again. It is not returned to inventory.
• After about a minute, the avatar is back on the ground, but arrow keys don't move it properly; the left and right keys do nothing, although if shifted, they move the avatar sideways without rotation.
• If the avatar is walked across a region boundary, there's a delay of about 6 seconds at each region cross.

Our internal logging to a server shows everything going great with no errors, then the vehicle goes missing.

Anyone else seeing this?

[Rachel1206]

Yes, experienced it in Bay City with different vehicles -. same crossings, several drives; no problem - slow or fast - then suddenly at same crossing one gets unseated and vehicle disappears without a trace and  it is not returned back to lost and found.

 

[Rachel1206]

 

Quite amazing - been driving mainland Sansara this week without any problems regarding SIM crossings - for example vehicles with high script counts, high impact:

• 23 scripts, 1327 KB memory, 0.531863 ms, 41 prims/LI 26
• 33 scripts, 1838 KB memory, 0.198204 ms 75 prims/LI 46.

Great praise for Linden Lab team for improving SIM crossings in general

Edited May 18, 2018 by Rachel1206

[animats]

With the fixes LL has put in, and the fixes we've put into our vehicles and viewer, our sim crossings are now either a complete success or a total failure. All the total fails are half-unsits. We're down to one major bug. That's the one that's getting the sailors. It's up to Linden Labs now.

Edited May 18, 2018 by animats

[Fionalein]

1 hour ago, animats said:

half-unsits

Again this unexplained term. What are you even rambling about?  You started to use your animats-newspeak term on page 3 of this thead and continue using it like everyone knows what you are talking about... I bet everyone in the thread thinks of this problem as something different as you didn't define it anywhere...

[evestones]

6 hours ago, Fionalein said:

Again this unexplained term. What are you even rambling about?  You started to use your animats-newspeak term on page 3 of this thead and continue using it like everyone knows what you are talking about... I bet everyone in the thread thinks of this problem as something different as you didn't define it anywhere...

@Fionalein: animats said: "half-unsit" - the vehicle makes it across the sim crossing but, for some reason, the avatar does not. In that situation you can't unsit.

@animats: Thanks for your work on all this. Your persistence and patience is much appreciated.

 

Edited May 18, 2018 by evestones
@ thanks

[animats]

1 hour ago, evestones said:

@Fionalein: animats said: "half-unsit" - the vehicle makes it across the sim crossing but, for some reason, the avatar does not. In that situation you can't unsit.

Right, and thanks.

All this detail is needed because SL's region crossing problems have dragged on, unfixed, for over ten years. To get them fixed, we as users have to be very clear about what we're reporting to LL. There's a tendency to ignore bug reports which are not clear enough, or are not reproducible, or combine unrelated problems into a single bug report. This is a generic problem with support organizations. Overcoming that problem requires extensive, even excessive, documentation of problems, until denial is no longer possible.

If you want the really technical version, see BUG-214653 and its links.

[animats]

I mentioned in another topic that we now have an "unsinkable bike" that doesn't sink at region crossings.  Just before hitting a region crossing, it starts hovering, maintaining the altitude before the crossing. Once the region crossing is complete, it goes back to normal. This gets it across on bad roads where there's no support prim reaching across from the other side of the crossing. As long as there's some support on both sides, it can handle it. As a side effect, it seems to reduce unexpected bogus movement at crossings. Any sinkage means a bump coming out of the hole, which may kick a vehicle sideways.

With this in place, sinkage is no longer common. But it happens once in a while. Why? Really, really bad spots in roads.

Nice looking road that doesn't work.

This is a mesh road in Larsen, in the snowy part of Sansara. I had a motorcycle fall in here, so I checked it out. The road here is a long mesh piece, and its root is in another sim 9 meters away. So it provides no support at all for a long distance. The terrain underneath is almost 2m down in spots. Hit the wrong spot and you're underground. An avatar standing here will fall in, every time. I showed this to another biker, who said of the road builder "you had one job". Filed a support ticket on this one.

Then, there's Kama City.

The mean streets of Kama City.

This is the end of a street in Kama City on Zindra. There's a region crossing in the center of the street. That's fine. But the end piece of the street is one mesh piece, which provides no support at all for half the street width. This spot is on a hill near the beach, so there's empty space under the road. Stand there and you will fall in, every time. Somebody thoughtfully put a barricade at the exit from the monorail station to prevent people from falling in. It's only a problem at street ends; our latest bike glides smoothly across Kama City's mis-designed intersections. You can get an avatar stuck walking across those.

We noticed this road problem testing with all the latest fixes. I'm driving around with Firestorm 5.1.5 and our latest bike. Most region crossings now go very smoothly. That uncovers previously hidden problems, like these.

These are not top priority problems. LL needs to remain focused on the "half-unsit" problem, which is much more serious - you lose all immersion and have to log out, killing whatever activity you were involved in.

 

[animats]

On 5/19/2018 at 10:12 PM, animats said:

 Filed a support ticket on this one.

LL fixed it. Thanks.

I try to file tickets only on road problems serious enough to crash vehicles. You're going along great, the road ahead looks fine, and suddenly you're off the road or under it.