Vehicles vs. sim crossings, why it's so awful

[animats]

Just a minor update. I've found a few more of those bad road holes. Not many. One spot on the outskirts of Zindra, and two in the snowlands of Sansara.

This isn't a bridge. Just a bad road.

Newer mesh roads on irregular terrain tend to have this problem. No support prims, and long mesh pieces which cross regions.

Half-unsits continue, at the rate of 1-2 per hour of driving. Waiting for LL on that; they're actively working on the bug. That's the last killer problem.

Other than that, we're doing OK with our bikes. With all these fixes, our bikes are much easier for new riders. I just had a customer who had an old "light cycle" bike from Tron. He couldn't keep that on a narrow straight road for 100 meters. But he could ride our bike all around town without problems. One woman who had become frustrated with bikes tried one of ours around Robin Loop in Heterocera, which is a moderate-difficulty ride of a few kilometers. She completed Robin Loop on the first try, at a reasonably good speed, without going off the road. That's what should happen. I tell people "Just drive." It works.

It's fun to give people rides or loan them bikes. There are many SL users who don't have a sense of how big a world SL is. You can ride for an hour and keep seeing new things.

[animats]

Sim crossing problems act as a pervasive drag on large scale activities in SL. Here are a few of them.

Today, Drivers of SL had a three hour drive/quest. Dozens of drivers crossed hundreds of sim boundaries. I had two half-unsits in 3 hours. That's about typical. Everything we can do script side and viewer side can't get us below about one fail per hour. We can do a lot to make crossings smoother, but that sim-side bug where avatar and vehicle part company remains the big problem. Only LL can fix that, and they don't know how yet.

Today, the first responder groups in SL had a trade show. Fire engines, ambulances, rescue helicopters, and associated equipment were on show. Most are very realistic, with many functional features. One of the helicopter designers showing there gave me a ride a few weeks ago. She's a helicopter pilot in real life, and her helicopters are as realistic as she can make them. They take training and real piloting skills to fly. Yet they can be crashed by a sim crossing bug. People make realistic air ambulance runs in those things. Sim crossing failures discourage dedicated users from doing things like that. This is a killer for large scale roleplay.

I mentioned in another topic that many new users expect to be given a quest, or something to do, when they enter Second Life. One of the shorter Drivers of SL scenarios would be great for that. They have an excellent HUD system which guides users from waypoint to waypoint, gives them tasks to do, gives them items, checks their progress, and keeps the quest going. Newer HUD versions use experience tokens and are quite effective.

The quest falls apart on a failed region crossing. The user has to log out and log back in. They probably won't be near a rez point, so they can't create a new vehicle. Immersion is broken. That makes this unusable as something to help new users get started - we'd lose about half of them with a half-hour quest.

It's hard to think of any other online technology with a mean time before failure of less than an hour. Fifteen years ago, yes. Maybe ten years ago. Not today. The quality of the experience just is not good enough to retain new users. LL needs to get their act together, and fast.

 

[Parhelion Palou]

1 hour ago, animats said:

It's hard to think of any other online technology with a mean time before failure of less than an hour. Fifteen years ago, yes. Maybe ten years ago. Not today. The quality of the experience just is not good enough to retain new users. LL needs to get their act together, and fast.

Funny, SL hasn't failed on me in weeks. If SL had been designed to be a vehicle-based game, you could say it's failing. You're trying to turn what's basically a general-purpose sandbox into a vehicle-based game. Of course it doesn't work smoothly.

About 9 years ago I started noticing that people were teleporting everywhere. They'd teleport from a store to another store that was only 50 or so meters away. They wouldn't even know that the two stores were so close. It's still that way for most of SL's population. They don't walk anywhere, much less use vehicles. The behavior of vehicles at sim crossings, good or bad, will have nearly zero effect on new user retention.

[animats]

It doesn't work because of very specific bugs. Vehicles are a part of SL and are supposed to work. They're just broken. SL is probably a few hundred lines of code changes away from region crossings that work reliably. It's a tough few hundred lines. There's a message protocol bug. Two Lindens are working on this.

The "teleporting everywhere" thing is real, but many of the roleplay sims don't allow teleports, for realism. Many new users want a more game-like experience. SL isn't a game in itself, but many people build games within it. Even LL does that, although not very well. Having more game-like experiences on offer for new users is important. Some people want that structure. It beats dumping them out in a new user area full of loud griefers.

[TugkanAdley]

On 4/30/2018 at 2:13 AM, Donovan Michalski said:

Had to give it a try myself.  

[16:11] YD Speed Camera: 
##Donovan Michalski -Aura2-, 
Owned by  Donovan Michalski
280.931458 KpH
Fine L$250

 

 

Hello, I have a question. What is speed camera fine? There are obligation for paying fine? Or just cosmetic item? 

Thank you.

[animats]

18 minutes ago, TugkanAdley said:

Hello, I have a question. What is speed camera fine? There are obligation for paying fine? Or just cosmetic item?

It does not really fine you. It is just a cosmetic item.

[animats]

Here's a kind of failure new to me. I was riding around in a Yava pod, and became unseated at a region crossing. Since then, walking across a region boundary has a consistent 10 second delay. Every time. And I can't fly. I can only jump. Without the usual loud clicking noise of a no-fly zone. My avatar is stuck in some stable broken state sim side. Some problem in Rhodenwald seems to have caused this.

I've seen such delays in the past, but it's taken very fast motorcycle riding to hit this bug. It was usually coupled with loss of controls; the controls were still tied to the vehicle. This totally repeatable 10 second delay is new.

[Sharie Criss]

While some people disagree, the anecdotal evidence makes it painfully clear that LL is "saving money" on SL so they can spend it on Sansar by cutting their operating expenses - such as technical support, and data center costs. I very very strongly suspect based on the significant reduction in server performance that they have increased the number of regions they place on a physical server. Based on LL's behavior, I REALLY doubt that any of these performance problems will ever be addressed. Some people point to continued development as "proof" LL is not doing what I claim, but if I were LL and wanted to keep my cash cow alive, I'd be attempting to retain gullible customers with the same type of lures. The reality though is that this development is painfully slow, and very very few existing bugs (some many years old) ever get fixed. Animesh? Bakes-on-mesh? Where are they? Will they go the way of Experience Keys where the concept of Grid-wide keys was dropped? You know - the version that would make Experience Keys actually - useful - outside of a very limited environment and use case.

[Theresa Tennyson]

1 hour ago, Sharie Criss said:

While some people disagree, the anecdotal evidence makes it painfully clear that LL is "saving money" on SL so they can spend it on Sansar by cutting their operating expenses - such as technical support, and data center costs. I very very strongly suspect based on the significant reduction in server performance that they have increased the number of regions they place on a physical server. Based on LL's behavior, I REALLY doubt that any of these performance problems will ever be addressed. Some people point to continued development as "proof" LL is not doing what I claim, but if I were LL and wanted to keep my cash cow alive, I'd be attempting to retain gullible customers with the same type of lures. The reality though is that this development is painfully slow, and very very few existing bugs (some many years old) ever get fixed. Animesh? Bakes-on-mesh? Where are they? Will they go the way of Experience Keys where the concept of Grid-wide keys was dropped? You know - the version that would make Experience Keys actually - useful - outside of a very limited environment and use case.

How are your problems now different from your problems in 2014?

https://jira.secondlife.com/browse/BUG-5475?jql=text ~ "sharie criss"

[Sharie Criss]

On 9/5/2018 at 7:03 AM, Theresa Tennyson said:

How are your problems now different from your problems in 2014?

https://jira.secondlife.com/browse/BUG-5475?jql=text ~ "sharie criss"

The difference now is that sim performance is horrible all the time rather that only when people TP in and out. The TPing issue simply makes things worse, now instead of the sim dipping to a time dilation of 0.5 with script run at 30%, the script run times now are NEVER better than 50% and dip to 1%. That's what's different. Before you could at least do things most of the time, now - it's just horrible and laggy all the time. And it's not just "my" region, it's every region. Perhaps you were unaware of this.

 

[Parhelion Palou]

5 hours ago, Sharie Criss said:

The difference now is that sim performance is horrible all the time rather that only when people TP in and out. The TPing issue simply makes things worse, now instead of the sim dipping to a time dilation of 0.5 with script run at 30%, the script run times now are NEVER better than 50% and dip to 1%. That's what's different. Before you could at least do things most of the time, now - it's just horrible and laggy all the time. And it's not just "my" region, it's every region. Perhaps you were unaware of this. 

 

Very interesting ...

Scripts run are at 93%, time dilation 0.996. So much for every region.

If every region you go to has bad performance, what's the common factor in each case? Perhaps your avatar is destroying region performance.

[animats]

On 8/29/2018 at 2:05 PM, animats said:

Here's a kind of failure new to me. I was riding around in a Yava pod, and became unseated at a region crossing. Since then, walking across a region boundary has a consistent 10 second delay. Every time. And I can't fly. I can only jump. Without the usual loud clicking noise of a no-fly zone. My avatar is stuck in some stable broken state sim side. Some problem in Rhodenwald seems to have caused this.

I've seen such delays in the past, but it's taken very fast motorcycle riding to hit this bug. It was usually coupled with loss of controls; the controls were still tied to the vehicle. This totally repeatable 10 second delay is new.

Got that one again. It's associated with a really strange effect - the motorcycle sound goes up and up in pitch and then stops. The script isn't doing that. LSL scripts can't do that; there's no "adjust pitch" feature. Puzzled. Looking at viewer code.

It's striking that there's nothing in Firestorm's log that looks seriously abnormal during or after a region crossing failure. The avatar has jumped into midair and is stuck, but the viewer log shows nothing wrong. More debug checks would help diagnose the problem.

 

Edited September 12, 2018 by animats

[animats]

(Technical update, for those interested.)

I've been looking at how the viewer processes object updates, in llviewerobject.cpp. I've been adding debug code to a self-compiled version of Firestorm and forcing region crossing failures to get detailed info about how they fail.

The "half-unsit" bug appears in world with the avatar usually stuck in the air a few hundred meters from the vehicle. That bad position is not coming from the update message from the sim. I've been dumping those messages, and they show legit avatar positions when the viewer is showing bogus ones. The viewer is adjusting the avatar position incorrectly. This is a minor bit of progress.

The avatar, vehicle, and attachments all make the trip from sim to sim separately and asynchronously.  They can arrive in any order. Parenting relationships are temporarily disconnected during the transfer. There is code in llviewerobject.cpp to put the viewer's model of those parent relationships back together as all the parts arrive. It's very messy code. Positions get adjusted based on relationships between parent and child, and there is something going wrong in there which generates bogus avatar positions, ones that may be hundreds of meters away from the right place. I suspect that coordinates relative to one sim are being applied to the wrong sim, but that's still speculation.

Incidentally, there's a timer for attachment arrival. If an attachment arrives more than 30 seconds late, it will not be reconnected. That's part of how attachments get lost.

All this may not be the cause of "half-unsits". It may just be why they look so bad in world when they happen.  The real problem is that you get stuck in that state - avatar in the old sim, vehicle in the new sim - and there' s no way forward.

Has anyone been inside of llviewerobject.cpp trying to figure out that section?

 

[Klytyna]

15 hours ago, animats said:

The avatar, vehicle, and attachments all make the trip from sim to sim separately and asynchronously.  They can arrive in any order. Parenting relationships are temporarily disconnected during the transfer. There is code in llviewerobject.cpp to put the viewer's model of those parent relationships back together as all the parts arrive. It's very messy code.

So... 

After NINE months of mucking about in viewer-fork code, you have made "an AMAZING discovery", namely that...

On 30 November 2017 at 2:46 PM, Klytyna said:

1. The sim you are on bundles up the current state of your inventory, every parameter of every face of every 'prim' of every object you wear or have in inventory, the current state of every script, every memory variable, the whole damn thing.

2. You inventory is fired at the asset servers

3. you are fired at the target sim, naked, bald, and ruthed

4. The target sim decides if it wants to accept delivery of you

5. The target sim sucks down your total inventory state from the asset servers and tries to figure out which bits of it are supposed to be attached to you

6. The target sim starts up any scripts in the stuff thats been attached, cue massive lagspike as your script cpu time goes up about 900% for several seconds

7. Congratulations you have arrived

Well done...

Only NINE months to discover what you were told in the 1st reply post in this thread...  
 

[animats]

6 hours ago, Klytyna said:

So... 

After NINE months of mucking about in viewer-fork code, you have made "an AMAZING discovery", namely that...

Well done...

Only NINE months to discover what you were told in the 1st reply post in this thread...  
 

This is about the viewer/sim relationship, not the sim/sim relationship. Region crossings are three-player operations, and the viewer can mess up the process. The process would work better if it was more like what Klytara describes, with the viewer just being an observer. But it's not. (I hear it's more like that in OpenSim, which reportedly doesn't have this bug. Is that correct?)

[animats]

This is preliminary, but I may be getting close to the underlying cause of the "half-unsit" problem. See my last comment in the JIRA.

 

[animats]

11 hours ago, animats said:

This is preliminary, but I may be getting close to the underlying cause of the "half-unsit" problem. See my last comment in the JIRA.

Informally, it looks like the "half-unsit" problem occurs when certain object update messages come in out of order. They come in over UDP, so that happens occasionally. If this happens just as a region crossing of a vehicle is starting, code in the viewer incorrectly sets the parent of the avatar to null. This detaches the avatar from the vehicle without doing a proper "stand". That's the half-unsit.

So the viewer thinks the sim told it to unseat the avatar and await further orders. Meanwhile, the sim doesn't know that happened, and thinks the avatar is still sitting on the vehicle and moving with it  The avatar is now stuck, going nowhere. Things go downhill from there. This inconsistent situation persists until either vehicle or avatar does something major like die or teleport or log out, which forces both sides back into sync.

After adding much debug output to a self-compiled copy of Firestorm, I can see where things first go bad. In the standard logs, everything looks just fine at the point things are going bad. No errors, no warnings. But when you log parenting changes in the viewer, this jumps out at you. Normally, there are no viewer side parenting changes at a region crossing. If one is logged, something has gone badly wrong.

Until now, I'd been looking mostly at the aftermath of a half-unsit, when things were already messed up. That was mostly a dead end. Now we can see where things start to go wrong.

No fix yet, but some of us are looking suspiciously at parts of LLViewerObject::processUpdateMessage. 

Edited September 23, 2018 by animats

[animats]

"Simulator malfunction: sim killed vehicle. SL BUG-214653."

At last, we know what triggers a half-unsit.

The cause of the half-unsit problem, at least at region corners, is sim-side. The sim is sending a "kill object" message to the vehicle before telling the viewer the sim crossing has started. A sim crossing begins with a "crossed region" message and later has an agent_movement_complete message. There are normally some kill messages at the end of a sim crossing, to clean up things in the losing sim, but this one, which is sent all alone, is way too soon. The avatar hasn't even started to move to the new sim yet. The kill message tells the viewer to disconnect the avatar "prim" from the vehicle parent. This leaves the avatar in limbo - still sitting, but on nothing.

I now have a test version of Firestorm which detects this bug. When it receives a kill message from the sim aimed at something the avatar is sitting on, and the viewer is not yet in region crossing mode, it puts the message above in local chat. This makes the problem unambiguous. 

This is very similar to the way attachments get lost in sim crossings. Attachments are lost because they get bogus kill messages from the sim. Firestorm has a workaround to help with that. See FIRE-12004. That workaround also puts a message in local chat, if the appropriate debug switch is turned on.

I tried ignoring the kill message for the vehicle, like the FIRE-12004 workaround. The avatar then stays on the vehicle through the sim crossing, but loses controls and can't steer. About 10 seconds later, there's another half-unsit and the avatar ends up in midair. So the simple workaround isn't enough. But maybe with more work, a temporary viewer side workaround can be found.

Beq Janus has been capturing these failures with WireShark, to look directly at the network messages to see this.

All this should provide Linden Labs with enough information to fix the problem.

 

 

[Love Zhaoying]

What new mystery will you pursue next, now that this is found?

[Qie Niangao]

4 hours ago, animats said:

I tried ignoring the kill message for the vehicle, like the FIRE-12004 workaround. The avatar then stays on the vehicle through the sim crossing, but loses controls and can't steer. About 10 seconds later, there's another half-unsit and the avatar ends up in midair. So the simple workaround isn't enough. But maybe with more work, a temporary viewer side workaround can be found.

If the avatar loses control connection to the vehicle, it would seem something else must be going wrong besides the kill message. Not sure how much deeper one can delve looking only at the viewer messaging, especially if the viewer's problems are just side-effects of some inter-sim race condition. I guess one could confirm that the viewer is trying to send controls to the correct simulator (assuming that sim is ready to get them), but otherwise it's going to need watching the sims' states along with the messaging through the handoff.

[animats]

5 hours ago, Qie Niangao said:

If the avatar loses control connection to the vehicle, it would seem something else must be going wrong besides the kill message. Not sure how much deeper one can delve looking only at the viewer messaging, especially if the viewer's problems are just side-effects of some inter-sim race condition. I guess one could confirm that the viewer is trying to send controls to the correct simulator (assuming that sim is ready to get them), but otherwise it's going to need watching the sims' states along with the messaging through the handoff.

We can now detect this error, and point to a specific message from the sim which is clearly wrong. That gives Linden Labs' developers enough information to look at the sim code and find out why that totally bogus object kill message was sent. We've been able to reproduce the problem in an empty sim on the beta grid for months. Now we can show the exact message where the sim did something wrong. We've even instrumented the Firestorm viewer to put a message in local chat when this happens.

All excuses for Linden Labs not fixing the problem have thus been eliminated.

OpenSim doesn't have this bug, one of the Lindens told me. This  is not an impossible problem.

Anything we can do about this viewer-side is a temporary workaround.

[Love Zhaoying]

Why are you so passionate about fixing this? You don’t have to answer. But, I’ve seen you working on this for a long time.

[animats]

Just now, Love Zhaoying said:

Why are you so passionate about fixing this? You don’t have to answer. But, I’ve seen you working on this for a long time.

 

I like to finish what I start. I've had a long career of solving hard problems. Having started on this one, I want to see it through.

[Fionalein]

4 hours ago, animats said:

 

I like to finish what I start. I've had a long career of solving hard problems. Having started on this one, I want to see it through.

... and it never occured to you finishing other folks problems might be considered rude or impolite?

[Nalates Urriah]

On 9/5/2018 at 2:39 AM, Sharie Criss said:

... the anecdotal evidence makes it painfully clear ...

There is anecdotal evidence the Earth is flat. Subjective experience is often misinterpreted by people. Effects in proximity do not prove causation or even relationship.  

Try to get back in touch with reality.