What's up with sim crossings?

[Sister Nova]

5 hours ago, MBeatrix said:

Ghosted avatars are fairly common after a crash. That's something I've always took as "normal", as it's something I've always seen around since I started using vehicles.

What's not normal is the crashes rate crossing sims on a vehicle these days. But that is something that's always happened in cycles, it seems — it comes and goes away without any apparent fix.

You can quite often see ghosted avatars just sitting by a sim crossing. Right clicking them will usually exorcise them

[Alyona Su]

9 hours ago, animats said:

Sim crossings with more than one avatar just got really weird. I took a friend for a ride in a boat to look around the new Linden Homes continent. After about five sim crossings, she messaged me for help. She fell out of the boat. But on my screen, she was still in the boat.

I was experiencing this long before the TP and border-crossing issues started. I am talking months ago, like back to September '18 and such. Just saying.

[Alyona Su]

47 minutes ago, Walpurgisnacht said:

You can quite often see ghosted avatars just sitting by a sim crossing. Right clicking them will usually exorcise them

Yes, this. And no Excorsist chant is necessary even. But their heads don't spin around in circles, though.

[JoyofRLC Acker]

Can people please confirm that the viewer is not a factor.  Almost everything Ive read says this is strictly server side, and the LL notice on this says all viewers are affected.

However a sailing friend seems pretty convinced that the SL viewer is more robust in this regard (sailing / sim crossing specifically) than Firestorm.   Other than possibly different thresholds for time-out disconnects, I cant see the viewer making a difference if it IS strictly server side.

I'm probably being naive but I find it a bit worrying that it is taking the 'crack team' so long to diagnose the problem.  (Yes, the fix can be lengthy depending on the problem, but Im surprised that diagnosis is proving so difficult).  It suggests to me that there may not be a single cause (other than the basic architectural design and the inherent byzantine nature of region crossings).

It does also raise the question of how they test system releases.

 

[Solar Legion]

5 minutes ago, JoyofRLC Acker said:

Can people please confirm that the viewer is not a factor.  Almost everything Ive read says this is strictly server side, and the LL notice on this says all viewers are affected.

However a sailing friend seems pretty convinced that the SL viewer is more robust in this regard (sailing / sim crossing specifically) than Firestorm.   Other than possibly different thresholds for time-out disconnects, I cant see the viewer making a difference if it IS strictly server side.

I'm probably being naive but I find it a bit worrying that it is taking the 'crack team' so long to diagnose the problem.  (Yes, the fix can be lengthy depending on the problem, but Im surprised that diagnosis is proving so difficult).  It suggests to me that there may not be a single cause (other than the basic architectural design and the inherent byzantine nature of region crossings).

It does also raise the question of how they test system releases.

 

As you've already stated: Linden Lab has stated that all client programs are affected. Your friend is showing bias, based on their own experience and conjecture.

Put another way: In the past they have not made similar statements easily or lightly.

As for system release testing - what do you think the Release Candidate (RC) regions are for?

Edited April 23, 2019 by Solar Legion

[Qie Niangao]

27 minutes ago, JoyofRLC Acker said:

I'm probably being naive but I find it a bit worrying that it is taking the 'crack team' so long to diagnose the problem.  (Yes, the fix can be lengthy depending on the problem, but Im surprised that diagnosis is proving so difficult).  It suggests to me that there may not be a single cause (other than the basic architectural design and the inherent byzantine nature of region crossings).

If I were a developer, I wouldn't even look at reports of sim crossing problems while they're able to collect data on teleport disconnects. That's a solid all-or-nothing thing that we know was very rare before. Sim crossings suck worse than before, maybe, but they've always sucked and discerning any signal in that mess of noise has always been nearly impossible. Maybe, maybe the sim-crossing problems are affected by something distinct from the teleport disconnects, but:

1. Occam says no, and
2. Even if so, it'll likely be easier to find it without whatever's causing the teleport disconnections contaminating the data.

There could be multiple factors contributing to the teleport disconnections, too, but it almost doesn't matter: race conditions are notoriously difficult to pinpoint, even if it's always the same out-of-sequence event in every failure, and always the same conditions triggering it (which is rarely the case, and almost surely not the case here).

Still, yeah, it has been a long time now, and the developers must be going crazy with stress about it -- if they're not numbed to the pain by now.

Oh, also a word about viewers: there's practically zero chance the Linden viewer fares any better than the others here. For one thing, for me it has happened more with the Linden viewer than Firestorm or Catznip, but that's possibly due to time-of-day when I use the different viewers -- or just randomness misinterpreted as pattern, the source of all superstitions.

Edited April 23, 2019 by Qie Niangao

[Aishagain]

9 hours ago, MBeatrix said:

Ghosted avatars are fairly common after a crash. That's something I've always took as "normal", as it's something I've always seen around since I started using vehicles.

What's not normal is the crashes rate crossing sims on a vehicle these days. But that is something that's always happened in cycles, it seems — it comes and goes away without any apparent fix.

This surprises me.  I thought the sim-border ghost had been eliminated a few years back during the tuning of server-side appearance.  I once had the slightly surreal experience of disconnecting at a sim, relogging and being told by my partner that my avatar was still where I had disconnected.  She even took a photo of it!

What REALLY spooked me was on returning to that sim I was confronted with my own avatar still in situ.  As many others have since found, left clicking on that avatar caused it to vanish.  On reporting this (including photos) to LL Support, I was told such a phenomenon was "impossible".

I too have recently seen an increased number of ghosts at region edges.

On the subject of ignores, ignoring Solar is a mistake, since, despite himself, he actually DOES sometimes talk sense!

Perhaps I should clarify that the above incident occurred to my alter ego, not this account and yes, it was two instances of the same account, just in case you weren't sure I was sane!

 

Edited April 23, 2019 by Aishagain

[JoyofRLC Acker]

Surely they have data on sim crossing failures as well as the TP failures - its the same underlying issue I thought.  They dont have to rely on "reports" (but see below) for sim crossings they would have the same server log data.   I do not see why you say (well, imply) that a disconnect on a sim crossing is not a solid all or nothing event.  It is no less catastrophic than a disconnect on a TP.   To be clear, I am talking specifically about the horrendous number of DISCONNECTS on sim crossings that started about a month ago - not the other sim crossing problems that, as you say, have been around in varying degrees for a long time (but are also much increased currently) - like detachment, cam going kaflooey (sorry for the technical term) etc etc - and which may or may not be related.

Regarding your reference to "reports" - I infer that you are implying they are less useful that the hard data.  I think both may be helpful, the reports may contain hints as to that else is going on, and may yield hints on where to look. 

An example of the contextual stuff is the apparent fact that TP triggered by scripts are less likely to fail than ones initiated by user action.   (Some one else put me on to this which explains something that had puzzled me ... for myself Ive been having very few TP problems at all the last month, but 90% of TPS are initiated by my Nav HUD) whereas I have been getting very frequent disconnects when sailing; and even just standing on my dock chatting or editing scripts.

Edited April 23, 2019 by JoyofRLC Acker

[Fluf Fredriksson]

1 hour ago, Solar Legion said:

As you've already stated: Linden Lab has stated that all client programs are affected. Your friend is showing bias, based on their own experience and conjecture.

Put another way: In the past they have not made similar statements easily or lightly.

As for system release testing - what do you think the Release Candidate (RC) regions are for?

After many many *many* hours of testing sim crossings, I can tell you that at the moment Henri's CoolVL is much less prone to the disconnection errors on sim-crossings (Teleports not tested to a degree where I could confirm or deny either way). The now horribly out of date "Alchemy" viewer also seems to do surprisingly well on sim-crossings, but I've not put enough hours in to testing Alchemy to confirm one way or another yet. I was hoping the problem would be fixed before I'd be able to.

It does seem to me that this is a two sided problem. Viewers that I've tried based on LL's latest viewer code (Firestorm / Kokua), seem to give up earlier on a sim crossing error or delay than CoolVL and possibly Alchemy. However the source of the errors are server side, and even CoolVL / Alchemy are still sometimes tripped up. Yes it is a very random error. Yes you need a lot of hours testing over many weeks to be able to confidently say what I just said. Yes I have done that.

This is where Solar tells me that I must be wrong.

I'd like to be wrong. I'd far rather be using Firestorm at the moment.

EDIT: Going back over the stats, I don't have enough data on Kokua to group it in with Firestorm on failure rates.

Edited April 23, 2019 by Fluf Fredriksson

[Alyona Su]

57 minutes ago, JoyofRLC Acker said:

However a sailing friend seems pretty convinced that the SL viewer is more robust in this regard (sailing / sim crossing specifically) than Firestorm. 

It is certainly the Simulator side (at the LL side). The viewer, however, *does* strongly appear to play a role in the likeliness of the disconnect happening though.

I use Catznip viewer (my main) - and *rarely EVER have TP or border-crossing issues.

I use the Official Viewer (a.k.a. "LL Viewer") - and still have a decent performance with few TP and border-crossing issues, though Catznip outperforms this one for me.

I use Firestorm viewer and (*for ME*) it is the worst-performing, most often (likely) TP and border-crossing issues I experience.

I use Kokua Viewer and get similar performance as the Official viewer.

I can only express what I, myself, experience with these viewers. I always have, currently do, and always will very strongly advise anyone and everyone: try different viewers. Because even without these glitches that we experience now, different viewers will perform differently.

[Qie Niangao]

17 minutes ago, JoyofRLC Acker said:

An example of the contextual stuff is the apparent fact that TP triggered by scripts are less likely to fail than ones initiated by user action.  

This is news to me. I've had TP disconnects either way, but not enough volume to say one way or another. I'm assuming that within-sim teleports are excluded in both cases; of course within-sim TPs are very often scripted, but they're probably not part of the distinction being made here. I don't think I've ever heard of anybody getting disconnected during a within-sim teleport but come to think of it, that does kinda reinforce the relationship between the current sim-crossing and teleport problems.

I didn't mean to make any point at all about "reports"... I guess I shouldn't have used that word because I really meant all the data about the problem, whether reported by users, sim-monitoring code, statistics from normal operations, or anything else. I agree they almost certainly installed some code for collecting extra data on sim-crossings as well as teleports, but the teleport problems are more "all-or-nothing" in the sense that we (or I, at least) never got them and now we do, whereas I've gotten sim-crossing disconnections (yes, I mean disconnections) for as long as I've been in SL (although, to be fair, I have a crappy Canadian ISP, so others may not have as much trouble).

As I was typing, I was reminded of another challenge in diagnosing race conditions: a few lines of debug code can completely change the triggering circumstances, sometimes completely masking the whole event, or shifting it unrecognizably. They really are the worst bugs.

[MBeatrix]

31 minutes ago, Aishagain said:

This surprises me.  I thought the sim-border ghost had been eliminated a few years back during the tuning of server-side appearance.  I once had the slightly surreal experience of disconnecting at a sim, reclogging and being told by my partner that my avatar was still where I had disconnected.  She even took a photo of it!

What REALLY spooked me was on returning to that sim I was confronted with my own avatar still in situ.  As many others have since found, left clicking on that avatar caused it to vanish.  On reporting this (including photos) to LL Support, I was told such a phenomenon was "impossible".

I too have recently seen an increased number of ghosts at region edges.

On the subject of ignores, ignoring Solar is a mistake, since, despite himself, he actually DOES sometimes talk sense!

Perhaps I should clarify that the above incident occurred to my alter ego, not this account.

 

Support person didn't know what they were talking about — something that used to happen quite often. I haven't contacted support for over a year, I believe, but since LL's CEO changed Support got much better.

The increased number of ghosted avatars is due to an increased number of crashes/disconnects crossing regions. This is an issue that never went away, so after some point I started taking it as a "normal" thing. Nothing to worry about, I think.

[Solar Legion]

The difference between clients as described by both Fluf and Alyona is a difference that has several variables which include but are not limited to: Network speeds, internal timeout margins, hand off channels used, some system hardware ....

Fun one for both: With very few exceptions, I haven't had many issues. TP disconnects? Only happened sometime last week and even then at the "peak" of all of this. Sim crossing? I don't (and generally never have) bothered walking between regions/sims or taking a vehicular trip.

No, I've had a few seemingly random disconnects and am one of the Premium users that can no longer acquire Voice Morphs. That's really about it.

Funnily enough, I use Firestorm. Also funnily enough the computer being used runs Linux (up to date kernel, X Ubuntu, 18 series) ...

ETA: Before anyone chimes in with "But Solar, why list the OS? It doesn't matter!" - It's a variable.

Edited April 23, 2019 by Solar Legion

[Fluf Fredriksson]

8 minutes ago, Solar Legion said:

The difference between clients as described by both Fluf and Alyona is a difference that has several variables which include but are not limited to: Network speeds, internal timeout margins, hand off channels used, some system hardware ....

Fun one for both: With very few exceptions, I haven't had many issues. TP disconnects? Only happened sometime last week and even then at the "peak" of all of this. Sim crossing? I don't (and generally never have) bothered walking between regions/sims or taking a vehicular trip.

No, I've had a few seemingly random disconnects and am one of the Premium users that can no longer acquire Voice Morphs. That's really about it.

Funnily enough, I use Firestorm. Also funnily enough the computer being used runs Linux (up to date kernel, X Ubuntu, 18 series) ...

ETA: Before anyone chimes in with "But Solar, why list the OS? It doesn't matter!" - It's a variable.

Network speed - Same for all tests, Internal timeout margin? - Yes could well be different between viewers. That's what I was trying to point out. Despite LL claiming all viewers have the same problems, they do not. Hand off channel used - ?? Care to explain ??. Some system hardware - Same for all tests. OS - Up to date Linux

I specifically said my testing is on sim crossings. Welcome to the "I only TP sometimes and I don't have much problem" club. Take a seat, read a book.

[Solar Legion]

So you're testing on something that Linden Lab isn't even - according to the Status Page anyway - working on?

That's nice.

And yes, the claim that all the clients have the "same" problem is quite accurate: The difference is in the degree. Any client with a longer internal timeout margin is going to see said problem quite a bit less. This does not mean they do not have the problem as in order to not have said problem - on the client end - you'd have to either disable that internal timer if possible or have it be an absurd length of time.

Your system hardware being the same for all tests is irrelevant. Your hardware is not the same as my hardware or that of Alyona or Whirly or Qie or ... Get the point on that? It's a variable.

Hand off channels - which protocol is being used/ports being used (the latter of these should be universal between client variants) and would be, client side anyway, transferring very little actual information. I should have tossed that as server side.

Server crossings have always been a bit flaky - hence part of why I do not bother walking or using vehicles.

Now, pop quiz: What was the point I was making with my earlier post?

Edited April 23, 2019 by Solar Legion

[MBeatrix]

I just went from Nuggy to Honah Lee Trudeau (as usual, via North of Nautilus City and turning South at Crows Nest) at high speed — and I mean HIGH speed — on two different motor boats and back without any issues. Actually, crossings were quite smooth.

Many of the sims I went through were restarted afresh today with the new code, so it's early to know if it all going so well is due to the new server version or part of it (no idea to what extent, of course) was due to the fact they were restarted.

[EDIT] Running Firestorm, as always.

[EDIT 2] Oops! No, I was wrong — most regions I went through are on the Main Channel and RC Magnum. I just hopped around them to check. (laughing at myself now)
So, I went at high speed through more than 50 regions and only two of them got what was rolled today. As I wrote above, no issues at all, and crossings were even quite smooth.

Edited April 23, 2019 by MBeatrix
adding more info & corrections

[Fluf Fredriksson]

33 minutes ago, Solar Legion said:

So you're testing on something that Linden Lab isn't even - according to the Status Page anyway - working on?

That's nice.

And yes, the claim that all the clients have the "same" problem is quite accurate: The difference is in the degree. Any client with a longer internal timeout margin is going to see said problem quite a bit less. This does not mean they do not have the problem as in order to not have said problem - on the client end - you'd have to either disable that internal timer if possible or have it be an absurd length of time.

Your system hardware being the same for all tests is irrelevant. Your hardware is not the same as my hardware or that of Alyona or Whirly or Qie or ... Get the point on that? It's a variable.

Hand off channels - which protocol is being used/ports being used (the latter of these should be universal between client variants) and would be, client side anyway, transferring very little actual information. I should have tossed that as server side.

Server crossings have always been a bit flaky - hence part of why I do not bother walking or using vehicles.

Now, pop quiz: What was the point I was making with my earlier post?

What makes you say I'm testing on something that Linden Lab isn't working on? That doesn't make sense.

Yes all clients are affected. As I said in my post, the source of the problem is server side. Some clients however are having a lot more trouble than others. You said "Your friend is showing bias, based on their own experience and conjecture" .. no they are possibly not. Different clients are handling the current server problems differently.

WOW do we all use different computers? It's amazing anyone ever debugs anything! *_* tell me more of your wisdom! You are the one that raised hardware as a possible issue in testing. I was simply stating that all my tests are on the same hardware. They show a clear difference in some client sim crossing disconnect rates under the current circumstances. Are we clear on that?

Now pop quiz. Will you change your statement "Your friend is showing bias, based on their own experience and conjecture" ? Because it's clearly wrong.

Completed Yavascript routes during testing sim disconnection issues:

C1, C2/N2, C3, C4-Tea1, C4-Tea2
H1, H2, H3, H4, H6, H7, H8
G1-North, G1-South, G2, G5, G6, G7
J1, J2, J3, J4, J5, J6, J7
M1, M2, M3, N1
S0, S1, S2, S3, S4, S5, S6, S7, S8
T1, T2
Z1, Z2, Z3, Z4,

Incomplete routes
C4-Tea1
G7
H5
N3

That's evidence of the testing I've been doing.

Would you care to provide me with evidence of extensive testing that you've been undertaking that supports your opinions?

[Solar Legion]

Ah yes, because the only valid opinion is that of a user who tosses their time into diagnosing a problem ...

While that is not what you are directly saying, that is how it comes across. Word things a bit more carefully.

Sorry, not retracting my statement as there is nothing at all factually wrong with it: The user was basing their opinion on a bias (be it personal or otherwise) and conjecture. No substance was given, no reasoning behind their statement made.

I raised hardware as a variable specifically due to no two users having the same hardware. If you thought there was some other meaning there, it's on your own end. Stop reading things into a user's posts that are not explicitly stated.

Test all you'd like. Meanwhile I'll be going about my day - on and off of Second Life - as I've done since the day I first logged in. Get hit with TP or Region Crossing issues? I'll log back in - after checking Grid Status - and try again. Problems persist? I'll try again tomorrow.

It's called patience, something I have for most technical issues and something I lack when dealing with people. And yes, I'm quite fine with that.

Now then, do you always have such a hard time stowing your attitude and remaining objective?

No need to answer that, your current responses are answer enough.

Have fun and have a good Second Life.

Edited April 23, 2019 by Solar Legion

[Fluf Fredriksson]

1 minute ago, Solar Legion said:

Ah yes, because the only valid opinion is that of a user who tosses their time into diagnosing a problem ...

Sorry, not retracting my statement as there is nothing at all factually wrong with it: The user was basing their opinion on a bias (be it personal or otherwise) and conjecture. No substance was given, no reasoning behind their statement made.

I raised hardware as a variable specifically due to no two users having the same hardware. If you thought there was some other meaning there, it's on your own end. Stop reading things into a user's posts that are not explicitly stated.

Now then, do you always have such a hard time stowing your attitude and remaining objective?

No need to answer that, your current responses are answer enough.

Have fun and have a good Second Life.

I have given you substance and reasoning to support their position over many hours and many hundreds of sim crossings. You are choosing to ignore that well informed advice because it doesn't suit your narrative. I am the one being objective. You are the one who is being subjective by ignoring well research information because it is contrary to your un-researched opinion. Others on here already have you marked as a block worthy troll. I hope I've managed to make that view seem more reasonable for others in this thread.

Have a nice day.

[Solar Legion]

That's nice.

So none of your responses were made in good faith and held the sole intention of leading/feeding your personal conclusion.

Good to know.

Have fun wasting your day.

[Dakota Linden]

Greetings all!

We appreciate and understand that this issue is a royal pain for everyone and is causing many users upset and irritation, but taking it out on each other is not appropriate.  Linden Lab IS working on the issue.

If you believe you have found something that you feel is pertinent to the issue, please submit a Bug Report. 

Please play nice, and if you believe you cannot, then please refrain from posting to the forums.  Doing so may result in your own account being suspended, and that is never fun. 

 

[JoyofRLC Acker]

1 hour ago, Qie Niangao said:

This is news to me. I've had TP disconnects either way, but not enough volume to say one way or another. I'm assuming that within-sim teleports are excluded in both cases; of course within-sim TPs are very often scripted, but they're probably not part of the distinction being made here. I don't think I've ever heard of anybody getting disconnected during a within-sim teleport but come to think of it, that does kinda reinforce the relationship between the current sim-crossing and teleport problems.

I didn't mean to make any point at all about "reports"... I guess I shouldn't have used that word because I really meant all the data about the problem, whether reported by users, sim-monitoring code, statistics from normal operations, or anything else. I agree they almost certainly installed some code for collecting extra data on sim-crossings as well as teleports, but the teleport problems are more "all-or-nothing" in the sense that we (or I, at least) never got them and now we do, whereas I've gotten sim-crossing disconnections (yes, I mean disconnections) for as long as I've been in SL (although, to be fair, I have a crappy Canadian ISP, so others may not have as much trouble).

As I was typing, I was reminded of another challenge in diagnosing race conditions: a few lines of debug code can completely change the triggering circumstances, sometimes completely masking the whole event, or shifting it unrecognizably. They really are the worst bugs.

Sorry I mis-interpreted - thought you were making a distinction but you weren't really.   For the sake of clarity, my scripted TP are just about always into different sims (and as I say they are nearly always successful even now).  The connection (haha) between regions crossings and TP is that, according to what I was told, a region crossing in fact involves a TP, the agent and the vehicel cross separately and then reattach.  But no idea of the credibility of my sources.

Until now, region crossing disconnects were VERY rare for me, unlike falling off and cam dislocations which were more common.   I wonder if somehow the latter problems have turned into the former.   ie the underlying cause was always there, but with some new code somewhere its being handled differently by the servers / viewers, and so a different symptom for what is in fact the same cause.

Finally - Heisenberg is alive and well!

[JoyofRLC Acker]

RC channels ... to the extent they are seamlessly attached to the main grid and are scattered as it were they are 'production' sims for all practical purposes.  In most places you cant sail on "just" RC channel or "just" main channel (Blake Sea aside).   Code released to production should be tested. 

[MBeatrix]

20 minutes ago, JoyofRLC Acker said:

RC channels ... to the extent they are seamlessly attached to the main grid and are scattered as it were they are 'production' sims for all practical purposes.  In most places you cant sail on "just" RC channel or "just" main channel (Blake Sea aside).   Code released to production should be tested. 

It would be possible to test it more or less properly if the new code had been rolled to RC Magnum, at least for me, as there are quite some contiguous Magnum regions I cross daily.

[Profaitchikenz Haiku]

Regarding different viewer experiences, if one viewer seems to crash less often than another, having different cache content in them might also be a factor? A proper test would be to run each of them with a freshly emptied cache.

I've tried the latest official viewer, Catznip, Firestorm and Singularity 7509, and have started them all off on empty caches (this dates from the "are we there yet?" experiments in another thread), and I can't pin down any particular viewer, any particular region, or any particular configuration of AO and scripted wardrobe content that is more prone to crashing.