What's up with sim crossings?

[Fluf Fredriksson]

A bit early for 100% accuracy after the "should reduce" sim roll out fix. But.. It is about a month since all this became a noticeable problem, and it's all we have so far. I'm not sure if it has improved TP's yet. Hard to say with such a random problem. But it looks like it hasn't helped with sim crossings at all.

It's a bit worrying that LL keep emphasising they are working on Teleport Failures, and never mention sim crossings. Presumably they hope that if they manage to figure out the cause and fix TP's, it will magically also cure sim crossings. While the two things do have a lot in common, there is a main difference. You can increase network timeouts for TP's and people won't mind waiting a few seconds more (if that helps fix it), but if you do that for sim crossings, you further degrade an already sometimes unpredictable experience. Vehicles have time to de-rez or travel to the next sim while you're stuck waiting, your avatar has time to fly over 3 more sims with no control... Sim crossings are dynamic, teleports are static.

Predictably for tech problems at LL. It's not looking great one month later.

It's worrying they still don't seem to have any real idea what the root of the problem is, even though it must be something they introduced over the last couple of months or so.

It's a worry that they seem fixated on trying to fix Teleports and not Sim Crossings when both are a problem, and both will cause people to "log out and leave it".

Oh well. Here's to the next "should reduce problems" fix.

[Sister Nova]

51 minutes ago, Fluf Fredriksson said:

It's a worry that they seem fixated on trying to fix Teleports and not Sim Crossings when both are a problem, and both will cause people to "log out and leave it".

I think teleports are probably used more than sim crossings generally, as most stores and events are situated on stand alone sims. It would be interesting to canvas an event on a mainland sim though to see how many had arrived there by crossing sims rather than teleporting in.

[MBeatrix]

 

@Fluf Fredriksson

What is really worrying is that they don't know what they have done... Or if they do, why have they done it, then? It doesn't hold much hope for the future.

[Fluf Fredriksson]

8 minutes ago, Walpurgisnacht said:

I think teleports are probably used more than sim crossings generally, as most stores and events are situated on stand alone sims. It would be interesting to canvas an event on a mainland sim though to see how many had arrived there by crossing sims rather than teleporting in.

People who regularly cross sims aren't usually heading for an event. They are either using vehicles on road / water / air to travel around the continents of SL, or are simply flying around or wandering on foot to explore the continents.

So your survey would prove that there's no need to fix sim crossings at all (and ignore all the people who regularly try and cross sims)

[Fluf Fredriksson]

2 minutes ago, MBeatrix said:

 

@Fluf Fredriksson

What is really worrying is that they don't know what they have done... Or if they do, why have they done it, then? It doesn't hold much hope for the future.

Well it's back to conjecture time. My best guess is that EEP itself probably isn't the problem, but either the EEP roll out or the server change on the 20th March pushed things over the edge and exposed problems that had already been introduced in both the LL viewer code and sim code.

There's no way LL will want to admit defeat and roll back the EEP changes, so now they are stuck trying to fix the underlying problems which they weren't previously aware of.

I say both viewer code AND sim code, because despite LL's official announcement, you can still cross hundreds of sims using Henri's v1 based "CoolVL". Even during all the current TP / crossing disconnect problem. I managed the entire H7 Yavascript pod route using CoolVL without a disconnect (well over 2 hours of constant sim crosssings). Using Firestorm at the moment the absolute maximum I've reached is 45 minutes of Yavascript pod riding before a disconnect. Usually it's 10 to 20 minutes.

So viewers based on LL's current viewer code (like Firestorm) are far more ready to give up and give you a disconnect message than an ancient v1 based viewer. For the record, I'd only recommend CoolVL if disconnects are your main priority. It still has problems with teleports sometimes at the moment (although I think less often), and I'm sure many people would rather stick with their current viewer and suffer the disconnections than try and deal with CoolVL.

[MBeatrix]

14 minutes ago, Fluf Fredriksson said:

Well it's back to conjecture time. My best guess is that EEP itself probably isn't the problem, but either the EEP roll out or the server change on the 20th March pushed things over the edge and exposed problems that had already been introduced in both the LL viewer code and sim code.

[...]

Making short what you wrote there: they don't know exactly what they are doing. 😞

[Sister Nova]

23 minutes ago, Fluf Fredriksson said:

People who regularly cross sims aren't usually heading for an event. They are either using vehicles on road / water / air to travel around the continents of SL, or are simply flying around or wandering on foot to explore the continents.

So your survey would prove that there's no need to fix sim crossings at all (and ignore all the people who regularly try and cross sims)

'My' survey would not be to prove anything either way, it was just a thought.

I can only speak for myself, in that I cross sims way more than I teleport. I currently have 8 properties and travel crossing MANY sims to 7 of them. The 8th one, which is on a different land mass to the others, I travel to using an airport on each continent that have a teleport link between them. I cross many sims to travel to and from the respective airports. I do all my shopping (except when buying a vehicle that I want to demo first) through the marketplace, so am very much in the 'sim crossing camp'.

I would say, in my own experience this last week or two, that I have had more teleport failures than crossing failures, and considering my travel habit, I cross way more sims than teleport, which surely would prove the need to be more dire with regard to teleports.

[Fluf Fredriksson]

17 minutes ago, MBeatrix said:

Making short what you wrote there: they don't know exactly what they are doing. 😞

Well ... no.

Apparently there's been fairly regular staff turn over at LL over the years, so not many (if any?) of the people there now will have the kind of in depth knowledge of the code that the original coders would have. Pile on top of that a reluctance to improve on or solve problem areas that have no visible financial gain for years now (accountancy led project management at it's finest), see long standing JIRA issues for proof. Anddddd ... No ... No they don't. Not any more.

If they can't be bothered to debug why sometimes you can't see the sim next to you for several years. Don't be surprised if they can't debug why teleports and sim crossings fail. Until it looks like it will cost them money, they aren't interested, and when it does look like it will cost them money, they haven't invested any time or effort in being ready to fix the problem.

¯\_(ツ)_/¯

[Fluf Fredriksson]

21 minutes ago, Walpurgisnacht said:

I would say, in my own experience this last week or two, that I have had more teleport failures than crossing failures, and considering my travel habit, I cross way more sims than teleport, which surely would prove the need to be more dire with regard to teleports.

 

Well, as previously said in the thread. The problems people are having are very random. Sometimes you get a few days with very few problems, then boom, you get a few terrible days in a row. I've had both teleport and sim crossing failures, using Firestorm I can't cross sims for more than about 20 minutes before I get a disconnect message. Teleports sometimes work, sometimes don't. We sort of have to rely on LL collecting the stats on how many disconnects they are seeing and of which type. It seems any individual's experience is unique.

[Qie Niangao]

Y'all saw this, right? It's the theory that the new Linux version recently rolled-out includes a subtle race condition that can cause teleport disconnects. I can't vouch for that one way or another, but I've done enough system updates to know that they can uncover bugs deep, deep in the code, undiscovered by the original programmers and QA.

(I'm also just assuming that the current increase in sim crossing troubles has the same root cause as the teleport disconnects. That may not be true at all, but in lieu of countering evidence for now it seems parsimonious.)

[MBeatrix]

26 minutes ago, Qie Niangao said:

Y'all saw this, right? It's the theory that the new Linux version recently rolled-out includes a subtle race condition that can cause teleport disconnects. I can't vouch for that one way or another, but I've done enough system updates to know that they can uncover bugs deep, deep in the code, undiscovered by the original programmers and QA.

(I'm also just assuming that the current increase in sim crossing troubles has the same root cause as the teleport disconnects. That may not be true at all, but in lieu of countering evidence for now it seems parsimonious.)

The thing is, whatever was rolled yesterday to the whole grid didn't fix the issues. Only made it a bit less bad, maybe. We'll see how it goes till the end of next week. The funny thing — not — is that teleports seem to be working better and crossings got worse, at least for me.

Edited April 19, 2019 by MBeatrix
correction

[Fluf Fredriksson]

45 minutes ago, Qie Niangao said:

Y'all saw this, right? It's the theory that the new Linux version recently rolled-out includes a subtle race condition that can cause teleport disconnects.

 

systemd strikes again :) (Jessie introduced systemd to Debian)

[MBeatrix]

Teleport failures are striking hard now. Just got two within less that 10 minutes, trying to do the CarperTech Eggs Hunt. This is going to kill Fantasy Faire, as crossing sims isn't working there most of the time... 😥

[EDIT] And another teleport failure... So, 3 in about 15 minutes.

Edited April 20, 2019 by MBeatrix
adding more info

[Aishagain]

I have to back up MBeatrix' opinions.  TP disconnects do actually seem worse today.  Maybe factors outside LL are involved in the triggering of these disconnects, but whatever the cause, it is impacting online numbers noticeably.  I'd estimate it is now more than 10% down, and while the Easter holiday will have an impact some of the more populous events, such as Fantasy Faire, are being badly hit by this bug.  THAT will get LL's attention but the question remains: what to do?  I know they are trying but the old school comment comes to mind "must try harder". 

Edited April 20, 2019 by Aishagain

[MBeatrix]

7 minutes ago, Aishagain said:

I have to back up MBeatrix' opinions.  TP disconnects do actually seem worse today.  Maybe factors outside LL are involved in the triggering of these disconnects, but whatever the cause, it is impacting online numbers noticeably.  I'd estimate it is now more than 10% down, and while the Easter holiday will have an impact some of the more populous events, such as Fantasy Faire, are being badly hit by this bug.  THAT will get LL's attention but the question remains: what to do?  I know they are trying but the old school comment comes to mind "must try harder". 

If what Fluf Fredriksson wrote above is true, they are now paying for not having fixed some long standing bugs. And maybe some of them should go back to school. As you can tell, I'm very upset with all this.

[Fluf Fredriksson]

20 minutes ago, MBeatrix said:

If what Fluf Fredriksson wrote above is true, they are now paying for not having fixed some long standing bugs. And maybe some of them should go back to school. As you can tell, I'm very upset with all this.

While I'll stand by what I wrote above, it's still basically guesswork as to what the current problem is. Deeper knowledge of the system would well help them debug it faster, having less long standing undiagnosed bugs in the code would probably make it more resilient to problems like this, but the source of the current problem is apparently unknown to all. [My Magic 8 Ball says .... "systemd" .. hm .. shakes it again]

[Qie Niangao]

If it turns out that the OS upgrade has anything to do with the current problems, then indeed it's related to the move to the cloud, inasmuch as the OS upgrade is seen as a prerequisite for that eventual move. Eventually.

I have to admit that in my limited excursions, I haven't even noticed sim crossings being worse than usual, but they're always pretty awful for me, so it's a signal-to-noise problem I guess. Teleports, however, are intermittently dreadful now -- and then everything clears up and it's all back to normal and I'd think it just something flaky with my often-flaky internet connection were it not for everybody else having troubles too.

As unfortunate as the immediate effect this is having on current in-world events, it also has a pretty tragic opportunity cost: before this all started, next up for focused attention was to be the weird degradation in sim script performance some of us have been watching develop in recent months. Not to rehash all that here, but it's another nest of dragons that could bite us at any time.

I must say, the current server developers are of intrepid stock. I hope they're well appreciated and compensated. Boy, do I hope that.

[MBeatrix]

47 minutes ago, Qie Niangao said:

[...] I must say, the current server developers are of intrepid stock. [...]

I just hope you are right, Qie... And yes, I was overreacting back there.

[Trin1]

I hope they get this issue sorted out soon, it is quite annoying.

[Aishagain]

They say a little knowledge is a dangerous thing...I've been reading about "systemd" in Linux and I am not feeling comfortable about what I've read.  According to the article on Wikipedia I read, systemd has shown some serious flaws in testing and is vulnerable to several types of DDoS attacks.  This is hardly something that LL should be incorporating into server code, surely?

Having said that it seemed to me on my time inworld that TP is even more intermittent today than it was a week ago.  I was never terribly bothered by that bug until a few days ago, though the sim-crossing bug was a pain whether sailing OR gliding. High-speed crossings were just about impossible to control and low speed ones were a lottery!

In my experience (highly subjective of course) LL have NOT got a handle on this and next week had better see some genuine improvement or SL will take a serious body-blow, and at the moment it "cannae tak much more o' this, Cap'n".  It's logical, Jim.

By the way, I assume the disappearance of the EEP "Black Stars" issue is a consequence of those last-minute rolls on Thursday?

Edited April 21, 2019 by Aishagain

[MBeatrix]

20 minutes ago, Aishagain said:

[...]

By the way, I assume the disappearance of the EEP "Black Stars" issue is a consequence of those last-minute rolls on Thursday?

No idea. And also no idea why no one else mentioned that.

[Whirly Fizzle]

16 hours ago, Aishagain said:

By the way, I assume the disappearance of the EEP "Black Stars" issue is a consequence of those last-minute rolls on Thursday?

I believe a fix was bundled in for the black stars.  The black stars are still there but they are much smaller & less noticeable now.
Problem is, they broke every region windlight that has any fog or non default sun position in the process...  so now it's a million times more broken.

Bug report: BUG-226815 - [EEP] Since the grid was rolled to 19.04.15.526263 yesterday, region windlight is broken for all legacy viewers - settings reset on Apply.

[Qie Niangao]

1 hour ago, Whirly Fizzle said:

19.04.15.526263

First, thanks for the information. It's more than we're seeing in the release notes, that's for sure. At this point, I really think @Oz Linden needs to direct Development and Operations to practice radical transparency about this mess with TP disconnects, region crossings, and I guess EEP. Maybe the OS upgrade stuff, too, to the extent they're investigating that as a contributing factor. I mean, we're all in it together now, and I feel like I'm stuck on a plane on the tarmac with no word from the pilot.

[Whirly Fizzle]

5 minutes ago, Qie Niangao said:

First, thanks for the information. It's more than we're seeing in the release notes, that's for sure. At this point, I really think @Oz Linden needs to direct Development and Operations to practice radical transparency about this mess with TP disconnects, region crossings, and I guess EEP. Maybe the OS upgrade stuff, too, to the extent they're investigating that as a contributing factor. I mean, we're all in it together now, and I feel like I'm stuck on a plane on the tarmac with no word from the pilot.

The best info we have came from Vir at the CCUG.
Inara blogged it here: https://modemworld.me/2019/04/19/linux-jessie-update-a-cause-of-sl-tp-issues/

I don't know what happened to cause the further windlight breakage after that OS rollback.
The EEP regions did have the OS update bundled in from quite early on so I guess rolling back the OS update has broken some server stuff for EEP & how EEP regions affect legacy viewers or there was a bad merge of the EEP server stuff back into the rolled back server code or there was a deliberate fix for the black stars bug that actually broke "legacy" windlight even more.

Worst bit is that the TP disconnects don't seem to be fixed either so I wonder what's going to happen next.
I'd be very interested to know of LLs stats show any improvement in the TP disconnects since the grid was rolled back.

There's an interesting theory about the cause of the TP disconnects here: BUG-226825 - Reason why teleports fails to many people
Honestly no idea if that theory has any merit - I doubt it but this is SL & I've learned to expect the unexpected  lol.

[Fluf Fredriksson]

2 hours ago, Whirly Fizzle said:

There's an interesting theory about the cause of the TP disconnects here: BUG-226825 - Reason why teleports fails to many people

Honestly no idea if that theory has any merit - I doubt it but this is SL & I've learned to expect the unexpected  lol.

 

It seems very very very unlikely that the February DNS changes would be biting SL in the ass. One of the things I spend a while quadruple checking when the disconnects started happening was DNS queries. I couldn't find any significant problems getting DNS replies for any of LL's servers (but I did slightly improve my DNS caching and response times).

The rest of your summary describes a painful predicament for LL to be in. With so many potentially contributing changes it's going to be hell eliminating possible causes, or even just rolling back to a previously known good state. Practically impossible in fact since they seem intent on keeping EEP running. The previously known good state didn't have EEP rolled out.

I think by now the LL crew need to be nearing panic mode. I can't see many new users making it past a day or two before they decide "SL is buggy and why would anyone pay money to log in to that?". When there's still no fix after a month, it's time to attempt damage limitation by at least letting the users know what's going on in a lot more detail.