Sim Crossing far worse since Last Tuesday?

[Anna Nova]

It might be my imagination, but sim crossing in vehicles seems to have got far far worse since the reboot of the servers a week ago.  I keep getting totally borked in the Blakes Sea region.  Has anyone else noticed this?  They may be just picking on me.

 

[Whirly Fizzle]

@Caleb Linden
@Mazidox Linden
@Bugsly Linden

[Kyrah Abattoir]

I know that Spangle currently has an issue so vehicles get eaten as you region cross into it.

[animats]

26 minutes ago, Kyrah Abattoir said:

I know that Spangle currently has an issue so vehicles get eaten as you region cross into it.

Just tried; not seeing that. I've been back and forth the Spangle/Satellite and Spangle/Plusia borders without problems. An SLRR rail vehicle just went through OK, too.

[Qie Niangao]

I too saw some vehicle traffic through Spangle, but the sim is having some problems. A little time dilation and delayed scripts due to unusually high Physics time, and something especially funky with (at least) Experience KVP scripts being very slow or worse, even though some other scripts are only moderately slowed.

[EDIT: oh, wait, no, Spangle has huge overall script delay, with less than 1% of scripts running each frame. I was momentarily misreading that statistic as the fraction of scripts run, not the percentage, so two orders of magnitude worse than I thought.]

Edited May 17, 2018 by Qie Niangao

[Callum Meriman]

We know there were changes around region crossing pushed out to the whole grid after just one week in Snack in the Blake Sea. Peopl were complaining about it there too, but the change looks to have been rushed out with only the one weeks test.

Mostly they seem quicker and smoother to me, but there are quite a few cases where my camera shoots up a few hundred meters in the air, or travels along beside me.

Worse? Not sure, just different.

 

[Callum Meriman]

Oh yes, experience teleport scripts have been delayed up to 40 seconds for me!

[animats]

3 hours ago, Callum Meriman said:

There are quite a few cases where my camera shoots up a few hundred meters in the air, or travels along beside me.

Camera shoots up and comes back, or stays in the wrong position? If it comes back, that's probably a viewer bug. If it stays wrong, server bug.

I think there's a viewer bug where position updates, which have region-local coordinates, sometimes get interpreted relative to the wrong region. Firestorm's network logging doesn't display coordinates from the ImprovedTerseObjectUpdate message, which is what's needed to diagnose this problem. (Message logging is normally off. If you turn it on, you see the message types and sequence numbers, but not much content from the messages.)

Self-compiling of Firestorm on Linux is broken at the moment. The Firestorm devs are trying to digest the latest LL changes, and the self-compiled version is out of sync. I've been able to build a version, but I had to pull pieces from three repositories to do it, and it wasn't current, so finding bugs in it annoyed the Firestorm devs. So I'm not going to take a look at this until the Firestorm devs get a usable Linux source tree again.

Edited May 17, 2018 by animats

[Callum Meriman]

37 minutes ago, animats said:

Camera shoots up and comes back, or stays in the wrong position? If it comes back, that's probably a viewer bug. If it stays wrong, server bug.

Stays wrong. The fix is to move the camera back, and lock it into position during the next sim crossing, where the numbers seem to reset to be correct.

My main concern with this, is that Snack was only there for one week, and yet, despite the number of people saying things were weird, it was rolled out grid wide. Not great change management on the part of the Lindens, in my eyes.

Edited May 17, 2018 by Callum Meriman

[Amanda Crisp]

Even teleports have gotten worse for me since then . If I TP to any busy sim I crash-upon-arrival and have to wait for my session to time out to get back in.

Edited May 17, 2018 by AmandaKeen

[Anna Nova]

3 hours ago, animats said:

Camera shoots up and comes back, or stays in the wrong position? If it comes back, that's probably a viewer bug. If it stays wrong, server bug.

I think there's a viewer bug where position updates, which have region-local coordinates, sometimes get interpreted relative to the wrong region. Firestorm's network logging doesn't display coordinates from the ImprovedTerseObjectUpdate message, which is what's needed to diagnose this problem. (Message logging is normally off. If you turn it on, you see the message types and sequence numbers, but not much content from the messages.)

Self-compiling of Firestorm on Linux is broken at the moment. The Firestorm devs are trying to digest the latest LL changes, and the self-compiled version is out of sync. I've been able to build a version, but I had to pull pieces from three repositories to do it, and it wasn't current, so finding bugs in it annoyed the Firestorm devs. So I'm not going to take a look at this until the Firestorm devs get a usable Linux source tree again.

Do we take it that you are subcontracted by the Lab to fix these mistakes?

I, for one, would like the Labs to comment on this.  Actually I think they should immediately back out the mistake.

I do use Linux, but my partner, who travels with me most of the time, uses that Microsoft proprietary stuff, and she gets borked at the same crossings that I do.

The usual effect is that the vehicle continues, but both avatars are displaced about a sim away (in different and random directions as far as I can see), they cannot be brought back to the vehicle, and eventually, after several minutes, end up together stuck in a sim crossing with left and right movement inhibited.  Only a re-log works, and if we are lucky we can then re-sit - it depends if the vehicle hits EoW first or not.

[Anna Nova]

10 hours ago, Kyrah Abattoir said:

I know that Spangle currently has an issue so vehicles get eaten as you region cross into it.

That's been like that for months though.  Not related to the changes made by LL at the last reboot.

[Callum Meriman]

28 minutes ago, anna2358 said:

Do we take it that you are subcontracted by the Lab to fix these mistakes?

With the greatest respect for you anna, and to be quite fair, animats has been closely watching sim crossings for a few months and has without doubt done a lot of good and valuable work in that direction. As we very correctly respect Whirly for her brilliant ability of knowing the client back to front, to support people with politeness and class, and see exactly which Jira is tied in with any problem, we should also recognise animats is not a run of mill noob on the issue of sim crossings.

I, for one, would like the Labs to comment on this.  Actually I think they should immediately back out the mistake.

I agree they need to comment or look. It's not worse, or too much worse, but it's quite different, mostly though the mere one week test as Snack worries me no end. That seems highly sloppy in change management terms, as an IT professional.

Edited May 17, 2018 by Callum Meriman

[Anna Nova]

16 minutes ago, Callum Meriman said:

With the greatest respect for you anna, and to be quite fair, animats has been closely watching sim crossings for a few months and has without doubt done a lot of good and valuable work in that direction.

Point taken.  No offence intended.  I'm sure the Labs can use every bit of help they can get, and knowledgable and expert help, like Animats and Whirly, in particular.

[Qie Niangao]

Pretty sure I found the Spangle problem. There are over 30,000 active scripts in the sim, with a rezzer and a cloud of spam balls around 119, 99, 3627 delaying all scripts. I filed a "Region Performance" support ticket on the theory that "Region Offline" would get attention but only for a restart and that an Abuse Report would fall into the Governance black hole from which no information can escape.

[Theresa Tennyson]

2 hours ago, anna2358 said:

Do we take it that you are subcontracted by the Lab to fix these mistakes?

I, for one, would like the Labs to comment on this.  Actually I think they should immediately back out the mistake.

 

I remember people seeing very similar things and having almost exactly the same reaction right after the roll-out of the increased land impact allowance, and those problems soon went away without any changes that I could see being necessary. Personally I'm not seeing anything out of the ordinary and I've been doing a lot of high-stakes region crossings lately.

One thing that may be a factor is when region A's server code is changed but region B hasn't been re-started since region A's code has changed, therefore meaning that region B expects region A to be running a different version of the code than it is. With the way the server code is rolled out this will happen often - I think this was exactly the problem at the time of the region capacity increase and may be the same situation now. If so, it should go away when all regions are restarted after they all get the new code.

Edited May 17, 2018 by Theresa Tennyson

[Anna Nova]

I got zapped twice in 10 minutes this morning.  Halsey to Zumwalt at 2.00Am PDT, and then VonSpee to Hipper 2.09.  I then gave up.

In both cases my crossing speed was under 9m/s, and the vehicle itself survived, but I had to relog to regain fly-left-right control.

This sort of thing makes SL sailing, GTFO!, and even just romantic cruising impossible.  So for me, it either gets fixed soon, or I will be off to pastures new.

[Klytyna]

24 minutes ago, anna2358 said:

This sort of thing makes SL sailing, GTFO!

Ah, sailing... Right...

Depending on exactly where you sail, you may find that sim crossings will involve passing from Main Channel to one or more of the RC channels, the area around TWYC, over on Dex, top end of the Blake used to be very bad for this.

Especially if the different channels are running different proposed 'fixes' for sim crossing problems...

The "Sim Crossing Hero" has manly been concentrating on motorbikes and cars, on the Madlands roads, especially Kama City, and less so on boats and planes, check his thread, even he has made posts where he thought he'd solved it all and then got in a boat...

His fix requires a custom modded Whinux FS viewer, and custom vehicle scripts that your boat probably doesn't have.

So... Wrong viewer, wrong script set, wrong vehicle type, and LL playing games trying assorted 'micro fixes' on different RC channels all over your planned route, so coming from or going to the wrong server version...

It's another rehash of the mess that Pathfinding RC made of trying to sail out of TWYC back when...

Give it, hmmm Server roll on Le'tigre, two weeks of ignoring jiras and support tickets, ditto for the roll on Bluesteel, ditto for the roll on Magnum, add 2 weeks for them to decide the reports to support can't ALL be user error, 4 weeks to develope a fix for the fix, test it on an empty dry land sim on Beta, then roll it RC on LeTigre...

Yeah should be working again in say 3 months...  

...On some sims...

 

[Theresa Tennyson]

12 hours ago, Callum Meriman said:

We know there were changes around region crossing pushed out to the whole grid after just one week in Snack in the Blake Sea. Peopl were complaining about it there too, but the change looks to have been rushed out with only the one weeks test.

 

Nope.

Tested for a week on Snack, then a week on the RC regions, then rolled out to the main channel, just the way it usually works.

[Anna Nova]

3 hours ago, Theresa Tennyson said:

Nope.

Tested for a week on Snack, then a week on the RC regions, then rolled out to the main channel, just the way it usually works.

But not tested very well, clearly.  Still needs to be backed out, at least for water sims like the Blake Sea and Bingo Strait.

[Anna Nova]

3 hours ago, Klytyna said:

His fix requires a custom modded Whinux FS viewer, and custom vehicle scripts that your boat probably doesn't have.

Whinux means Linux?  You don't want to hear what I call the products of a certain anti-trust-convict organisation domiciled in the lovely city of Redmond......

[Klytyna]

38 minutes ago, anna2358 said:

Whinux means Linux?  You don't want to hear what I call the products of a certain anti-trust-convict organisation domiciled in the lovely city of Redmond......

Unix & Linux ... Pro OS's used on some mainfraims and mini computers, and PC based server systems.

The 31 Flavours of Whinux...

"Hi I'm using Venusian Asshat Whinux with the Sino-Peruvian alternate Jupeterian Dingbat Whinux kernel, that I downloaded from that dark-web server in Mongolia, the one that's only open for an our a day on Feb 29th, once every four years, if you have the  site password from winning the Underground Dark Web Latvian suduko challenge. 

I've found that after waiting 3 and a half years to get the build components, and recompiling everything twice, in spanish, and spending 8 months trying to find the documentation for the new kernel, and getting it translated from the original mix of Akkadian and Mayan, that my PC now runs  0.002389% FASTER than it did when I ran Microbloat Windont Hate point Doh!"

Take that Gill 'Master' Bates and the Ebil Microbloat Corp!

PS. I know all the Microbloat users told me there was almost no popular home user software for Whinux before I installed it but... Why isn't there any popular home user software for Whinux? I simply can't believe that rubbish about how we Whinux users are too few for expected sales to justify the cost of hiring Linux coders to port stuff over to Whinux! 

Having to use Windont versions running poorly under Whine just isn't the same as having a PROPER Whinux version..."

*rolls eyes*

 

 

Edited May 17, 2018 by Klytyna

[animats]

10 hours ago, anna2358 said:

I got zapped twice in 10 minutes this morning.  Halsey to Zumwalt at 2.00Am PDT, and then VonSpee to Hipper 2.09.  I then gave up.

In both cases my crossing speed was under 9m/s, and the vehicle itself survived, but I had to relog to regain fly-left-right control.

This sort of thing makes SL sailing, GTFO!, and even just romantic cruising impossible.  So for me, it either gets fixed soon, or I will be off to pastures new.

I just tried that, and it failed in exactly that way. I used a Belliza power boat, 9 knots, and crossed OK from Halsey to Zumwalt. Crossing back from Zumwalt to Halsey, I got a half-unsit. The avatar is exactly one sim from the boat. I still have arrow key control over the boat, and if I turn the boat, the avatar moves, remaining one sim from the boat.

Avatar on map is exactly one sim from viewpoint near boat.

Turn boat 90 degrees, and the avatar is still one sim away, but now to the east.

That's a strange error. The avatar to seat distance has one sim width added to it. 

We know that half-unsits are network timing dependent. One second of network delay at a double region crossing will cause a half-unsit almost every time. The change last week was supposed to change the packet retransmission timing, which is going to affect network delay. That may affect how often this bug appears. Something is way too time-sensitive. This seems to be the same old bug, but with a changed occurrence rate.

The underlying problem is that half-unsits are a stable error - they don't get quickly resolved by the system. I've written about this before. The LL people say this is really hard to fix, but there's now enough attention on it that it's being worked on. Keep up the pressure. Send in JIRAs. Go to the Server User Group meeting. Find repeatable bugs and document them well. Take pictures. Take video.

There are about half a dozen problems with sim crossings, but this is the big one. The one that totally ruins the user experience. It makes sailing not fun. It ruins races. It messes up military combat roleplay that uses vehicles. It makes driving tough. It makes flying fail far too often. It breaks a whole range of things that otherwise work in SL. We need to get this fixed.

(No, I don't work for Linden Labs. I just want the only good virtual world around to work right.)

 

 

[Ayesha Askham]

I hadn't been paying attention...this unholy FUBAR explains the deafening silence about the RC release notes on the Release thread.  It would seem that LL don't want to tell us just what mess they rolled out to the RCs this week at all.  Judging from the above posts, I'm not surprised!

[Theresa Tennyson]

8 hours ago, anna2358 said:

But not tested very well, clearly.  Still needs to be backed out, at least for water sims like the Blake Sea and Bingo Strait.

After fairly extensive testing, I'd say that yes, the crossings from Halsey to Zumwalt and from Zumwalt to Rickover are pretty bad.

All the other crossings I made in that area, in essentially similar conditions with the same server software, aren't bad - with three different power boats by three different scripters at full throttle while wearing my script-pig regular AO instead of the lighter-scripted one I usually use for traveling, and over enough regions that I got lost a couple of times and had to work my way back.

And I've seen worse crossings regularly with the old version you'd want to roll back to.

As far as can tell, you had bad luck hitting a couple of isolated problem areas that could have happened any time since the Blake Sea was created.

1 hour ago, Ayesha Askham said:

I hadn't been paying attention...this unholy FUBAR explains the deafening silence about the RC release notes on the Release thread.  It would seem that LL don't want to tell us just what mess they rolled out to the RCs this week at all.  Judging from the above posts, I'm not surprised!

An interesting statement, seeing that the problem areas aren't running on RC channels.