New kind of region crossing failure

[animats]

Have not seen this before.

Region crossing fail at Burns->Neumogen.

This is a failure I haven't seen before. I was driving around Robin Loop fast, and the second time I approached Neumogen from Burns, it wasn't visible. I drove into the crossing expecting Neumogen to appear. Instead, my motorcycle disappeared, and I was stuck like this.

What's unusual is that this was more or less permanent. Teleporting out didn't work; about two minutes of waiting and then back here. Logout would work cleanly. While stuck, could not IM. Attempts to group chat returned errors. Could not read profiles. But I could look around in mouselook.

I've had this happen twice now at this crossing. Seeing this once wouldn't be that much of a problem, but twice indicates something is wrong.

 

 

[Ardy Lay]

You have reached the end of the Internet.

Or you are on the 13th Floor.

[animats]

Uh oh. This fix may have introduced a new bug:

2020-08-24T21:00:45.547626

Thursday, August 27, 2020

Release Notes

• Internal fixes (to help Lindens make better tests)
• Logging improvements
• SL-13785 improves some instances where vehicles would stop responding to control inputs after a region crossing. We don’t believe we’ve fixed all instances of this issue yet, but this patch should offer some relief for the issue.

Resolved Issues

• SL-13785 Vehicle LSL scripts do not see any passengers shortly after a region crossing

[animats]

Seeing lots of strange stuff. Approaching a region fast, the region ahead draws, then disappears. Then I end up sitting on the edge of the region as above.

Slow region crossings used to be OK with my bikes. They'd stop and wait for the avatar to catch up. Now that's stopped working reliably. When the region crossing fails, it used to be possible to get off, go to the bike, and get back on. Now I have to TP home. And after failed crossings, I keep ending up at <0,0,1> in the region.

[animats]

Filed JIRA: https://jira.secondlife.com/browse/BUG-229312

[animats]

(duplicate)

Edited September 4, 2020 by animats

[Profaitchikenz Haiku]

6 hours ago, animats said:

What's unusual is that this was more or less permanent. Teleporting out didn't work; about two minutes of waiting and then back here. Logout would work cleanly. While stuck, could not IM. Attempts to group chat returned errors. Could not read profiles. But I could look around in mouselook.

I started seeing something like this yesterday when TP-ing. TP failed, an attempt to try again got a message saying TP was currently disabled/not working, it wasn't possible to move around, attempts to IM failed, ultimately I crashed out. It happened about 8 or 9 times, on both Firestorm and Catznip, both normally very stable viewers for TP-ing.

[Rachel1206]

Drove around same roads at Burns, Neumogen, no problems - and this in an ACS based vehicle, which has more impact.

Meet #animats and drove with him as passenger, no problems encountered. Smooth flawless region crossings, no passenger hanging or eject, vehicle passed crossings perfect.

I would say the above problems, are due to a region not appearing online or a region in trouble. Hence it is always good to drive a vehicle with mini maps open, and simply stop and return if a region appear offline for what ever reason.

 

 

[animats]

7 hours ago, Rachel1206 said:

Drove around same roads at Burns, Neumogen, no problems - and this in an ACS based vehicle, which has more impact.

Meet #animats and drove with him as passenger, no problems encountered. Smooth flawless region crossings, no passenger hanging or eject, vehicle passed crossings perfect.

I would say the above problems, are due to a region not appearing online or a region in trouble. Hence it is always good to drive a vehicle with mini maps open, and simply stop and return if a region appear offline for what ever reason.

I'm thinking this depends on SL load. No fails after 10:30PM or so SLT last night. Two this morning.

Fail at Vine. Nowhere near a region corner.

This is happening far too often.

Edited September 4, 2020 by animats

[animats]

Just had this fail with a Manji Automotive Works car. So it's not my bikes that are the problem.

[Mollymews]

18 hours ago, Rachel1206 said:

I would say the above problems, are due to a region not appearing online or a region in trouble. Hence it is always good to drive a vehicle with mini maps open, and simply stop and return if a region appear offline for what ever reason.

yes I think so as well. Sometimes the region isn't there for our connection, when it is for other people. So we just have to wait for it

is not as frequent this as it used to be. And yes agree. I watch the minimap. When the minmap fully updates to show the next region then is usually ok to cross.. When the minimap still shows the next region as dark blue then it isn't yet fully there at least for our connection. And we we will pretty much lock up our viewer every time we cross into dark blue

[Aethelwine]

I can see last comment on the Jira seems to be relating to problem a lot of sailors have been having for the last week with unsits on region crossings (reported on the thread for the server deploy)

Frustrating SL-13785 is hidden.
 

Edited September 8, 2020 by Aethelwine

[animats]

1 hour ago, Aethelwine said:

Frustrating SL-13785 is hidden.

Yes. Once a problem is accepted, it goes into some kind of internal JIRA queue where users can see neither the contents nor the status.

Closing a user JIRA as a duplicate of an SL JIRA is kind of bad, though. The user JIRA should remain live, but dependent on the internal one. Users may have more info to add.

This was discussed at Server User Group today. The server dev team is totally focused on getting SL onto AWS by the end of the year. The two big server side problems right now seem to be region crossings and group chat, both of which have been modified in preparation for the move to AWS and both of which seem to have been badly impacted by recent mods.

Simon Linden has way too much to do. LL is trying to hire more people to work on the server side of SL.

[arabellajones]

The weekend Grid Drive had a lot of problems. Even walking, no vehicle, every region crossing took at least six seconds.

It was a timed run, and I logged 88 grid crossings, So a minimum of 8m 48s, and the leader board being used displayed time from under 5 minutes to 12 minutes. So, obviously, this all varies a lot. I was seeing maybe one third of my total time down to the region crossings.

I connect from the UK, with an internet ping of about 150ms. The draw distance may also have some effect on total load on my end of the connection. I was using a vehicle I was pretty familiar with.

I can't avoid a feeling that the Lindens have never been able to cope well with the idea that customer connections can vary so much. The cry of "It's your connection!" almost seems like their Get-Out-Of-Jail-Free card. And both AWS and the Content Delivery Network risk making sensitive assumptions about the communications.

 

[Aishagain]

@arabellajones: You highlight an issue that pervades the whole operation of SL.  They, like so many other Americans in the US, seem incapable of understanding that there is a WHOLE world out here.  It is akin to the idiotic attitude of some Brits who consider that "****(perjorative word redacted) begin at Dover".

If your ping to SL is greater than 125ms, you must have a problem (or live overseas) and therefore do not fit our functional model of the World. 

[animats]

Once on AWS, Linden Lab can put sims in various Amazon data centers around the world, if they want.  Estate sims should be allowed to pick their AWS region, near where most of their users are.

All of SL doesn't have to be in the same data center. Right now, the beta grid is partly in AWS-West and partly in the old Arizona data center. Sometimes for adjacent sims you can cross between. That's working reasonably well.

[Rachel1206]

Here is some raw ping times from Scandinavia:

Second Life server
Pinging sim10786.agni.lindenlab.com [216.82.56.76] with 32 bytes of data:
Reply from 216.82.56.76: bytes=32 time=148ms TTL=50
Reply from 216.82.56.76: bytes=32 time=148ms TTL=50
Reply from 216.82.56.76: bytes=32 time=149ms TTL=50
Reply from 216.82.56.76: bytes=32 time=148ms TTL=50

Ping statistics for 216.82.56.76:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 148ms, Maximum = 149ms, Average = 148ms

AWS
Pinging aws.com [13.33.152.114] with 32 bytes of data:
Reply from 13.33.152.114: bytes=32 time=12ms TTL=241
Reply from 13.33.152.114: bytes=32 time=13ms TTL=241
Reply from 13.33.152.114: bytes=32 time=13ms TTL=241
Reply from 13.33.152.114: bytes=32 time=11ms TTL=241

Ping statistics for 13.33.152.114:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 11ms, Maximum = 13ms, Average = 12ms

Google.com
Pinging google.com [172.217.21.238] with 32 bytes of data:
Reply from 172.217.21.238: bytes=32 time=19ms TTL=115
Reply from 172.217.21.238: bytes=32 time=19ms TTL=115
Reply from 172.217.21.238: bytes=32 time=22ms TTL=115
Reply from 172.217.21.238: bytes=32 time=20ms TTL=115

Ping statistics for 172.217.21.238:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 19ms, Maximum = 22ms, Average = 20ms

-

Clearly AWS and Google are routed to geographic nearby data servers or hub mirroring. Ping time to my local IP is 5-6 ms.

 

Edited September 15, 2020 by Rachel1206

[arabellajones]

On 9/15/2020 at 7:42 AM, arabellajones said:

The weekend Grid Drive had a lot of problems. Even walking, no vehicle, every region crossing took at least six seconds.

 

I did a similar test today, when the number of logins was around the minimum. It's a poor measure, but it's one of the few bits of hard data we can get.

The region crossing delays were less than one second.

All the measures of my connection were similar.

[arabellajones]

On 9/15/2020 at 7:42 AM, arabellajones said:

And both AWS and the Content Delivery Network risk making sensitive assumptions about the communications.

What I am thinking of here is the relative location of the asset servers, the sim servers, the CDN, and the user. Past models have has the asset servers close to the sim servers, and the CDN is caching certain types of content being sent to the user. It hasn't much mattered whether the asset servers and the sim servers have been the source, they're in essentially the same place.

But let's assume a university wants to run a set of sims. They could run OpenSim and have the whole operation under their control. But they choose to take advantage of the Linden Lab support structure, and the huge amount of conter other users have created.. Let's call AWS-target the optimum AWS location for the sim servers. There might be a CDN node at the university itself. What happens with the asset servers, and all the others. Where does texture baking get done?

And that's before we even think about different legal systems. Look at all the fuss we have with US financial laws. It's simple enough when the servers are all in Arizona, but when the servers could be anywhere, what then?

[Ardy Lay]

1) Are the prim object and mesh assets passed between simulators?

If yes, is what is passed the entire asset or just the digested representation used by the simulator?

If no, is what is passed like an inventory item?  (a pointer to the asset needed, and the simulator must request, receive and digest the asset to get the physics information it needs)

2). Are scripts in task inventory saved as an asset to the asset store any time during a typical region crossing or are they passed from simulator to simulator directly?

We know that texture, animation, sound, script source and some additional assets are not used by simulators so assume those have no effect on region crossings.

[bigmoe Whitfield]

I remember when this was very common, sim to sim hand off would not complete and you would basically get stuck in limbo until it timed you out.

[animats]

More problems with delays before the next sim appears. This time I have video and Firestorm logs, and will post once I edit the video down. It's strange. The sequence of event is:

• Sim ahead is drawn, but only with the ground texture. No objects.
• A few seconds later, sim ahead disappears, leaving empty space.
• 5-30 seconds later, sim ahead reappears, this time with all objects. Then it's safe to enter the sim.

Light load, I was the only avatar around most of the time. Never saw this before the next week or two.

[animats]

More info. Here's what this bug looks like, from recorded video.

Approaching a sim crossing, next sim is visible, but only the terrain is drawn.

Then, next sim is gone. Driving over the edge will leave the avatar stuck and a logout will be necessary.

But wait a few seconds (sometimes as long as 20) and the next sim reappears and is fully drawn with its objects and terrain.

I've seen this six or seven times now. It's usually at the same places. Here I'm driving round Robin Loop. This is the Burns - Neumogen crossing, and I've seen it there at least three times now. Anyone else seeing this?

[Rachel1206]

2 hours ago, animats said:

But wait a few seconds (sometimes as long as 20) and the next sim reappears and is fully drawn with its objects and terrain.

I've seen this six or seven times now. It's usually at the same places. Here I'm driving round Robin Loop. This is the Burns - Neumogen crossing, and I've seen it there at least three times now. Anyone else seeing this?

I just did a drive of the Robin Loop (2020-09-25 22:45 PM SLT) and did not encounter any of the problems above.

Last time we both were there, I had the problems as you just had - I would not call it a bug, but some latency internet problems.

BTW - proof I was there, got a ticket passing The Nightbell Fire Station😂:

RS Wasp v1.25,
Owned by  Rachel1206 Resident
53.072479 KpH
Fine L$23

Edited September 26, 2020 by Rachel1206

[animats]

Go around the loop a few times without stopping. This seems to happen on repeat visits. I suspect it may be that the region didn't fully disconnect yet from the previous visit, and that may be why you see the region shown as bare terrain first.