Variable Latency/Ping values

[Zalificent Corvinus]

2 hours ago, animats said:

Those huge ping times are from where in the world?

36 hops? Is AWS having a routing problem, or re-routing around some failure?

Just reran the AWS ping tester...

Virginia, average ping 750ms

Ohio, 500ms

LONDON (same damn country, no transatlantic travel needed), 198ms!

Melbourne, surprisingly is a flat line at about 390 ms, none of those wild peaks at all, how the hell can I connect to AWS melbourne at a steady 390, absolute flatline, but London, is a rollercoaster at 200 plus?

AWS looks like it's own internal shuffle is fubar.

 

Edit: the AWS ping tester page is now showing 404...

Edited August 27, 2023 by Zalificent Corvinus

[Gabriele Graves]

Earlier I also took a look at the aws ping tester from here in NZ and saw a similar cyclic effect ranging from 200ms to 600ms.

[AmeliaJ08]

AWS pings very high for me too, seems to be global. Even servers very close to me are showing far higher (3-4x more than Azure) than would be expected.

Azure seems to be fine, expected values given distance in all cases so doubt it's a subsea cable issue.

Edited August 27, 2023 by AmeliaJ08

[Henri Beauchamp]

1 hour ago, AmeliaJ08 said:

AWS pings very high for me too, seems to be global. Even servers very close to me are showing far higher (3-4x more than Azure) than would be expected.

Azure seems to be fine, expected values given distance in all cases so doubt it's a subsea cable issue.

AWS latency also reports too high a ”ping” (300-450ms, US West Oregon, i.e. not even Seattle's SL AWS servers location) for me, here (France) when compared to what I do get from the viewer Statistics (180-200ms), the latter being totally in line (genuine ping + server frame time / 2 + viewer frame time / 2, averaged) with what I get from Azure latency (West US 2: Washington, collocated with SL's AWS servers in Seattle: 160-180ms).

This could totally be due to a broken/congested undersea cable from UK to US (should AWS' latency web site test go through UK, unlike Azure's), when compared to an intact/non-congested France to US cable: the variations you would see would just be due to Internet routers deciding, from time to time, that no, indeed, the shortest but broken route is just too slow, and  instead route your traffic via the longest but intact/non-congested route...

Edited August 27, 2023 by Henri Beauchamp

[Zalificent Corvinus]

29 minutes ago, Henri Beauchamp said:

AWS latency also reports too high a ”ping” (300-450ms, US West Oregon, i.e. not even Seattle's SL AWS servers location) for me, here (France) when compared to what I do get from the viewer Statistics (180-200ms), the latter being totally in line (genuine ping + server frame time / 2 + viewer frame time / 2, averaged) with what I get from Azure latency (West US 2: Washington, collocated with SL's AWS servers in Seattle: 160-180ms).

This could totally be due to a broken/congested undersea cable from UK to US (should AWS' latency web site test go through UK, unlike Azure's), when compared to an intact/non-congested France to US cable: the variations you would see would just be due to Internet routers deciding, from time to time, that no, indeed, the shortest but broken route is just too slow, and  instead route your traffic via the longest but intact/non-congested route...

That doesn't really explain why I got 390ms from England to MELBOURNE Australia, and 200ms to London, if the test site is in the USA, maybe that would explain a 200 ms ping to London from northern England, but 390 to Australia, and 750 to Virginia, Eastern USA? 

Numbers have shifted again...

[Henri Beauchamp]

12 minutes ago, Zalificent Corvinus said:

That doesn't really explain why I got 390ms from England to MELBOURNE Australia, and 200ms to London, if the test site is in the USA, maybe that would explain a 200 ms ping to London from northern England, but 390 to Australia, and 750 to Virginia, Eastern USA?

You cannot predict what one of the many routers on your route from point A to point B in the World, will decide to direct your packets through: maybe one packet will take one of the cables from Melbourne to West US coast, and the next will take the route via South Africa/UK/East US coast...

'traceroute' (UNIXes) or 'tracert' (Windoze), might tell you more about what actual route is taken between your PC and the server, but nothing about what route is taken for those latency testing sites...

Edited August 27, 2023 by Henri Beauchamp

[Zalificent Corvinus]

50 minutes ago, Henri Beauchamp said:

You cannot predict what one of the many routers on your route from point A to point B in the World, will decide to direct your packets through: maybe one packet will take one of the cables from Melbourne to West US coast, and the next will take the route via South Africa/UK/East US coast...

'traceroute' (UNIXes) or 'tracert' (Windoze), might tell you more about what actual route is taken between your PC and the server, but nothing about what route is taken for those latency testing sites...

I ran a tracert, set to show a maximum of 50 hops...

Results were not good.

 

Quote

Tracing route to simhost-0bcc34323c1ea6cea.agni.secondlife.io [54.184.44.5]
over a maximum of 50 hops:

  1     2 ms     2 ms     2 ms  [censored]
  2    36 ms    28 ms    23 ms  [censored]
  3     *        *        *     Request timed out.
  4    39 ms    26 ms    30 ms  [censored]
  5     *        *        *     Request timed out.
  6    37 ms    26 ms    30 ms  109.249.132.36
  7    41 ms    26 ms    45 ms  62.6.204.147
  8    39 ms    30 ms    33 ms  166-49-214-194.gia.bt.net [166.49.214.194]
  9    35 ms     *        *     ix-be-68.ecore1.ldn-london.as6453.net [195.219.83.104]
 10   171 ms   175 ms   175 ms  if-ae-65-2.tcore2.ldn-london.as6453.net [80.231.20.16]
 11   189 ms   178 ms   173 ms  if-ae-32-2.tcore3.nto-newyork.as6453.net [63.243.216.22]
 12   193 ms     *      179 ms  if-ae-22-2.tcore1.nto-newyork.as6453.net [63.243.128.17]
 13   184 ms   173 ms     *     if-ae-17-2.tcore1.sqn-sanjose.as6453.net [63.243.205.186]
 14     *        *        *     Request timed out.
 15   184 ms     *      190 ms  if-ae-0-2.tcore1.sv1-santaclara.as6453.net [63.243.251.1]
 16     *        *        *     Request timed out.
 [snip] All timed out
 50     *        *        *     Request timed out.

Trace complete.
 

The pings are fine crossing the Atlantic, all the way to Santa Clara, then it's into the AWS Cloudcrap shuffle, and it's so lost in there, that it hit 50 effing hops without finishing the trace to the sim server. Last time I ran one of these it only took 36 hops, more than 50 means it's going round and round and round in damn circles somewhere in AWS's network.

Ran a second tracert, with less waiting.

Quote

Tracing route to simhost-0bcc34323c1ea6cea.agni.secondlife.io [54.184.44.5]
over a maximum of 50 hops:

  1     3 ms    12 ms     3 ms  [censored]
  2    29 ms    23 ms    29 ms  [censored]
  3     *        *        *     Request timed out.
  4    43 ms    25 ms   [censored]
  5     *        *        *     Request timed out.
  6    31 ms    45 ms    54 ms  109.249.132.36
  7    36 ms    55 ms    35 ms  62.6.204.147
  8    32 ms    34 ms     *     166-49-214-194.gia.bt.net [166.49.214.194]
  9    34 ms     *        *     ix-be-68.ecore1.ldn-london.as6453.net [195.219.83.104]
 10   192 ms   178 ms     *     if-ae-65-2.tcore2.ldn-london.as6453.net [80.231.20.16]
 11     *      185 ms   182 ms  if-ae-32-2.tcore3.nto-newyork.as6453.net [63.243.216.22]
 12   183 ms     *        *     if-ae-22-2.tcore1.nto-newyork.as6453.net [63.243.128.17]
 13     *      175 ms     *     if-ae-17-2.tcore1.sqn-sanjose.as6453.net [63.243.205.186]
 14   183 ms   172 ms     *     if-ae-18-5.tcore2.sv1-santaclara.as6453.net [63.243.205.37]
 15     *      186 ms     *     if-ae-0-2.tcore1.sv1-santaclara.as6453.net [63.243.251.1]
 16     *      171 ms     *     if-ae-8-3.tcore1.lvw-losangeles.as6453.net [63.243.250.59]
 17     *        *        *     Request timed out.
 [snip] All timed out
 50     *        *        *     Request timed out.

Trace complete.
 

This time it makes it to Los Angeles with no problems, then vanishes into AWS Hell. Third time's the charm.

Quote

Tracing route to simhost-0bcc34323c1ea6cea.agni.secondlife.io [54.184.44.5]
over a maximum of 100 hops:

  1     2 ms     2 ms     2 ms  [censorred]
  2    32 ms    20 ms    33 ms  [censored]
  3     *        *        *     Request timed out.
  4    35 ms    24 ms    28 ms [censsored]
  5     *        *        *     Request timed out.
  6    37 ms    26 ms    33 ms  109.249.132.36
  7    36 ms    25 ms    35 ms  62.6.204.147
  8    33 ms    33 ms    29 ms  166-49-214-194.gia.bt.net [166.49.214.194]
  9    41 ms     *        *     ix-ae-68-0.tcore1.ldn-london.as6453.net [195.219.83.104]
 10   169 ms   175 ms     *     if-ae-65-2.tcore2.ldn-london.as6453.net [80.231.20.16]
 11     *        *      188 ms  if-ae-32-2.tcore3.nto-newyork.as6453.net [63.243.216.22]
 12     *        *        *     Request timed out.
 13     *      186 ms     *     if-ae-17-2.tcore1.sqn-sanjose.as6453.net [63.243.205.186]
 14     *      179 ms     *     if-ae-18-5.tcore2.sv1-santaclara.as6453.net [63.243.205.37]
 15   186 ms   193 ms     *     if-ae-0-2.tcore1.sv1-santaclara.as6453.net [63.243.251.1]
 16     *      173 ms   171 ms  if-ae-8-3.tcore1.lvw-losangeles.as6453.net [63.243.250.59]
 17     *        *        *     Request timed out.
 [Snip] All timed out
100     *        *        *     Request timed out.

Trace complete.
 

Once again, no problems crossing the Atlantic, no problems crossing the US, reached los Angeles ok, then vanishes into AWS Cloudcrap and... still not returning a "reached destination in x ms" after 100 hops.

This is an AWS Cloudcrap problem, not some Atlantic cable problem.

AWS: Awful Worthless Sh*te

 

[Henri Beauchamp]

6 minutes ago, Zalificent Corvinus said:

AWS: Awful Worthless Sh*te

Amazon's Worst Services...

[Ardy Lay]

[Ardy Lay]

Oh, and that "tool" is setting up TCP flows and TLS application connections to perform a "latency test", so take that into consideration.

[AmeliaJ08]

I went region hopping yesterday testing Henri's theory and yep, he's right. Latency was approx half of what awsspeedtest.com reports. 180-240ms for every region I visited, didn't notice any spikes.

Did notice a lot of very sluggish regions though, is slow Sundays still a thing? it feels like it is, very delayed attach/detach of inventory items is the thing I always notice, can be up to 30s delays but you also notice it on teleport, long delay before the the region even seems to respond.

 

 

 

[Merive Vermilion]

Kinda curious on what's been causing it, since wednesday it happened and yet before that there were seemingly no issues going on. I noted that in the mornings it seems less affected ping-wise around 200 but by early afternoon the ping rises until it's around 400. And how will it it take to resolve the issue.

[Aishagain]

@Merive Vermilion I'd guess that's a reflection of the volume of general internet traffic relative to your timezone, which is the same as mine.  It is a pattern I've seen before when either a carrier or transatlantic cable is compromised.

As to how long it will take to fix, it all depends on whether or not a major client is impacted.  Money shouts.  LL is small fry.

Edited August 28, 2023 by Aishagain

[Kristy Aurelia]

For anyone seeing *.gin.ntt.net in their tracert routes, I have received an update from them:

Quote

Unfortunately NTT has multiple outages impacting our network between the US and EU which are causing the increase in latency. Until those are resolved latency and/or packet loss may be encountered.

Keep in mind that if you tracert doesn't go through those severs then you might be experiencing a different issue.

[Aishagain]

OK, that clarifies the "who" let's seen what the "how long" is.  That they acknowledge it suggests that someone big is yelling "FIX IT".

Edited August 28, 2023 by Aishagain

[Monty Linden]

There's a lot in this topic so I'm just going to say a bunch of things in no particular order.

• We're currently blocking ICMP on simhosts so you can't do a true ping to them.  (A subject of internal debate.)  Other tools need to be brought to the task.
• One of the better ones is a true 'traceroute' program.  You will find un-cooperative hosts along the way including simhosts.  However, the numbers it reports are decent and a good version of traceroute is very configurable.  For example, you could target the simulator UDP port or the simhost TCP capability port (12043) for the probe.  How to do this is an exercise for the reader.
• The internet does seem to be generally ill.  I'm on the US east coast and I'm getting 90ms roundtrips to us-west-2 where I usually see 55-60ms.  400ms from the UK is worse than what I'd expect from .au or .za or .aq.
• us-east-1 is the greatest region - but everything is in us-west-2 (Oregon)
• True 'ping' requires ICMP which should not be available to in-browser javascript.  (If it is, I'd like someone to point me at that.)  So the AWS latency test raises some questions.  Its numbers for me are high compared to traceroute and I get the frequent peak others report so I don't trust it except as a very noisy upper limit.
• Application ping (Shift-Ctrl-1) is as Henri describes.  It should look something like actual ping plus frame time.  Unless we're under real load requiring multiple frames to service requests or a retransmission regime.
• The 'Simulator' portion of the 'Statistics' window is most interesting.  If there is a simulator explanation for latency, it can show up there first.  It doesn't cover everything but it does cover most main loop activity.  (Would really like historical metrics charts on the viewer so trends and glitches can be watched a la viewer fps.)
• Services like 'login.agni.lindenlab.com' and the asset CDN are not at AWS.  Don't test against those unless you specifically care about their behavior.
• Simulators share resources on simhosts and perfect isolation isn't possible.  Sometimes the dice don't roll your way.

[Merive Vermilion]

"

1 hour ago, Monty Linden said:

The internet does seem to be generally ill.  I'm on the US east coast and I'm getting 90ms roundtrips to us-west-2 where I usually see 55-60ms.  400ms from the UK is worse than what I'd expect from .au or .za or .aq.

At the moment currently my home region is wavering between 516 and 616ms, I wasn't really doing anything, so I visited the region next door for comparison and it's... ridiculous; 166-183 strong and consistent. Literally right next door is a normal measurement.

 Actually I just tried either side to my region; folling a Linden road and both adjacent regions have a normal ping, so I'm going to keeping until I encounter a bad one.

Edited August 28, 2023 by Merive Vermilion

[Monty Linden]

36 minutes ago, Merive Vermilion said:

At the moment currently my home region is wavering between 516 and 616ms,

Network (not application) ping times like that are the stuff of geosynchronous satellite networking.

[Merive Vermilion]

Today was different for me, I logged on and the ping was normal, back to normal. Everything seemed good, really good, made a few teleports out, visited other places expecting that this was going to be a fluke and that the ping would revert back to 400 but; Nope everything was fine, there I was thinking; almost a week and it's been resolved but then... the rolling restarts happen and what happens when I return... back to 350-400ms.

[Ardy Lay]

Hey!  Guess what?  Some of the major cable damage I was talking about earlier has been repaired.  Now much less of the Internet's traffic is having to take alternate routes and, thus, there is less congestion overall.  Maybe this helps your Second Life experience.

[Merive Vermilion]

Trying not to jinx it, there are some regions behaving weird but not as many, most seem to have reverted to normal, occasional "jumps" in crossing sims come up but for the most things.. seem settled for now at least. Here's hoping things stay .

[Aishagain]

Jinx.  I logged into my home region (Woods of Heaven) today to a nice stable 170ms about an hour after it had restarted (unusually, twice within an hour).  Several other regions that "had" been problematic were also as they used to be.  Now (10 minutes ago) after a further 2-2.5 hours the ping has risen to 220ms nothing changed on the region of on the avatar, I wonder why it happens?  Memory leak or just a rise in internet traffic?

Edited August 29, 2023 by Aishagain
spelling (I kan't spel)