Connectivity issues?

[mygoditsfullofstars]

Assuming is it a specific network carrier that's at fault, traceroutes (if enough of them are supplied), might help in identifying which one it might be (ie: comparing a bunch of traceroutes provided by people having problems against users in similar geographic locations who are not having problems).

Whether anything can be done about a problematic intermediate network carrier is another matter entirely, depends on where along the path they are, if its something within Amazon or one of their upstreams its likely Amazon can get it resolved. If its a carrier beyond that boundary then all I can say is "good luck" since it would require sending an email to the NOC of that carrier (who most of the time don't respond to emails (based on my past experiences when in such a situation)).

Its also possible this problem will just go away on its own if its a network carrier who is aware of the problem and is currently working to resolve it.

Anyway I might be completely barking up the wrong tree with it being a specific intermediate network carrier being the issue, but is a plausible theory.

[Monty Linden]

1 hour ago, Reuben Blakewell said:

What are the logs they need and where to access I can upload mine.

The SecondLife.log or SecondLife.old file associated with a bad session.  It can be attached when filing a support ticket.

[Whirly Fizzle]

I'm seeing a lot of people affected by this in the SL Discord chats I'm in & Firestorm tickets, Affected users seem to be from all over the world.

Fwiw, I'm having no issues yet myself - location Edinburgh, UK.  ISP Talk-Talk.
Win 11, No VPN or Proxy.
Windows Firewall & only using Windows defender for Antivirus.

[Qie Niangao]

This may mean nothing, but a few hours ago Qie lost connection on the Linden viewer and took a very long time to reconnect. It was taking so long I launched Firestorm to login an alt, and that actually timed out. Meanwhile Qie connected and the alt on Firestorm choose the "teleport home" option to continue, which promptly connected (to the same region that had taken so long for Qie before) and neither session had even a hint of further trouble. So this was a brief (maybe ten minute) network glitch somewhere that may or may not be related to the much more serious problems others are seeing. (I'm on Bell Canada FWIW.)

[Monty Linden]

We're coming up on the holiday break and Lindens are going to start disappearing (myself included).  If you are comfortable posting viewer logs to Jira, file a BUG jira, attach the logs from a bad session, then mention the Jira number here so it can be found quickly.

[MBeatrix]

The issue is intermittent now. I could login fine about an hour ago at my home parcel and teleport around for a while... until it's back.

[EDIT] I was logged out and now I get stuck waiting for region handshake, no matter where I try to login.

[EDIT 2] Yes, it comes and goes away. I'm back in and could teleport to a store.

Edited December 15, 2023 by MBeatrix
providing more info

[Axel Naxos]

24 minutes ago, MBeatrix said:

O problema é intermitente agora. Eu poderia fazer login perfeitamente há cerca de uma hora em minha casa e me teletransportar por um tempo... até que ele voltasse.

[EDITAR] Eu estava desconectado e agora fico esperando pelo handshake da região, não importa onde eu tente fazer login.

[EDIT 2] Sim, ele vem e vai embora. Estou de volta e posso me teletransportar para uma loja.

yes, intermittent problem for me too 😕

[Brilliant Price]

As of 12.30am SLT I'm in with FS 6.6.17 and was able to TP out and back home without problems. The region handshake stuff grumbled a bit at first but at least it didn't hang or crash. I've got too much to do today so I won't be spending all day juggling and swapping viewers, alts, machines etc. I'll just stay firmly logged, in my eyrie, catching up on yesterdays missed tasks and fielding customers requests. Thanks LL for what you tweaked (if anything)

[MBeatrix]

Changing DNS service from Google to Cloudflare. We'll see how it goes.

[Axel Naxos]

At the moment all my 75 alts are fine 🙂 I hope the problem doesn't return

[Wired Georgia]

Hi, at least today i had no issues logging in or TPing around.

So whatever happened its being fixed now.

[xXSuicidalIdolXx Candycane]

On 12/15/2023 at 3:43 AM, MBeatrix said:

Changing DNS service from Google to Cloudflare. We'll see how it goes.

can you follow up on how cloudflare worked?? 

[MBeatrix]

8 hours ago, xXJupiterHeightsXx Starchild said:

can you follow up on how cloudflare worked?? 

It's all good. Google DNS is working too.
May have been some network in the way that was slow or blocking traffic. As the Lindens don't say what it was, all we can do is guessing.

[EDIT - a lot of editing, as not being a techie it takes me quite sometime to have an idea about it all and I'd hate to be unfair to any of those involved]
Here's some info gathered from a traceroute to a403.b.akamai.net I was asked to perform on that horrible day compared to how it is right now (have a look at that particular slow hop):

then >
  4     6 ms     6 ms     5 ms  lis1-cr1-be10-200.cprm.net [195.8.30.237] 
  5    27 ms    41 ms    41 ms  akamai1.cprm.net [195.8.10.110] 
  6     6 ms     6 ms     5 ms  a95-100-100-10.deploy.static.akamaitechnologies.com [95.100.100.10] 

now (while staying in-world) >
  4     6 ms     6 ms     6 ms  lis1-cr1-be10-200.cprm.net [195.8.30.237]
  5     8 ms     6 ms    13 ms  akamai1.cprm.net [195.8.10.110]
  6     6 ms     5 ms     5 ms  a95-100-100-17.deploy.static.akamaitechnologies.com [95.100.100.17]

akamai.net is a cloud services provider. cprm.net is my ISP. Hmmm... It looks like my ISP was having trouble accessing the cloud. But what about the other ISP's? (France, Germany, UK and Brazil, at least)

But why an SL unscheduled login maintenance that seems to have solved the problem? Was it the login system routing some connections (ISP's) though a troubled network? Yeah, I'm totally clueless.

(yes, only now I had the time to look at it more closely)

Edited December 18, 2023 by MBeatrix
adding info

[Ardy Lay]

Some network operators, access and transit, discard UDP traffic in times of congestion, perhaps thinking that only BitTorrent still uses UDP.

[MBeatrix]

1 hour ago, Ardy Lay said:

Some network operators, access and transit, discard UDP traffic in times of congestion, perhaps thinking that only BitTorrent still uses UDP.

Right.
Logging in was fine, the problem was connecting to the cloud, where SL really is, as you know. Now I'm wondering if all the other people who experienced the issues use the service of ISP's belonging to the Altice group (my ISP is a company that is owned by that group).

[Beq Janus]

Today, I have had repeated failures. Sometimes 2 or 3 attempts are enough; other times it might take 5 or more. Keep banging on the door, and it will eventually open. 

@Monty Linden these are not DNS lookup failures but timeouts in the curl request. 

2023-12-19T00:38:42Z WARNING #CoreHttp# llcorehttp/_httppolicy.cpp(403) LLCore::HttpPolicy::stageAfterCompletion : HTTP request 000001EEF4197E40 failed after 0 retries.  Reason:  Timeout was reached (Easy_28)
2023-12-19T00:38:42Z WARNING # newview/llxmlrpctransaction.cpp(266) LLXMLRPCTransaction::Handler::onCompleted : LLXMLRPCTransaction error 0000001c: Timeout was reached
2023-12-19T00:38:42Z WARNING # newview/llxmlrpctransaction.cpp(268) LLXMLRPCTransaction::Handler::onCompleted : LLXMLRPCTransaction request URI: https://login.agni.lindenlab.com/cgi-bin/login.cgi
2023-12-19T00:38:42Z INFO #LLXMLRPCListener# newview/llxmlrpclistener.cpp(344) Poller::poll : login_to_simulator result from https://login.agni.lindenlab.com/cgi-bin/login.cgi: status CURLError, errorcode OPERATION_TIMEDOUT (Despite our best efforts, something unexpected has gone wrong.

Could we have an unresponsive login instance in a load balancer or somesuch?

This has been dragging on for a while suggesting it is something more than transient network issues. Notably, I am seeing no issues with any other connectivity outside of SL. 

A DNS lookup resolves as follows:

Name: a403.b.akamai.net
Addresses: 62.253.3.195
          62.253.3.240
Aliases: login.agni.lindenlab.com
          login.agni.lindenlab.com.edgesuite.net

 

[Monty Linden]

2 hours ago, Beq Janus said:

Could we have an unresponsive login instance in a load balancer or somesuch?

Are the timeouts consistently to the login endpoint host (login.agni.lindenlab.com)?  Worst-case metrics look good there so it's either stuck in front of the service or some *really* unhealthy instances.

[Beq Janus]

7 hours ago, Monty Linden said:

Are the timeouts consistently to the login endpoint host (login.agni.lindenlab.com)?  Worst-case metrics look good there so it's either stuck in front of the service or some *really* unhealthy instances.

For me, it is always on the login. I am not necessarily the best canary for this mine, though, as my online time is frequently spent in one place while I code. 

I just spent a while hopping about places. A mixture of busy venues, script-heavy shopping regions and numerous texture and asset-heavy scenic places, no issues at all. 

[Monty Linden]

44 minutes ago, Beq Janus said:

I am not necessarily the best canary for this mine, though, as my online time is frequently spent in one place while I code.

Understood.  Happy to have any insight into a possible pattern.  So thanks for the report, there is something to dig into there.

[xXSuicidalIdolXx Candycane]

so when I asked about cloud flare above I wanted to know what it was doing because sometimes I bounce between google's DNS and Cloud Flare's. seems like most viewers are effected but what I personally seem to notice is when I'm on idle at my property or at a store, last night I was on from 6:30 PM to 1:30 AM SLT and had no problem, before making this comment about 10 minutes ago I was logged out with the black and white screen, I logged back in came to the forum and while idle it same thing, for me it seems to happen more when on a flat idle. I am using firestorm, tried the second life viewer and same thing like others have reported. right now I'm back on google's DNS, Cleared Cache, restarted hotspot, computer and tweaked some settings in firestorm and nothing. hope in some way I helped, hang in there everyone! ❤️

[xXSuicidalIdolXx Candycane]

I noticed nobody followed up on this post but I'm still having connectivity issues like what we discussed before if I'm just sitting on idle or if I'm in the middle of something I get a greyed out screen saying that there's some kind of connectivity issue I don't know what to do I know it's not my internet..