server status 499

[BlueXBeta]

Hi. Today is the day where my scripts can't connect to the offworld database. 

It always works, and I receive the friendly status 200 codes when requesting anything. 

But not today, it just doesn't work anymore. All I receive is status 499. which is an error. The hosting company doesn't find any issues with their connection, or the mysql. And Linden support thinks it is a script malfunction. 

Does anybody have more information or knowledge about what that error code refers to ? 

[Kavarek]

I am having the same issue, Not sure if its dreamhost that I use for the databases or Secondlife. It started last night about 5pm. Funny thing is it happened at the same time LAST FRIDAY.

all my php pages are correct, set correct to the databases can log into the database

but hud and objects do not respond and also get a 5xx erroneous code error.

 

What host do you use?

 

 

 

 

[BlueXBeta]

20 minutes ago, Kavarek said:

I am having the same issue, Not sure if its dreamhost that I use for the databases or Secondlife. It started last night about 5pm. Funny thing is it happened at the same time LAST FRIDAY.

all my php pages are correct, set correct to the databases can log into the database

but hud and objects do not respond and also get a 5xx erroneous code error.

 

What host do you use?

 

 

 

 

I also have dreamhost. They can't find any issue. I can access my db just fine, and my scripts are pretty solid.  It's just that from inside of sl it won't connect properly. I restarted the region but that didn't change anything. 

I'm not sure what to do. 

 

[Kavarek]

Me either ill keep you posted in Secondlife, can try to trouble shoot. IF monday it isnt resolved i will set up a different host and database, test some objects and let you know. THing is a did the dang 3 year pre pay special on the current host. 

[BlueXBeta]

25 minutes ago, Kavarek said:

Me either ill keep you posted in Secondlife, can try to trouble shoot. IF monday it isnt resolved i will set up a different host and database, test some objects and let you know. THing is a did the dang 3 year pre pay special on the current host. 

The thing is, i can access my dreamhost files just fine, the webpage, the database it all works through web interface. Only from within second life I can't access them. So I am leaning strongly to the believe that this is something on Linden's side. A protocol, a security setting something. 

[Kavarek]

So I did trouble shooting. The issue is sl members who use dreamhost.  Sl 8s not sending to them. Hjey have many reports all from sl users using them with  issue happening last Friday at 5pm and yesterday at 5pm. They said data is not reaching people folders with the php files . Database is fine as if I recode an object to goto my otjer host. It routes just fine. Submitted ticket. It would be rediculous to have to recode hundreds of objects in our stayed to fix a issue from sl to them. 

[animats]

I  just tried that with one of my test bikes. The ones that say DEMO or REVIEW log region crossing info to a Dreamhost server. I'm getting an error 499.

Connecting to the logging URL from a browser on the same machine works fine. Server "animats.dramhosters.com" is up. Last successfully logged trip was 2022-11-06 20:15:53. Nothing from today is getting through.

My stuff is in Go, not PHP, so this is not PHP related.

So something is broken SL side.

[Gwyneth Llewelyn]

The 499 error is LL's own 'fake' internal error and (mostly) means that something went wrong on LL's end (see https://wiki.secondlife.com/wiki/Http_response#Status_499). I have encountered this error often when my own scripts hit the simulator's limit in outgoing HTTP requests (a tell-tale sign to throttle down the number of requests), although LL lists that that the 'appropriate' error code would be 420 ('Enhance Your Calm' — a non-standard code number allegedly introduced by Twitter). This doesn't seem to be your case, since your scripts clearly worked before, and you get a 499, not a 420.

Did you try to get the so-called extended errors, using JSON? These might show a little more insight in what is going on.

It's conceivable that LL does use a third-party blacklist database and automatically feeds its own firewalls with IP addresses found on those databases, but a quick search through public databases does not show any of Dreamhost's IP addresses having been placed into a blacklist. Granted, I just use a few of those (most notably AbuseIPDB), so I might simply not have searched on whatever LL might be using. The same applies to wherever LL is hosting simulators these days — if on AWS, it's not impossible that Amazon might be blocking incoming traffic from Dreamhost for some reason. Because Dreamhost is so cheap and popular, it's often used by hackers, spammers and scammers — which will get parts of Dreamhost's network to get blacklisted for a while. The same happens to lots of others, obviously, and that includes the 'big' providers such as Amazon themselves. All this is just speculation, based on:

• All your scripts worked flawlessly before
• Accessing your server-side software via the same API endpoints will show everything is operational; it's just from within SL that nothing works
• Both https://status.secondlifegrid.net/ and https://www.dreamhoststatus.com/ do not show any problem
• Your scripts already check for the internal limits (see https://wiki.secondlife.com/wiki/LlHTTPRequest#Throttles) in some way, so that particular scenario is not likely

Of course, it could also be the case that LL's own addresses are being blacklisted by Dreamhost. This is not so unusual as it might sound: aye, some residents think they're clever and have abused LL's simulator server as a tool for attacks on external servers... this is unlikely, but you can always check for your current simulator's IP address from About > Second Life and check it on popular blacklists to see if it's there. It would also be helpful if you could figure out if this affects any simulator on the grid or just some simulators.

I had a Dreamhost account for well over a decade — and I also did test it out with Go as @animats did, which was way cool — but not anymore, so I cannot make any tests myself...

[Sandry Logan]

Same problem here.

Our hosting hosting is at Dreamhost and as @BlueXBeta and @Kavarek have reported, all our http requests fail with a 499 error when using https, or 502 if using http.

The parts of our system that we access with a web browser, from outside of SL, work as expected. If I make an llHTTPRequest to URLs other than those we use at DH, they also work as expected. Finally I sneaked out onto OSGrid and ran a few test scripts there. Connecting to my scripts on DH works as expected.

ETA: Did anyone open a JIRA for this? I couldn't see one.

 

Edited November 13, 2022 by Sandry Logan

[BlueXBeta]

I had the 499 error before, but only occasionally and it always went away quickly, and by itself. I didn't change anything, it just didn't happen anymore. This time though it is an ongoing blockage. 

Thanks for the wiki link to the 499 error. So it is clear that there is an issue on the SL side in connection with this particular hosting company. So the blacklisting idea sounds possible. I did not know that dreamhost is cheap and used by hackers. Is that actually true, or any different to other hosting companies ? 

here's the Jira I just created: i hope this is useful, it's my first Jira report: 

1. BUG-232915

[Sandry Logan]

From time to time we have had those 499 errors in the past too. As you say, they are usually resolved within a few hours. But not this time. It's about 36 hours as I write.

I don't know about DH being cheap or used by hackers. I have used them for about 15 years. Until today I have had prescious little to complain about with them. But some of the tech support responses I received last night were utter twaddle and I'm more than a little disappointed.

Gwyneth mentioned about trying to establish whether it's any simulator or just a few that are affected. Our system is for virtual dogs and cats, people have them all over the grid and I can see errors coming in from masses of different regions. While I cannot prove it's any/all regions, I'm pretty sure it's grid-wide.

Thanks for opening the jira :c)

[Ron Khondji]

I just tried Sandry's test url from the jira with this script.
 

key HttpRequest (string HtmlMsg) {
    return llHTTPRequest ("https://www.sandry.co.uk/dreamhost_test.txt", [HTTP_METHOD, "POST"], "");
}

default {
    state_entry() {
        llSay(0, "Hello, Avatar!");
    }

    touch_start(integer total_number) {
        HttpRequest ("Hallo");
    }
    http_response (key request_id, integer status, list metadata, string body) {
        if (request_id) {
            llOwnerSay ("Status: " + (string)status);
            llOwnerSay ("Metadata: " + llList2CSV (metadata));
            llOwnerSay ("Body: " + body);
        }
    }
}

This came back:

[08:22] Object: Status: 200
[08:22] Object: Metadata:
[08:22] Object: Body: Hi Dreamhost Tester! .

 

Maybe the problem has resolved itself?

 

(Second Life Server 2022-11-04.576376)

 

[Sandry Logan]

Ron, thank you so much for posting your results.

You're absolutely right that "https://www.sandry.co.uk/dreamhost_test.txt" works and I was completely flummoxed when I pasted your code into my script and it worked.  Sadly I have made a bit of a fubar and I have to put my hands up and say that I chose a particularly bad example URL. 

I had been testing it in-world without the "www" in the address. Before I pasted it into the JIRA, I typed the URL into my browser where it had added "www". When I copied the URL back from the browser address, I didn't notice that the "www" was added.  When I picked this URL as a test, I thought I was choosing something simple with which to illustrate the problem and also trying to avoid exposing any of our possibly more sensitive production scripts. But upon further investigation I see that this particular domain uses Cloudflare and requires the "www" in the URL. The browser puts it in automatically, but I guess that doesn't happen with the LSL request. I think it makes it a bad example for using with LSL. My apologies for wasting your time :c(

That being said, it does serve to illustrate that the problem might lie with Dreamhost. I tried using a different domain of ours, which also uses Cloudflare, "www.turingestates.com". This too returns a status of 200 and displays the html in the output from the request body.

But, when I try a domain hosted at DH which doesn't use Cloudflare, it fails. 

New example URLs: https://www.fireworkgirl.com"  or "https://fireworkgirl.com"  Both should work in the browser, but both fail from LSL.

I better go and edit my jira post. Thanks again for testing and posting.

[animats]

I've been trying various URLs. The 499 error seems to be associated with trying to reach "dreamhosters.com" in HTTPS mode. HTTP mode is OK, and non "dreamhosters.com" sites are OK. Let me look at the certs.

[animats]

Still puzzled. It's something about "dreamhosters.com" TLS, but I can't figure out what.

• www.sandry.co.uk is on a site that supports obsolete TLS 1.1. Works from SL.
• animats.dreamhosters.com is on a site that does not support obsolete TLS 1.1. Fails from SL with a 299 error.

So I thought that maybe the outgoing HTTP servers from SL were still on TLS 1.1. But no. I had an LSL script connect to "www.howsmyssl.com", and email me the results. That got me "Your client is using TLS 1.3, the most modern version of the encryption protocol.". So that's not it.

On the Dreamhost side, they don't support TLS 1.1 any more, but that should not matter.

Test sites here: https://www.site24x7.com/tools/tls-checker.html

Something is incompatible at the Transport Layer Security Level (the machinery behind HTTPS) but I can't figure out what.

 

 

[Sandry Logan]

9 minutes ago, animats said:

www.sandry.co.uk is on a site that supports obsolete TLS 1.1. Works from SL.

This site runs through Cloudflare (https://www.dreamhost.com/partners/cloudflare/) as does www.turingestates.com which I also see in the site24x7 recent tests lists.

Perhaps in its role as a proxy Cloudflare is preventing whatever is tripping up the LSL http requests from SL.

[animats]

Trying sites with https://www.site24x7.com/tools/tls-checker.html

• www.fireworkgirl.com  - TLS 1.1 disabled - report above indicates it does not work from SL.
• www.turingestates.com - TLS 1.1 enabled - report above indicates it works from SL.

But the info I got from tls-checker, querying it from SL LSL, shows it using TLS 1.3, so that shouldn't matter.

I checked the list of supported cyphers. SL supports a long list, too long for TLS 1.3, actually. Most of the old cyphers were booted out. The three you have to have now, TLS_AES_256_GCM_SHA384, TLS_CHACHA20_POLY1305_SHA256, and TLS_AES_128_GCM_SHA256 are all present.

 

[animats]

Chatting with Dreamhost support. They say "the TLS 1.1 version stopped being supported through our Managed Servers as of January 28th, 2018."

[animats]

I'm out of ideas for now. It seems that TLS negotiation is failing, but no idea why. Maybe there are server logs that can help.

Has someone filed a JIRA on this yet?

Edited November 13, 2022 by animats

[BlueXBeta]

32 minutes ago, animats said:

I'm out of ideas for now. It seems that TLS negotiation is failing, but no idea why. Maybe there are server logs that can help.

Has someone filed a JIRA on this yet?

yes i did: it's 

1. BUG-232915

[animats]

39 minutes ago, BlueXBeta said:

yes i did: it's 

1. BUG-232915

OK, I updated that with everything I know.

[Kavarek]

The issue is SL to  dreamhost.

I did trouble shooting saturday when not working again like last friday.  Did tracing from SL to dreamhost. Scripts do not connect to nor reach Dreamhost hosting.

Moved my website files to my secondary cloud server site, but kept access to the same database (making sure other is authorized)

 

 tested on rescripted object for correct url - WORKS

SL and dreamhost need to figure out the issue, and due to the error thrown to me it is a issue on dreamhost due to their sorta slow lag in time. 

[animats]

LL support:

 

Quote

 

Maestro Linden added a comment - 14/Nov/22 10:07 AM - edited

Thanks for the report, BlueXBeta. We are aware of this issue. It indeed appears that Dreamhost is currently blocking LSL HTTP requests from SL to sites that they host. The block seems to be a response to a very popular LSL script sending requests to one Dreamhost-hosted site at an unreasonably high rate. We have alerted the creator of the LSL script about this issue, and they are taking steps to mitigate it.

 

 

Dreamhost support:

Quote

Upon checking the error log for your website, I'm afraid I wasn't able to find any requests being blocked recently, I was also able to test the connection to https://animats.dreamhosters.com/ and wasn't able to find any errors either. If you continue to experience any trouble and there's any additional information you can provide to help us troubleshoot your issue, please let us know and we'll be more than happy to help.

You guys work it out between the two of you.

[BlueXBeta]

Is ANYBODY from SL trying to solve this ?  The Jira ticket is set to solved and it is not solved. 

[Kavarek]

Dreamhost has a slow response rate on average for the past 5 years, a number one complaint. an average of 1750ms is way too high. 

 

Dreamhost stated traffic is not reaching them at all. Those people affected they monitored. 

Moving the pHP files to another host on Clouds storage fixed it, even using the same database on dreamhost, even though it goes to a central server for the database. 

I had to recode so many objects in my system and have the final batch to do. It worked a over a year with no issues. The issue started last Friday at 5pm then repeated this Friday at 5pm but has not resolved. 

I divided the loads with the php requests on a much faster more stable service then dreamhost.