HTTP error 503: temporary overload or scheduled maintenance

[KT Kingsley]

Over the last couple of weeks an HTTP server script of mine has started occasionally returning error 503:

Quote

 

SERVICE UNAVAILABLE

The server is currently unable to handle the request due to a temporary overload or scheduled maintenance, which will likely be alleviated after some delay.

 

And yes, after a few more attempts it starts responding normally.

Nothing at my end has changed between the time this never happened and the time it started.

When I move the script to a different region this doesn't happen, which solves the problem, but doesn't answer the question, why?

Are simulators or servers limited in the volume of HTTP traffic they can handle? Could there be another script (or scripts) in this region (or another region sharing the server) especially busy handling HTTP traffic?

Any ideas?

Is this a JIRA or a support ticket issue?

[Rolig Loon]

Perhaps relevant, from the LSL wiki:

Requests made at approx 0625 SLT may fail with a 503 status code, with "ERROR: The requested URL could not be retrieved", and "(111) Connection refused" in the body of the response. This has been confirmed as expected behaviour by Kelly, due to the nightly maint & log rotation. It does reliably impact object to object HTTP at that time, and quite probably may impact object to/from web around the same time. The interruption in service is fairly brief, and the precise timing may vary as LL adjust their nightly maint processes, or due to server load.

[KT Kingsley]

Thanks, Rolig.

I don't think this is what's happening here because it happens at any time of day, and usually intermittently and several times during the day.

[Rolig Loon]

OK, just a shot in the dark. Since you didn't say when it happened or how many times you tried at different parts of the day, I figured it was worth mentioning anyway.  You might check out the other caveats there too, if you haven't already. 

[animats]

503 is "Service Unavailable", which is usually worth a delay and a retry. Exponential backoff is useful - try again in 1 second, 2 seconds, 4 seconds, 8 seconds ... up to however long you can afford to wait.

[Oz Linden]

15 hours ago, KT Kingsley said:

Are simulators or servers limited in the volume of HTTP traffic they can handle? Could there be another script (or scripts) in this region (or another region sharing the server) especially busy handling HTTP traffic?

Yes, there are limits.  Depending on circumstances you may get a debug channel notice when a limit is reached, or you may get this error.

Yes, competing with other scripts that are making large numbers of requests can cause this error.

As @animats said, a backoff and retry is the appropriate way to handle this.  There's an example in the wiki of a simple structure for doing these retries (it's written in terms of an initial registration of your script with an external server, but the method used is general and can be applied to any outgoing request).

[KT Kingsley]

Thanks, Oz.

I've fixed the issue by installing the server script in a different region.

Because restarts are a thing, I have a backup server in the other of my two parcels that kicks in when the main server doesn't respond after three failures at ten second intervals and closes down when the main server comes back online. The active server URL is saved as an experience KVP, and the client scripts read that URL and retry after a failure. Because the clients are in the same regions as the servers, their backoffs are enforced by the region's downtime.

[Phate Shepherd]

Could this be exacerbated by the current scripting situation as of the 9/24 server update?

Edited September 27, 2019 by Phate Shepherd