HTTPrequest POST data truncation - any ideas?

[Sasun Steinbeck]

I've been plagued by a very occasional problem for a very long time now (years?) where a few HTTP POST requests from in-world to my web server get truncated. So for example I make a normal HTTPrequest() POST request from a script to my web server with data like param1=bob&param2=fred and it arrives at the server truncated, for example I might get "param1=b" or "param1=bob&pa". These cause an error on the server because required parameters are missing, so they all get logged.

These result in an error 499 returning to the script inworld.

Since I typically have lots of http requests coming in, I'll get these in very small batches, maybe 2-3 over the course of just a few milliseconds, sometimes it might be the same API call and the location of the truncation in the post data appears to be pretty random. I might get two identical API calls a few ms apart that are truncated, but at different positions in the data. It isn't related to a particular API, it happens to all of them, but there are a few that get called far, far more than others so naturally the distribution is weighted toward those heavily called APIs.

These are extremely occasional, maybe I'll get a batch of 1-3 all within a few milliseconds, a week apart, or two weeks or who knows. Considering I'm getting maybe 1 HTTP request per second (some GETs, mostly POSTs) from in world scripts, pretty much 24x7, over the course of a week that's a lot of perfectly fine requests until another little batch of truncated requests comes in.

Has anyone else seen anything like this that's logging bad HTTP calls at the web server level? I just don't see any pattern to these mystery truncations. It's not a huge issue but it's been nagging me for ages and I just wonder what is going on. I'm convinced it's some kind of LL server glitch and they are being sent truncated. I've never seen this happen to any HTTP requests other than from an inworld LSL script. I really don't think it's an issue with my web server (Azure web app), but hey you never know. But I heavily suspect they are being sent truncated.

[Wulfie Reanimator]

Without seeing the logs and/or the in-world script's code, this is just a blind guessing game. You probably know that llHTTPRequest has a HTTP_BODY_MAXLENGTH that determines where the truncation would happen if the request was longer. If you're running just an API server and don't control the scripts you're getting requests from, it's probably just someone's broken script you're seeing.

[Sasun Steinbeck]

Sorry I should have noted that the maxlength is far, far beyond the small number of characters where the truncation is happening. by default of course it's 2k and I'm never setting it to less than 2K, ever. I realize this is really a shot in the dark, I'm just so stuck, I really think it's not my code/server. But never say never, I know that all too well. I just wonder if anyone else has ever seen this happening, is it just me? And I should have noted that ALL the requests are coming from my own scripts so I know exactly what is going on, on both ends. The reqests are nothing special... and umpteen zillion of them work just fine with no errors until it glitches once in a blue moon.

llHTTPRequest(URL, [HTTP_METHOD, "POST", HTTP_MIMETYPE, "application/x-www-form-urlencoded", HTTP_VERBOSE_THROTTLE, FALSE], postdata);

Where URL is my web server where I'm logging the errors.

Just for fun here's the full error report for a typical truncation error event. Note that the expected form value for this particular API should look something sorta like: "source=subscribers2&groupname=blahdeblah&mode=subscriber&owner_key=a30b48f3-ecc2-4004-b5a2-fc1f26ef344x"

But it's chopping it off right at the beginning of "groupname". So it's a hard-coded parameter name... not any kind of variable... and all the rest of the params are just... gone.

Then .07 seconds before this there was a very similar error but the point of truncation was different. The other APi came from a different script in a totally different region, too. So it's not some kind of region glitch. Nor is it a specific LL server glitch since the other POST came from a different server. But it could be some kind of gateway problem or something, I don't know...

_____

Error count: 2 minutes since last error: 0.07685149
Base Exception: System.ArgumentNullException

System.Web.HttpUnhandledException in Boolean HandleError(System.Exception)
Exception of type 'System.Web.HttpUnhandledException' was thrown.

HTTP code: 500

System.ArgumentNullException in Void StringToNumber(System.String, System.Globalization.NumberStyles, NumberBuffer ByRef, System.Globalization.NumberFormatInfo, Boolean)
Parameter name: String

Form:
source=subscribers2&g

POST From: sim10637.agni.lindenlab.com
X-SecondLife-Object-Name: Syndicate Sunday Designer Admin
X-SecondLife-Object-Key: d20e5a72-5dcc-ebdc-70bf-3bbdd003a3b1
X-SecondLife-Owner-Name: Ylva Somebody
X-SecondLife-Owner-Key: x30b48f3-ecc2-4004-b5a2-fc1f26ef3441
X-Secondlife-Shard: Production
http://slurl.com/secondlife/Somewhere/167/22/2996

[Mollymews]

it could be that the value of "bob" in groupname="bob" has characters that need escaping

[Sasun Steinbeck]

That's a great thought. But in this case, that's not happening. Sometimes the truncation happens right in the middle of a hard-coded parameter name rather than something I'm pulling from the database. And those parameter names don't have any special characters in them. For example in the middle of the string I have "param1=customer_name" or something and it truncates right in the middle of "param1". I've gone over and over my LSL code that's making the HTTP call thinking it must be truncating it here, and that's always a valid possibility, but I really don't think so. I probably get a half million HTTP requests per week and only a few get chopped up like this. And again, I might get two in a row, truncated in different positions on what is almost an identical data string, coming from different scripts in different regions so... I really do not suspect a script bug. The only thing that could affect two http calls from two different scripts like that is not the script itself but something in between the sender script and the receiving website where the data is corrupted.

[Mollymews]

it may be that you have already done this, but all I can suggest now is to try and replicate the issue. Something like a test batch of strings that you know have been truncated, and send the batch within the daily time window(s) that the errors have occurred previously

[Wulfie Reanimator]

5 hours ago, Sasun Steinbeck said:

That's a great thought. But in this case, that's not happening. Sometimes the truncation happens right in the middle of a hard-coded parameter name rather than something I'm pulling from the database. And those parameter names don't have any special characters in them. For example in the middle of the string I have "param1=customer_name" or something and it truncates right in the middle of "param1". I've gone over and over my LSL code that's making the HTTP call thinking it must be truncating it here, and that's always a valid possibility, but I really don't think so. I probably get a half million HTTP requests per week and only a few get chopped up like this. And again, I might get two in a row, truncated in different positions on what is almost an identical data string, coming from different scripts in different regions so... I really do not suspect a script bug. The only thing that could affect two http calls from two different scripts like that is not the script itself but something in between the sender script and the receiving website where the data is corrupted.

You may want to create a JIRA or talk to a Linden about the issue, maybe they'll be able to look at their logs as well. (Though I don't know how useful that would be if you can't replicate this.)

[Sasun Steinbeck]

27 minutes ago, Wulfie Reanimator said:

You may want to create a JIRA or talk to a Linden about the issue, maybe they'll be able to look at their logs as well. (Though I don't know how useful that would be if you can't replicate this.)

Yeah it's so rare that bombarding my server with umpteen million of the same POSTs for a few months seems like a long shot. I definitely can't even imagine how to reproduce them on demand. I can't see a pattern - in frequency, or the position of truncation, or the originating script name, or region.. but I just realized I've been lax on saving these records for very long so I'll see if there's any kind of date/time pattern, thanks for the suggestion. Don't have enough data to rule that out...