Getting JSON_INVALID possibly due to truncated response

[Remi Alpha]

I am loading JSON data from my API (which is private) but I have made some publicly available and  redacted sample files here:

[Edit: I removed the URLs from the post because a. the issue is solved and b. medium-term these test files will be deleted so they will just return 404 errors]

The format is:

{"status":true,"agents":[//array of objects]}

...where status would be false if the server returned any non-code 200 response.

On load I check for validity by not only checking the status code but also checking the JSON value of status using llJsonGetValue(response, ["status"]);

It was working fine until I had about 6 or so agents in the list then it started failing but with a code 200 so I looked into it and found that llJsonGetValue was determining the data to be invalid.

The JSON itself is valid (checked with https://jsonlint.com/) but the longer example is giving me: "Status 200" and then "JSON is invalid". I am seeing the metadata list as: [0, 2048]

 

Is there a maximum length of 2048 on the http response that can be handled?

This is only a very small amount of JSON data and would be extremely prohibitive.

 

//string test_uri = "REDACTED";
string test_uri = "REDACTED";
key req_id;

api_get(string url){
    req_id = llHTTPRequest(url, [HTTP_METHOD, "GET", HTTP_MIMETYPE, "application/json"], "");
}

integer checkHTTPResponse(integer status, string response){
  if(status != 200){
    return FALSE;
  }
  llOwnerSay("Status 200");
  string json_status = llJsonGetValue(response, ["status"]); 
  if(json_status == JSON_TRUE){
    return TRUE;
  } else if(json_status == JSON_INVALID){
    llOwnerSay("JSON is invalid");
  } else if(json_status == JSON_FALSE){
    llOwnerSay("JSON status is FALSE");
  }
  return FALSE; 
}


default 
{
  state_entry()
  {
    api_get(test_uri, auth_list); 
  }

  http_response(key request_id, integer status, list metadata, string response)
  {
    llOwnerSay("metadata: "+llDumpList2String(metadata, ","));
    // output: 'metadata: 0,2048'      
    if(checkHTTPResponse(status, response)){
      // continue as ok
	} else {
      // handle error
    }
  }
}

 

 

 

Edited April 14, 2020 by lostware
fixed shorter test url

[Nova Convair]

Add    HTTP_BODY_MAXLENGTH, 16384    to the parameter list of llHTTPRequest to get the max of 16384 instead of 2048 default length of the reponse body.

[Remi Alpha]

1 hour ago, Nova Convair said:

Add    HTTP_BODY_MAXLENGTH, 16384    to the parameter list of llHTTPRequest

Ah... Earlier Google threw up this document which  says "(Setting HTTP_BODY_MAXLENGTH is not yet supported)"

So I had discounted that. 

But it does indeed work so obviously that document is out of date. Thank you!

I guess I never came across an http client that capped the response body size before. Though it makes sense given that trying to load a very large document could adversely affect the sim server running the script. Given we have such a limit I will work to further compress the data, and build in pagination too.

Edited April 14, 2020 by lostware

[Remi Alpha]

My payload is 2.8KB but only 728 Bytes over the network with gzip (and just 408b with Brotli... but I am assuming that is not supported Edit: confirmed as not supported)

As a side question - can anyone clarify if the header Accept-Encoding is sent by default (As: deflate, gzip) ?

Headers are listed in a table under llHTTPRequest but it’s not entirely clear if they are ALL sent.

Edit Having now tested, setting this request header:

HTTP_CUSTOM_HEADER, "Accept-Encoding", "deflate, gzip" 

...does not remove the issue (without HTTP_BODY_MAXLENGTH set, which does solve it) so this demonstrates that regardless of whether gzip encoding was already accepted by default, HTTP_BODY_MAXLENGTH is the uncompressed size and not the size of the network transfer, which is a shame.

Edited April 14, 2020 by lostware