Body of HTTP Response truncated early

[Ana Stubbs]

My scripting skills are very rusty, so i'm hoping someone on the forum can point out why this isn't working.

I've got a very simple hud that reads JSON from someone else's server, and then uses that.

Here is a stripped down version for reproducing my problem 

key http_request_id;
string url = "http://bible-api.com/";
integer listenHandle;

default
{
    
        state_entry()
    {
        listenHandle = llListen(1, "", llGetOwner(), "");
    }
 
    listen(integer channel, string name, key id, string message)
    { 
    http_request_id = llHTTPRequest(url + llEscapeURL(message), [HTTP_BODY_MAXLENGTH,16384], "");
    }

 
    http_response(key request_id, integer status, list metadata, string body)
    {
        
        if (request_id != http_request_id) return;// exit if unknown
        
        llOwnerSay("HTTP status: " + (string)status);
        llOwnerSay("HTTP body: " + body);
        llOwnerSay("Metadata: " + llDumpList2String(metadata,","));
        llOwnerSay("Length of HTTP body: " + (string)llStringLength(body));
    }
}

 

If I request Joshua 1:9 (that is, verse 9 of the first chapter of the book of Joshua), the response is truncated early

Quote

 {"reference":"Joshua 1:9","verses":[{"book_id":"JOS","book_name":"Joshua","chapter":1,"verse":9,"text":"Havenât I commanded you? Be strong and courageous. Donât be afraid. Donât be dismayed, for Yahweh your God is with you wherever you go.â\n"}],"text":"Havenât I commanded you? Be strong and courageous. Donât be afraid. Donât be dismayed, for Yahweh your God is with you wherever you go.â\n","translation_id":"web","translation_name":"World English Bible","translation_n

If I request Joshua 1:8-9 (which is longer than but inclusive of Joshua 1:9) all is fine. Likewise, Joshua 1:9-10 also works.

I've tried other verses with quotation marks in, and other verses with question marks in, and they all seem to work fine. The only other bible verse I've found so far that chokes it is Malachi 1:3, which stops at a different point

Quote

 {"reference":"Malachi 3:2","verses":[{"book_id":"MAL","book_name":"Malachi","chapter":3,"verse":2,"text":"âBut who can endure the day of his coming? And who will stand when he appears? For he is like a refinerâs fire, and like launderersâ soap;\n"}],"text":"âBut who can endure the day of his coming? And who will stand when he appears? For he is like a refinerâs fire, and like launderersâ soap;\n","translation_id":"web","translation_name":"World English Bible","translation_note"

 

HTTP_BODY_MAXLENGTH is set to 16k, which should be enough for the entire chapter!

Can anyone explain what is happening?

[Wulfie Reanimator]

It's actually llOwnerSay that's the issue. Its output is always truncated to 1024 bytes.

Edited July 28, 2019 by Wulfie Reanimator

[Ana Stubbs]

22 minutes ago, Wulfie Reanimator said:

It's actually llOwnerSay that's the issue. Its output is always truncated to 1024 bytes.

That's what I originally thought, but longer strings get through fine.

I also tried using llGetSubString with negative indicies to llOwnerSay only the very last part of the string, and it was truncated in the exact same place. Wherever the issue is, I don't think it's with llOwnerSay.

[Wulfie Reanimator]

53 minutes ago, Ana Stubbs said:

That's what I originally thought, but longer strings get through fine.

I also tried using llGetSubString with negative indicies to llOwnerSay only the very last part of the string, and it was truncated in the exact same place. Wherever the issue is, I don't think it's with llOwnerSay.

You're right. I found some time to look at different outputs more and one thing that really sticks out is the closing curly-quotation used at the end of 1:9. Comparing it to other verses, like 1:11, which also has the closing quote without issues, the difference is that 1:11 also has a matching opening quote. I can't see any other oddities on the string itself.

I didn't find any way to fix it though, or have the time to find other verses that only have the closing quote so I could double-check if it's that.

Edited July 28, 2019 by Wulfie Reanimator

[Ana Stubbs]

35 minutes ago, Wulfie Reanimator said:

You're right. I found some time to look at different outputs more and one thing that really sticks out is the closing curly-quotation used at the end of 1:9. Comparing it to other verses, like 1:11, which also has the closing quote without issues, the difference is that 1:11 also has a matching opening quote. I can't see any other oddities on the string itself.

I didn't find any way to fix it though, or have the time to find other verses that only have the closing quote so I could double-check if it's that.

Thanks, i hadn't spotted that mismatch.

Tested a few more places where people give long speeches that span several verses.

Matthew 1:21

Quote

She shall give birth to a son. You shall name him Jesus, for it is he who shall save his people from their sins.”

works fine.

So does Jeremiah 1:10

Quote

Behold, I have today set you over the nations and over the kingdoms, to uproot and to tear down, to destroy and to overthrow, to build and to plant.”

and, back in the first chapter of Joshua, so does Joshua 1:18

Quote

Whoever rebels against your commandment, and doesn’t listen to your words in all that you command him shall himself be put to death. Only be strong and courageous.”

[Ana Stubbs]

Other verses with overuse of or mismatched special chars that work fine:

Leviticus 10:6

Quote

Moses said to Aaron, and to Eleazar and to Ithamar, his sons, “Don’t let the hair of your heads go loose, and don’t tear your clothes, so that you don’t die, and so that he will not be angry with all the congregation; but let your brothers, the whole house of Israel, bewail the burning which Yahweh has kindled.

Deuteronomy 31:8

Quote

Yahweh himself is who goes before you. He will be with you. He will not fail you nor forsake you. Don’t be afraid. Don’t be discouraged.”

Jeremiah 9:23

Quote

Yahweh says, “Don’t let the wise man glory in his wisdom. Don’t let the mighty man glory in his might. Don’t let the rich man glory in his riches.

Malachi 2:10

Quote

Don’t we all have one father? Hasn’t one God created us? Why do we deal treacherously every man against his brother, profaning the covenant of our fathers?

And, to recap, the ones that don't work are

Joshua 1:9

Quote

Haven’t I commanded you? Be strong and courageous. Don’t be afraid. Don’t be dismayed, for Yahweh your God is with you wherever you go.”

and Malachi 3:2

Quote

“But who can endure the day of his coming? And who will stand when he appears? For he is like a refiner’s fire, and like launderers’ soap; 

 

[Fritigern Gothly]

What caught my eye here, not in the script but in your description of the issue, is that you are retrieving JSON values, but I don't see any JSON-related functions in your script so I was wondering if you are aware of the JSON functions in LSL?

http://wiki.secondlife.com/wiki/Json_usage_in_LSL

[Ana Stubbs]

Just now, Fritigern Gothly said:

What caught my eye here, not in the script but in your description of the issue, is that you are retrieving JSON values, but I don't see any JSON-related functions in your script so I was wondering if you are aware of the JSON functions in LSL?

http://wiki.secondlife.com/wiki/Json_usage_in_LSL

Thanks - I am, and I'm using them, but I chopped them out, along with all the rest of the postresponse processing, for the sake of a clearer script. They don't work at all with invalid JSON, which is what I'm left with when it's truncated early.

[Xiija]

mebbe parse the body?

It seems to be using either ISO-8859-1 or CP-1252 charset  from the API ?

no idea how to fix that except to use a string replace func?

(this is a prim- touch to GET, change for your needs )

string strReplace(string str, string search, string replace) {
    return llDumpList2String(llParseStringKeepNulls((str = "") + str, [search], []), replace);
}
key XMLRequest;
default
{
    state_entry()
    {     
    }
    touch_start(integer total_number)
    {   llOwnerSay("Sending HTTP Request, ...");
        string tail = "joshua1:9"; // "joshua1:9"; // Malachi 3:2
        tail = llEscapeURL(tail);
        string URL = "http://bible-api.com/";
      
        XMLRequest =
         llHTTPRequest( URL + tail , 
         [HTTP_USER_AGENT, "XML-Getter/1.0 (Mozilla Compatible)",
         HTTP_METHOD, "GET", 
         HTTP_MIMETYPE,  "application/json;charset=utf-8", // "text/plain;charset=    CP-1252", ISO-8859-1, utf-8 
         HTTP_BODY_MAXLENGTH,16384,
         HTTP_PRAGMA_NO_CACHE,TRUE], "");    
    }
    http_response(key k,integer status, list meta, string body)
    { 
        if(k ==  XMLRequest)   
        {  
            if(status != 200)
            {  llOwnerSay("\nRrror code : " + (string)status);              
               return;   
            }
            else     
            {  string txt =
               llGetSubString(body, llSubStringIndex(body, "\"text\":"  )+ llStringLength( "\"text\":"), llSubStringIndex(body, "}]," )-5);
               // replace non utf-8 characters...  â , â , â, â
               txt = strReplace(txt, "â", "'"); 
               txt = strReplace(txt, "â", " "); 
               txt = strReplace(txt, "â", " "); 
               llOwnerSay("Verse: \n" + txt);
             }
       }
    }
}

 

Edited July 28, 2019 by Xiija

[Ana Stubbs]

2 hours ago, Xiija said:

mebbe parse the body?

It seems to be using either ISO-8859-1 or CP-1252 charset  from the API ?

no idea how to fix that except to use a string replace func?

(this is a prim- touch to GET, change for your needs )

Firstly, thank you - that's a much better script than mine and shows nicely how i can improve mine. Can I ask about the extra parameters in the HTTP request?

Secondly, unfortunately it's still truncated with your script, it's just pretty and truncated. I see that you've gotten around it by dropping the idea of using the JSON functions, but i'm still holding out hope that there might be a way of stopping it truncating. I've got some plans for further development (eg, stepping through very long passages verse by verse instead of flooding chat) that would be much easier if I can use the json functions instead of a strided list.

[Qie Niangao]

This is an extreme long-shot, but: Is there any way you can see the logs of the server responding to your requests? I ask because I'm not seeing the problem on the LSL side. I know the requests get good responses in a browser, I'm just wondering if there's something the server sees or fails to see in the scripted request that leads it to send something different.

I agree that there's also something funky going on with character sets. I wrote a quick version of this that serves back to itself the text of one of the responses, captured from a browser, and when received from this little LSL http server it uses the proper Unicode characters for backquote, double quotes, etc. as seen in the browser, instead of the way they look when the script gets them from the bible-api server.

[Ana Stubbs]

53 minutes ago, Qie Niangao said:

This is an extreme long-shot, but: Is there any way you can see the logs of the server responding to your requests? I ask because I'm not seeing the problem on the LSL side. I know the requests get good responses in a browser, I'm just wondering if there's something the server sees or fails to see in the scripted request that leads it to send something different.

I agree that there's also something funky going on with character sets. I wrote a quick version of this that serves back to itself the text of one of the responses, captured from a browser, and when received from this little LSL http server it uses the proper Unicode characters for backquote, double quotes, etc. as seen in the browser, instead of the way they look when the script gets them from the bible-api server.

Not my server, so no, but I've opened an issue on github in the hope the creator might be able to shed some light on this.

https://github.com/seven1m/bible_api/issues/30

[Xiija]

I'm not sure what you mean by truncated?.. i put my script in a box, and it gives me the full verse?

 

Quote

[05:27] http get api thing: Verse:
"Haven't I commanded you? Be strong and courageous. Don't be afraid. Don't be dismayed, for Yahweh your God is with you wherever you go.

the params in the request are just basic ones i use for generic requests.

 I'll test the json thing and see how that goes heh

 

 

ETA: tested, i see why the JSON is not working, and the truncation now...

the end of the json should be....  "translation_note": "Public Domain"}

but all that is getting thru is ...... "translation_not  ...( making the JSON invalid)

mebbe the timeout or content length on the API side?

the Joshua one is 578 bytes, ...and the content length set at the api is  is 509 ::shrugs::

 

 

 

 

Edited July 29, 2019 by Xiija

[Xiija]

I did a fake json file to test functionality..  works ok, so it's all on the API dood

 

key XMLRequest;
default
{
    state_entry()
    {     
    }
    touch_start(integer total_number)
    {   llOwnerSay("Sending HTTP Request, ...");       
        string URL2 = "https://my-json-server.typicode.com/Xiija/FakeDB/db";
      
        XMLRequest =
         llHTTPRequest( URL2  , 
         [HTTP_USER_AGENT, "XML-Getter/1.0 (Mozilla Compatible)",
         HTTP_METHOD, "GET", 
         HTTP_MIMETYPE,  "application/json;charset=utf-8", 
         HTTP_BODY_MAXLENGTH,16384,
         HTTP_PRAGMA_NO_CACHE,TRUE], "");    
    }
    http_response(key k,integer status, list meta, string body)
    { 
        if(k ==  XMLRequest)   
        {  
            if(status != 200)
            {  llOwnerSay("\nRrror code : " + (string)status);              
               return;   
            }
            else     
            {  llOwnerSay("returnedJSON: \n" + llJsonGetValue( body, ["comments","0","body"]) );              
            }
        }
    }
}

 the return was...

[07:12] http get api thing 2: Sending HTTP Request, ...
[07:13] http get api thing 2: returnedJSON:
My body is ONE!

 

from my JSON file ... https://github.com/Xiija/FakeDB/blob/master/db.json

[Xiija]

for a temp fix, you could hack the end of the borken JSOn ?

 

key XMLRequest;
default
{
    state_entry()
    {     
    }
    touch_start(integer total_number)
    {   llOwnerSay("Sending HTTP Request, ...");
        string tail = "Malachi 3:2"; // "joshua1:9"; // Malachi 3:2       
        tail = llEscapeURL(tail);
        string URL = "http://bible-api.com/"; 
      
        XMLRequest =
         llHTTPRequest( URL + tail , 
         [HTTP_USER_AGENT, "XML-Getter/1.0 (Mozilla Compatible)",
         HTTP_METHOD, "GET", 
         HTTP_MIMETYPE,  "application/json;charset=utf-8", // "text/plain;charset=    CP-1252", ISO-8859-1, utf-8 
         HTTP_BODY_MAXLENGTH,16384,
         HTTP_PRAGMA_NO_CACHE,TRUE], "");    
    }
    http_response(key k,integer status, list meta, string body)
    { 
        if(k ==  XMLRequest)   
        {  
            if(status != 200)
            {  llOwnerSay("\nRrror code : " + (string)status);              
               return;   
            }
            else     
            {   body = llDeleteSubString(body, -5, -1); // ***hack off end, **
                body = body + "\":'non'}";             // ***and add json ending tag***         
                llOwnerSay("\nVerse: \n" + llJsonGetValue( body, ["text"]) );
            }
        }
    }
}

 

Edited July 29, 2019 by Xiija

[Ana Stubbs]

On 7/29/2019 at 4:13 PM, Xiija said:

for a temp fix, you could hack the end of the borken JSOn ?

I think that may wind up being the only fix.

I wish I knew what is triggering it - at least then I would know how much is likely to be truncated. If it ever hits the end of the verse, then I can't see that I can recover from that, other than maintaining a list of bad verses and substituting in what "should" be returned.

[Ana Stubbs]

The "API dood" has resolved this now for me, I'll leave his explanation of what the issue was in case someone encounters this with another server.

Quote

I made a change to the app so it now returns the headerContent-Type: application/json;charset=utf-8. Prior to that, that charset part was missing. While most clients I've encountered don't seem to care, I suspect that the library you're using was consuming the payload as US-ASCII or some other encoding, which could make for some weird interaction with Content-Length value.

The only question I'm left with is why does SL need this to be specified when other clients don't?