llHTTPRequest changes

[Oz Linden]

In the latest main channel simulator version (2017-07-11T22:13:46.327548), we deployed some updates that changed the llHttpRequest call. These were made necessary by updates to some of the underlying HTTP libraries we use, and are intended to make HTTP use from LSL easier to debug and more robust, but in a few cases they have broken some existing scripts. This note is intended to summarize the changes and provide guidance on how to update your scripts.

URLs

The most important change, and the one that seems to be responsible for most of the problems that have been reported, is in the handling of the URL parameter.

In previous versions, the URL was not checked, and some scripts had done things that shouldn't have worked (and now they don't):

• Control characters in the URL.  Specifically, this has usually been newline characters. There were widely used hacks that inserted newlines in order to get around restrictions on the use of some headers, and those hacks also truncated other important headers that the simulator inserts in all requests.  This will now produce a run time error without sending the request. The fix is to remove the newlines and any additional headers they were inserting; if it was the User-Agent header, there is a new parameter you can use to provide a value for that header.
• Spaces in the URL. Space characters are not allowed in URLs, but because many scripts insert them, we have put in special handling to convert them to a %20 in most cases.

If your script is passing values returned from other LSL calls that may return spaces, you should do the replacement before putting them into the URL, like:

string URL = "http://example.com/path?region=" + llEscapeURL(llGetRegionName())

depending on how your server works, you may need to substitute a plus sign ('+') for the spaces in parameter values rather than the '%20'.

Header Value Changes

We added a new HTTP_USER_AGENT parameter that lets you append to the User-Agent header value; in some cases, servers look for key words in this header.

We made requests shorter by sending one long Accept header with all the allowed MIME types rather than many headers with one each. This made requests shorter and more compatible with more servers. As far as we know it has not caused any problems.

The default User-Agent header server token value was changed from 'Second Life LSL' to 'Second-Life-LSL'; this appears to have caused problems for some servers that were checking for it. 

Problems

If you are having problems with these changes, you may reply on this thread or file a JIRA and we'll make an effort to help you. Please include the part of the script that is making the call to llHTTPRequest and the values that are being passed to it, and describe any errors you are getting on the Debug chat channel or from your web server.

Edited July 24, 2017 by Oz Linden
add info on User-Agent token value.

[Fox Reinsch]

I have 30,000 plus virtual radios that I have given out for free that have stopped working because of this.

How do I fix 30,000 radios if I can't remotely update the scripts in them.

It took me a few hours to find the cause here in this thread. Info and script as follows:

This music URL hover text script has worked for almost 10 years in the over 30,000 free radios I have given out for IndieSpectrum Radio.
It now returns a script error of: URL passed to llHTTPRequest contains a control character.

It does this on any Shoutcast stream URL that I have tried, not just my own which is http://indiespectrum.com:9000
It does this on three different sims
And it does it with both Firestorm and the stock LL viewer

I can't see how all the shoutcast servers changed how they send the now playing string.

Any help would be greatly appreciated as I run my station at a loss just to promote live music in SL.

Thank you in advance, Fox Reinsch

==================================
Script (with a different Internet station URL from mine):
==================================

// leave music_url blank if set in object's description
string music_url = "http://78.46.91.38:8000"; 

key HTTPRequest;
string feed;
string URL;
string currSongTitle;
string lastSongTitle;
list feedList;

default
{
    state_entry()
    {
        llSetText(" ",<1,1,1>,1);
        llSetTimerEvent(5.0);
    
        if(music_url) 
        {
            URL = music_url;
        } else {
            URL = llGetObjectDesc();
        }
    }

    timer()
    {
        HTTPRequest=llHTTPRequest(URL + "/7.html HTTP/1.0\nUser-Agent: LSL Script (Mozilla Compatible)\n\n",[],"");
    }

    http_response(key k,integer status, list meta, string body)
    {
        if(k != HTTPRequest) return;

        feed = llGetSubString(body,llSubStringIndex(body, "<body>") + llStringLength("<body>"), llSubStringIndex(body,"</body>") - 1);
        feedList = llParseString2List(feed,[","],[]);
        currSongTitle = llList2String(feedList,6);
        integer length = llGetListLength(feedList);
        
        if(llList2String(feedList,7))
        {
            integer a = 7;
            for(; a<length; ++a)
            {
                currSongTitle += ", " + llList2String(feedList,a);
            }
        }
         
        if (currSongTitle != lastSongTitle)
        {
            llSetText ("Now playing on IndieSpectrum Radio \n"+currSongTitle,<1,1,1>,0.75);
            lastSongTitle = currSongTitle;
        }
    }
}
 

[arton Rotaru]

To fix the script change it to this:

 timer()
    {
        HTTPRequest=llHTTPRequest(URL + "/7.html", [HTTP_USER_AGENT, "LSL_Script (Mozilla Compatible)"],"");
    }

Can't help with fixing 30.000 radios though, sorry.

[GoSpeed Racer]

It's not just Fox's Indie Spectrum station that is affected, but my station too, KONA Stream. In the over 10 years our station has been in SL we too have sent out thousands of radio tuners with floating text. Sadly in our case we bought the script and we cannot modify it. Let's not forget the hundreds of products out there that are now broken. Some of the creators are no longer in SL. This feels like a kick in the teeth to the media producers and consumers in SL.

[Fox Reinsch]

GoSpeed you are welcome to use the script I posted above, the fix arton posted does work. I do wonder why this server change was made.  Terrible that everything in SL that showed what was playing on a stream no longer works.

 

[Fox Reinsch]

Oz you should read this.

arton thanks for the fix, it works, but what a hassle, I have to change a dozen different radios, tell the 25K that listen each month about the problem.  I have to pdate the radios I give away on the marketplace. I can't get rid of the thousands of radios that were copy transfer that residents have, they will never go away.It will take me hours to do this!

 The many people that had old radios will never find out what is wrong and will just trash them and stop listening to SL musicians.  The only reason I run the station is to promote live music in SL and to make SL a better place.

Fox Reinsch

[Fox Reinsch]

The fix arton provided, thanks arton, worked for the radios that have now playing hover text but it did not for the ones that have alphanumeric displays.

Maybe someone can help with the alphanumeric root script, http call section below:

default
{
    state_entry()
    {
        llMessageLinked(LINK_SET,204000,"","3");
        HTTPReq = llHTTPRequest(URL + "/7.html HTTP/1.0\nUser-Agent: XML Getter (Mozilla Compatible)\n\n",[],"");
        llSetTimerEvent(Rate);
    }
    timer()
    {
        HTTPReq = llHTTPRequest(URL + "/7.html HTTP/1.0\nUser-Agent: XML Getter (Mozilla Compatible)\n\n",[],"");
    }
    http_response(key request_id, integer status, list metadata, string body)
    {
        if(request_id != HTTPReq)return;
        //
        CmdLin  = llCSV2List(body);
        //
        //        Arg(0)        =        Current Listeners
        //        Arg(1)        =        Connected?
        //        Arg(2)        =        Peak Listeners
        //        Arg(3)        =        Max Listeners
        //        Arg(4)        =        Reported Listeners
        //        Arg(5)        =        kbps (Steam Rate)
        //        Arg(6)        =        Song Playing
        //
        llMessageLinked(LINK_SET,204000,llGetSubString(Arg(6),0,-15),"3");
    }
}

[Oz Linden]

1 hour ago, Fox Reinsch said:

Oz you should read this.

arton thanks for the fix, it works, but what a hassle, I have to change a dozen different radios, tell the 25K that listen each month about the problem.  I have to pdate the radios I give away on the marketplace. I can't get rid of the thousands of radios that were copy transfer that residents have, they will never go away.It will take me hours to do this!

 The many people that had old radios will never find out what is wrong and will just trash them and stop listening to SL musicians.  The only reason I run the station is to promote live music in SL and to make SL a better place.

Fox Reinsch

I think you underestimate the ability of SL Residents to adapt.

For some history of why we made this change, see my posts in https://community.secondlife.com/forums/topic/406941-dj-board-display-not-working-properly/

I regret that it wasn't reasonable to make an automatic adjustment in the simulator to correct this, but sometimes that's the way it works out. We try to keep disruptions like this to a minimum, and I'm making a special effort on this one to reach out to the authors of scripts that are affected.

Frankly my biggest regret is that the URL hack was not noticed long ago and a better solution (like the current one) provided then (well before my time, but that's an imperfect excuse) - it should never have worked and it's surprising to me that it ever did. Software surprises you that way sometimes.

We have a couple of other improvements to llHTTPRequest in the pipeline (ones that won't break anything); watch this thread for announcements when they're available for testing.

 

[Cookie Bertone]

Sorry to post this but I took a long break from SL and my skills with LSL seem to have vanished.

My own script had the following command with the nPlayingURL being the stream URL.

llHTTPRequest(nPlayingURL+"/7.html HTTP/1.1\nUser-Agent:Mozilla\n\n",[],"");

I thought that the following would work as it was very close to Fox's code, but it doesn't.  If somebody could point me in the right direction I would be super grateful.

llHTTPRequest(nPlayingURL + "/7.html", [HTTP_USER_AGENT, "BAS_PMP (Mozilla Compatible)"],"");

 

[Oz Linden]

1 minute ago, Cookie Bertone said:

Sorry to post this but I took a long break from SL and my skills with LSL seem to have vanished.

My own script had the following command with the nPlayingURL being the stream URL.

llHTTPRequest(nPlayingURL+"/7.html HTTP/1.1\nUser-Agent:Mozilla\n\n",[],"");

I thought that the following would work as it was very close to Fox's code, but it doesn't.  If somebody could point me in the right direction I would be super grateful.

llHTTPRequest(nPlayingURL + "/7.html", [HTTP_USER_AGENT, "BAS_PMP (Mozilla Compatible)"],"");

 

That looks right ... what is the value of nPlayingURL ? Do you get an error message?

 

[Berksey]

People in SL do tend to bounce back rather quickly, I've noticed. And as big of a hassle as all of this is right now, I know there were good reasons for it.

Considering the fact that radios and electronic equipment in RL tend to last about two minutes these days, having anything in as ephemeral a place as the grid last ten years should probably be venerated as the miracle that it is, and the change that has broken the old hack could, if one thinks about it, be seen as an opportunity to put out some new stuff to sell... I know, I know, making inworld currency is the devil to a lot of people, and admittedly I won't buy anything I can make myself, even if I can't make it as nicely as some other people can, but maybe it's a good thing, you know?

If someone bought a radio from me ten years ago and complained today that it doesn't work anymore, and I didn't have a nice, new radio to sell them, yeah, I'd be a bit upset. But I went through worse hassles just learning how to throw bunny dolls at people without missing. And if they bought something ten years ago and feel like it means they should have it forever, well... not to go all Buddhist about it or anything, but...

"Everything is temporary. On the Internet, exponentially so."

"This too shall pass?"

Just my two $L.

[arton Rotaru]

11 hours ago, Fox Reinsch said:

The fix arton provided, thanks arton, worked for the radios that have now playing hover text but it did not for the ones that have alphanumeric displays.

Maybe someone can help with the alphanumeric root script, http call section below:

default
{
    state_entry()
    {
        llMessageLinked(LINK_SET,204000,"","3");
        HTTPReq = llHTTPRequest(URL + "/7.html HTTP/1.0\nUser-Agent: XML Getter (Mozilla Compatible)\n\n",[],"");
        llSetTimerEvent(Rate);
    }
    timer()
    {
        HTTPReq = llHTTPRequest(URL + "/7.html HTTP/1.0\nUser-Agent: XML Getter (Mozilla Compatible)\n\n",[],"");
    }
    http_response(key request_id, integer status, list metadata, string body)
    {
        if(request_id != HTTPReq)return;
        //
        CmdLin  = llCSV2List(body);
        //
        //        Arg(0)        =        Current Listeners
        //        Arg(1)        =        Connected?
        //        Arg(2)        =        Peak Listeners
        //        Arg(3)        =        Max Listeners
        //        Arg(4)        =        Reported Listeners
        //        Arg(5)        =        kbps (Steam Rate)
        //        Arg(6)        =        Song Playing
        //
        llMessageLinked(LINK_SET,204000,llGetSubString(Arg(6),0,-15),"3");
    }
}

 

 

It seems some servers don't like the lettering "Script" to be included in the user agents name. Try to change LSL_Script to something different, like:
 

HTTPReq = llHTTPRequest(URL + "/7.html", [HTTP_USER_AGENT, "XML-Getter/1.0 (Mozilla Compatible)"],"");

 

[Cookie Bertone]

15 hours ago, Oz Linden said:

That looks right ... what is the value of nPlayingURL ? Do you get an error message?

 

Oops, silly me for not posting that information.

But, this morning I go back to check and change the script and it is working fine.  Weird.  So yes it did look right because it was.  I cannot explain it as I even restarted my viewer and replaced the item to make sure it was reset last night.

So, thank you all for your help.  :-)

[Fox Reinsch]

Thanks for all the help, I have all my free IndieSpectrum Radio radios working again as a new Version 6.  Now I have to purge the old ones off the marketplace and get up the new ones.

[Pryda Parx]

Since last Tuesday (the last sim restart) all of my vendors have been offline (i.e. no fairy wings have been selling) and this thread seems to indicate the most likely explanation. But I checked the scripts and don't see any spaces or control characters in the request url. I also checked with my hosting servers and nothing there seems to account for the problem either.

The request is:

PendingRequestID = llHTTPRequest(WebPageURL+"/database/DB"+WebPagePart+".php", [HTTP_METHOD, "POST",HTTP_MIMETYPE, "application/x-www-form-urlencoded", HTTP_BODY_MAXLENGTH, 8000],"OwnerUUID="+(string)llGetOwner()+"&OwnerName="+llEscapeURL(ppGetName(llGetOwner()))+"&UUID="+(string)llGetKey()+"&Name="+llEscapeURL(llGetObjectName())+"&Description="+llEscapeURL(llGetObjectDesc())+"&Region="+llEscapeURL(llGetRegionName())+Parameters);

The "Parameters" bit is also fully llEscapeURL'ed before passing in.

Where should I look for the problem?

Also, not sure if this has changed or not, the PHP page on the web server validates the quest with a line:

if (strpos($headers["User-Agent"],"Second Life")===0)

Edited July 23, 2017 by Pryda Parx

[Pryda Parx]

I believe I've found the problem and I can fix my vendor system by changing the php pages on the server. No need for a full in-world vendor script change is a relief.

You've changed the http headers and a

if (strpos($headers["User-Agent"],"Second Life")===0)

if no longer valid it needs to be:

if (strpos($headers["User-Agent"],"Second-Life")===0)

Oz, I'm disappointed! Mucking about with http headers without warning can have a big impact on web applications, I would have liked backward compatibility for a transition phase at least. This has cost me 5 days of lost sales and quite a while investigating the problem too. Meow!!!

[Whirly Fizzle]

1 hour ago, Pryda Parx said:

I believe I've found the problem and I can fix my vendor system by changing the php pages on the server. No need for a full in-world vendor script change is a relief.

You've changed the http headers and a

if (strpos($headers["User-Agent"],"Second Life")===0)

if no longer valid it needs to be:

if (strpos($headers["User-Agent"],"Second-Life")===0)

Oz, I'm disappointed! Mucking about with http headers without warning can have a big impact on web applications, I would have liked backward compatibility for a transition phase at least. This has cost me 5 days of lost sales and quite a while investigating the problem too. Meow!!!

Possibly the same problem as BUG-100830 - HTTP_CUSTOM_HEADER no longer works on RC 17.06.13.327111

Before the update, regions sent

"User-Agent": "Second Life LSL\/17.05.26.326655 (http:\/\/secondlife.com)",

After the update, regions sent

"User-Agent": "Second-Life-LSL\/17.06.13.327111 (https:\/\/secondlife.com)",

 

[Pryda Parx]

Yes Whirly, that describes the same problem. I hope they do not "fix" it now or all my vendors will break again. Now I am validating the user-agent as "Second-Life" and last week I was validating "Second Life". I really don't want to have to change it again!

[Slan Sabre]

This change took down the whole DCS weapon control system. The encoding of the item parameter name as passed by your scripts to an url call changed. So non of the weapons could  be found in our database and were disabled. These sort of sim/lsl changes can really wreak havoc on existing applications. Thank for the heads up (not). We managed to make some server side adjustments so we did not have to update  all the weapons.  If some one wants to know how, contact me inworld.

Again Oz, many thanks for the extra work.

Slan Sabre

 

[Oz Linden]

On 7/20/2017 at 9:32 AM, Oz Linden said:

The default User-Agent header server token value was changed from 'Second Life LSL' to 'Second-Life-LSL'; this appears to have caused problems for some servers that were checking for it. 

I added the note above to the original post. 

Sorry that caused problems for a few services; I made the change because when we added the ability to append your own User-Agent values, we included checks on valid syntax and since our own value would not have passed that check, I thought it only fair to correct it. Perhaps that wasn't the best tradeoff, but now that it's done we won't change it again.

[NeoBokrug Elytis]

Even a little bit of warning about the headers change would have gone a long way to ease a ton of frustration for everyone involved.

I normally adapt rather well to LLs sudden LSL changes that break things simply because I am usually available for it.  And while it's not LLs problem personally, I actually had surgery a three days before this recent change.  Which means I had to find out about the change after the fact, drag myself to the computer when I should have been resting (because everything is broken), and make some quick fixes.

Warnings about intentional and significant LSL changes such as this, SHOULD HAVE BEEN MADE CLEAR in advance, so we could at least have the opportunity to plan to make changes.  To me communication seems like an ongoing issue, whether or not the changes were intentional.

 

Edited July 24, 2017 by NeoBokrug Elytis

[Lady Sumoku]

It would be nice if there was some mention of the "fixes" to the old workarounds on the wiki, instead of just deleting any mention of the workarounds entirely. As I can't even edit my own wiki description anymore, it will have to be someone else.  Or maybe documentation is just more optional than it used to be.

Also, the keys I'm getting in http_response don't match the key returned by the llHTTPRequest at all.  Makes it difficult to catch responses when they come back with totally different IDs. (This is returning 499 codes, but I don't know why the keys wouldn't match)

Edited July 27, 2017 by Lady Sumoku
Additional Info

[Necati Millet]

Greetings,

After yesterdays maintenance on some specific regions we started to get 499 status code and as it was mentioned above the request key and response keys don't match. Everything was fine before yesterday for the past 2 years, same scripts, same php APIs/codes. If we restart the region everything starts working again as it should. Keys match and 499 status goes away. Everything turns back to normal. However, after a while (like 30 minutes, sometimes earlier sometimes a bit later than that) it stops working again and returns 499 codes with the keys not matching. Needing to restart 4 regions we have every 30 minutes or so is getting pretty annoying both for us and for thousands of players in our role playing system.

[Oz Linden]

22 minutes ago, Necati Millet said:

Greetings,

After yesterdays maintenance on some specific regions we started to get 499 status code and as it was mentioned above the request key and response keys don't match. Everything was fine before yesterday for the past 2 years, same scripts, same php APIs/codes. If we restart the region everything starts working again as it should. Keys match and 499 status goes away. Everything turns back to normal. However, after a while (like 30 minutes, sometimes earlier sometimes a bit later than that) it stops working again and returns 499 codes with the keys not matching. Needing to restart 4 regions we have every 30 minutes or so is getting pretty annoying both for us and for thousands of players in our role playing system.

Please file a Jira on this one.  Be sure to include where it is happening and an object that demonstrates the problem.

We certainly didn't do anything like that on purpose, so it will take some work to figure it out but we need more specific information to start.

[Necati Millet]

23 minutes ago, Oz Linden said:

Please file a Jira on this one.  Be sure to include where it is happening and an object that demonstrates the problem.

We certainly didn't do anything like that on purpose, so it will take some work to figure it out but we need more specific information to start.

https://jira.secondlife.com/browse/BUG-134072

Thank you for your response. I have filed a jira. I hope it can get fixed soon and easily.