For JSON - To llStringToBase64, or Not To llStringToBase64? That is the Question.

[Love Zhaoying]

It occurred to me, since I'm now using JSON so heavily, that users can "break" the JSON by entering special characters.

Potentially, the user could potentially enter data that would end up with "injection" into the final JSON of additional JSON fields!

1. I could use llStringToBase64() / llBase64ToString()..but do I really need to? 

2. What combinations of characters could users actually enter that would "break" the JSON?  I will do some experiments to see if I can do break JSON with quotes, etc.

One reason NOT to Encode / Decode is of course, because the strings grow in size when encoded.

Example code and output for Encode / Decode:

default
{
    state_entry()
    {

        string sOriginal;
        string sEncoded;
        string sDecoded;
        
        sOriginal = "This is a test with special characters including {, }, [, ], and \".";
        
        sEncoded = llStringToBase64(sOriginal);
        sDecoded = llBase64ToString(sEncoded);

        llSay(DEBUG_CHANNEL,"==========================");
        llSay(DEBUG_CHANNEL,"~Test Base64Code~");
        
        llSay(DEBUG_CHANNEL, "Original legnth=" + (string)llStringLength(sOriginal));
        llSay(DEBUG_CHANNEL, "Encoded legnth=" + (string)llStringLength(sEncoded));
        llSay(DEBUG_CHANNEL, "Original=" + sOriginal);
        llSay(DEBUG_CHANNEL, "Encoded =" + sEncoded);
        llSay(DEBUG_CHANNEL, "Decoded =" + sDecoded);

    }

}

Output:
==========================
~Test Base64Code~
Original legnth=67
Encoded legnth=92
Original=This is a test with special characters including {, }, [, ], and ".
Encoded =VGhpcyBpcyBhIHRlc3Qgd2l0aCBzcGVjaWFsIGNoYXJhY3RlcnMgaW5jbHVkaW5nIHssIH0sIFssIF0sIGFuZCAiLg==
Decoded =This is a test with special characters including {, }, [, ], and ".

 

Edited January 8, 2023 by Love Zhaoying

[Love Zhaoying]

Here is the result of some testing with a modified version of the above code.

I am not sure, if it seems OK. Maybe if the user tries to add extra "\" characters? I will try that too.

TestJson(string sUserInput) {

    string sJsonWithUserInput;
    
    sJsonWithUserInput = llJsonSetValue("", ["UserData"], sUserInput);
    llSay(DEBUG_CHANNEL, "Json with Unencoded user input from listen()=" + sJsonWithUserInput);

}

default
{
    listen(integer channel, string name, key ID, string text) {
     
        llSay(DEBUG_CHANNEL, "listen(), User input= " + text);
        
        TestJson(text);
        
    }
    
    
    state_entry()
    {

        string sOriginal;
        string sEncoded;
        string sDecoded;
        
        sOriginal = "This is a test with special characters including {, }, [, ], and \".";
        
        sEncoded = llStringToBase64(sOriginal);
        sDecoded = llBase64ToString(sEncoded);

        llSay(DEBUG_CHANNEL,"==========================");
        llSay(DEBUG_CHANNEL,"~Test Base64Code~");
        
        llSay(DEBUG_CHANNEL, "Original legnth=" + (string)llStringLength(sOriginal));
        llSay(DEBUG_CHANNEL, "Encoded legnth=" + (string)llStringLength(sEncoded));
        llSay(DEBUG_CHANNEL, "Original=" + sOriginal);
        llSay(DEBUG_CHANNEL, "Encoded =" + sEncoded);
        llSay(DEBUG_CHANNEL, "Decoded =" + sDecoded);

        string sJsonOriginal;
        
        sJsonOriginal = llJsonSetValue("", ["UserData"], sOriginal);
        llSay(DEBUG_CHANNEL, "Json with string constants=" + sJsonOriginal);

        llListen(1, "", llGetOwner(), "");
        
        llSay(DEBUG_CHANNEL, "Ready for user input for additional tests..");

    }

}

Output:

==========================
~Test Base64Code~
Original legnth=67
Encoded legnth=92
Original=This is a test with special characters including {, }, [, ], and ".
Encoded =VGhpcyBpcyBhIHRlc3Qgd2l0aCBzcGVjaWFsIGNoYXJhY3RlcnMgaW5jbHVkaW5nIHssIH0sIFssIF0sIGFuZCAiLg==
Decoded =This is a test with special characters including {, }, [, ], and ".
Json with string constants={"UserData":"This is a test with special characters including {, }, [, ], and \"."}
Ready for user input for additional tests..
listen(), User input= This is a test.
Json with Unencoded user input from listen()={"UserData":"This is a test."}
listen(), User input= This { is } a " test \" with , special characters.
Json with Unencoded user input from listen()={"UserData":"This { is } a \" test \\\" with , special characters."}

 

[Love Zhaoying]

Another test and result:

listen(), User input= This is a test with extra "\" chars: \ \\ \" \\" \\\" \\\\"" \{
Json with Unencoded user input from listen()={"UserData":"This is a test with extra \"\\\" chars: \\ \\\\ \\\" \\\\\" \\\\\\\" \\\\\\\\\"\" \\{"}

It looks like the llJsonSetValue() function automatically escapes special characters..

So does this mean it's not a potential issues, and I don't need to "ever" consider using llStringToBase64() / llBase64ToString()?

[Quistess Alpha]

You'd have to do some research based on specific characters, but llEscapeURL() might be worth looking into to remove special characters.

[Love Zhaoying]

1 minute ago, Quistess Alpha said:

You'd have to do some research based on specific characters, but llEscapeURL() might be worth looking into to remove special characters.

I looked at that first, and it seemed to address different characters that aren't really a problem with JSON - like "spaces", etc.

[Quistess Alpha]

1 hour ago, Love Zhaoying said:

I looked at that first, and it seemed to address different characters that aren't really a problem with JSON - like "spaces", etc.

True, but you'd have to ask yourself whether expanding only "odd" characters by a factor of three is better or worse than expanding every character by a factor of 4/3 (and even especially special characters by more than that). for sake of argument, assuming spaces are the only special character. My back of the envolope says EscapeURL might be more efficient for strings with where 'special characters' are 1/8 or fewer of all the characters, base64 would be better if you have shorter words on average than that.

You could also do a combined approach where you escapeURL the string, then use the parse/dump trick to replace all the escaped spaces with actual spaces.

If you have fewer than 8 'bad characters' and you don't care about accurately recovering them,

str = llDumpList2String(llParseStringKeepNulls(str, ["[","other bad chars"], []), " ");

Edited January 8, 2023 by Quistess Alpha

[Love Zhaoying]

33 minutes ago, Quistess Alpha said:

True, but you'd have to ask yourself whether expanding only "odd" characters by a factor of three is better or worse than expanding every character by a factor of 4/3 (and even especially special characters by more than that). for sake of argument, assuming spaces are the only special character. My back of the envolope says EscapeURL might be more efficient for strings with where 'special characters' are 1/8 or fewer of all the characters, base64 would be better if you have shorter words on average than that.

You could also do a combined approach where you escapeURL the string, then use the parse/dump trick to replace all the escaped spaces with actual spaces.

If you have fewer than 8 'bad characters' and you don't care about accurately recovering them,

str = llDumpList2String(llParseStringKeepNulls(str, ["[","other bad chars"], []), " ");

It doesn't look like I actually have any "bad chars" due to llJsonSetValue() performing its own escaping as-needed of quotes.

I checked llEscapeURL() to see what characters it escapes, since I'm not a Web developer:

Returns a string that is the escaped/encoded version of url, replacing spaces with "%20" etc. The function will escape any character not in [a-zA-Z0-9] to "%xx" where "xx" is the  hexadecimal value of the character in  UTF-8  byte form.

Since this targets all characters not in [a-zA-Z0-9], it is actually escaping a lot more than I need.

So, I should be good for now but will keep your suggestion in mind!

Thanks as always!

Edited January 8, 2023 by Love Zhaoying

[primerib1]

If you store user-input data in your JSON, I think you only need to escape double-quotes " and backslash \

Everything else in a JSON string have no special meaning.

That is according to the JSON specification: https://www.json.org/json-en.html

So you have to insert a backslash in front of them.

Maybe the following code will work:

string EscapeJSONString(string orig) {
    list elems = llParseStringKeepNulls(orig, [], ["\"", "\\"]);
    string result = "";
    string elem;
    integer i;
    for (i=0; i<llGetListLength(elems); ++i) {
        elem = llList2String(elems, i);
        if (("\"" == elem) || ("\\" == elem)) result += "\\";
        result += elem;
    }
    return result;
}

Edited January 10, 2023 by primerib1

[Love Zhaoying]

2 hours ago, primerib1 said:

If you store user-input data in your JSON, I think you only need to escape double-quotes " and backslash \

Those are automatically escaped according to my testing (see earlier posts). Thanks!