Jump to content

List vs Json Array Manipulation Metrics

LepreKhaun

By LepreKhaun,
in LSL Scripting

 Share
You are about to reply to a thread that has been inactive for 3925 days.

Please take a moment to consider if this thread is worth bumping.

Recommended Posts

LepreKhaun

LepreKhaun

Posted
Posted

So, you may have heard that the Json functions were a wee bit on the slow side. Here's an idea of how slow they are, up to 3,800 times slower than a comparable LSL list manipulation...

		Retrieving String Value (256x)...
	JSON_ARRAY - "[1,2]"		LSL List - ["1","2"]
Length	Memory	Avg. time		Memory	Avg. time
   2:	60604	 0.442257		60544	 0.004657
   4:	60596	 0.552125		60544	 0.004675
   8:	60580	 0.803219		60528	 0.004614
  16:	60548	 1.349062		60496	 0.004705
  32:	60484	 2.146690		60432	 0.004588
  64:	60356	 3.440264		59792	 0.004553
 128:	59588	 5.721018		59024	 0.004531
 256:	58564	 6.860139		56976	 0.004681
 512:	56516	 6.865057		53392	 0.004871  
1024:	52420	 9.123466		46224	 0.004740  
2048:	44228	12.545660		31888	 0.004796  
4096:	27844	18.331900		 3216	 0.004824  
8192:	(on compile) 			(0, 20004) : ERROR : Syntax Error
Script run-time error 
Stack-Heap Collision

You might notice that the time increases with the number of values in the json array but stays constant for the LSL list retrieval. The same applies for json object "Key":Value pairs as well, though I leave it up to the reader to write 1k unique Key strings with their "leaning toothpicks". :smileylol: 

 

		Appending a Value (256x)...
	JSON_ARRAY - "[1,2]"		LSL List - ["1","2"]
Length	Memory	Avg. time		Memory	Avg. time
   2:	60604	 0.899813		60544	 0.023149
   4:	60596	 1.155598		60544	 0.022460
   8:	60580	 1.427069		60528	 0.023890
  16:	60548	 1.989700		60496	 0.026853
  32:	60484	 2.941872		60432	 0.030174
  64:	60356	 3.757151		59792	 0.035227
 128:	59588	 6.103055		59024	 0.059189
 256:	58564	 6.867533		56976	 0.099973
 512:	56516	 6.982137		53392	 0.195541
1024:	52420	 9.216980		46224	 0.404540
2048:	44228	12.668380		31888	 1.059620
4096:	27844	Stack-Heap Error 	 3216	Script run-time error
			Stack-Heap Collision (on touch)

 

Scripts used to determine metrics for those that wish to see for themselves. These were run on a moderately busy mainland region btw so your mileage may vary a bit, though the overall patterns will hold...

 

//			== Retrieving String Value... ==
//				Json Array
string jText = "[1,2]"; // "[1,2,1,2]" then "[1,2,1,2,1,2,1,2]"

default
{
    state_entry(){
        llOwnerSay((string)llGetFreeMemory());
    }

    touch_start(integer total_number)
    {
       string s;
       float t = 0;
       float tot = 0;
       integer i;
       integer j = 20;
       while(j--){
           i = 256;
           llResetTime();
            while (i--) {s = llJsonGetValue(jText, [0]);}
            t = llGetTime();
            llOwnerSay((string)t);
            tot += t;
        }
        llOwnerSay("Average time: " + (string)(tot/20));
    }
}

+++++++++++++++++++++++++++++++++++++++++++

//				LSL list
list List = ["1","2"]; // ["1","2","1","2"] then ["1","2","1","2","1","2","1","2"]

default
{
	state_entry(){
	llOwnerSay((string)llGetFreeMemory());
	}

    touch_start(integer total_number)
    {
	   string s;
       float t = 0;
       float tot = 0;
       integer i;
       integer j = 20;
       while(j--){
           i = 256*256; // *256 required to avoid 0 time!
           llResetTime();
            while (i--) s = llList2String(List, 0);
            t = llGetTime();
            llOwnerSay((string)(t/256));
            tot += t;
        }
        llOwnerSay("Average time: " + (string)(tot/5120)); // 256*20 = 5120
    }
}

 

// 			== Appending a Value... ==
//				Json Array
string jText = "[1,2]"; // "[1,2,1,2]" then "[1,2,1,2,1,2,1,2]"

default
{
    state_entry(){
        llOwnerSay((string)llGetFreeMemory());
    }

    touch_start(integer total_number)
    {
       string s;
       float t = 0;
       float tot = 0;
       integer i;
       integer j = 20;
       while(j--){
           i = 256;
           llResetTime();
            while (i--) {s = llJsonSetValue(jText, [JSON_APPEND], "3");}
            t = llGetTime();
            llOwnerSay((string)t);
            tot += t;
        }
        llOwnerSay("Average time: " + (string)(tot/20));
    }
}

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

//				LSL list
list List = ["1","2"]; // ["1","2","1","2"] then ["1","2","1","2","1","2","1","2"]

default
{
	state_entry(){
	llOwnerSay((string)llGetFreeMemory());
	}

    touch_start(integer total_number)
    {
	   list s;
       float t = 0;
       float tot = 0;
       integer i;
       integer j = 20;
       while(j--){
           i = 256*4; // *4 required to avoid 0 time!
           llResetTime();
            while (i--) s = List + "3";
            t = llGetTime();
            llOwnerSay((string)(t/4));
            tot += t;
        }
        llOwnerSay("Average time: " + (string)(tot/80)); // 4*20 = 80
    }
}

 

  • Like 1
Link to comment
Share on other sites
LepreKhaun

LepreKhaun

Posted
  • Author
Posted

So, you may have heard that the Json functions were a wee bit on the slow side. Here's an idea of how slow they are, up to 3,800 times slower than a comparable LSL list manipulation...

		Retrieving String Value (256x)...
	JSON_ARRAY - "[1,2]"		LSL List - ["1","2"]
Length	Memory	Avg. time		Memory	Avg. time
   2:	60604	 0.442257		60544	 0.004657
   4:	60596	 0.552125		60544	 0.004675
   8:	60580	 0.803219		60528	 0.004614
  16:	60548	 1.349062		60496	 0.004705
  32:	60484	 2.146690		60432	 0.004588
  64:	60356	 3.440264		59792	 0.004553
 128:	59588	 5.721018		59024	 0.004531
 256:	58564	 6.860139		56976	 0.004681
 512:	56516	 6.865057		53392	 0.004871  
1024:	52420	 9.123466		46224	 0.004740  
2048:	44228	12.545660		31888	 0.004796  
4096:	27844	18.331900		 3216	 0.004824  
8192:	(on compile) 			(0, 20004) : ERROR : Syntax Error
Script run-time error 
Stack-Heap Collision

You might notice that the time increases with the number of values in the json array but stays constant for the LSL list retrieval. The same applies for json object "Key":Value pairs as well, though I leave it up to the reader to write 1k unique Key strings with their "leaning toothpicks". :smileylol: 

 

		Appending a Value (256x)...
	JSON_ARRAY - "[1,2]"		LSL List - ["1","2"]
Length	Memory	Avg. time		Memory	Avg. time
   2:	60604	 0.899813		60544	 0.023149
   4:	60596	 1.155598		60544	 0.022460
   8:	60580	 1.427069		60528	 0.023890
  16:	60548	 1.989700		60496	 0.026853
  32:	60484	 2.941872		60432	 0.030174
  64:	60356	 3.757151		59792	 0.035227
 128:	59588	 6.103055		59024	 0.059189
 256:	58564	 6.867533		56976	 0.099973
 512:	56516	 6.982137		53392	 0.195541
1024:	52420	 9.216980		46224	 0.404540
2048:	44228	12.668380		31888	 1.059620
4096:	27844	Stack-Heap Error 	 3216	Script run-time error
			Stack-Heap Collision (on touch)

 

Scripts used to determine metrics for those that wish to see for themselves. These were run on a moderately busy mainland region btw so your mileage may vary a bit, though the overall patterns will hold...

 

//			== Retrieving String Value... ==
//				Json Array
string jText = "[1,2]"; // "[1,2,1,2]" then "[1,2,1,2,1,2,1,2]"

default
{
    state_entry(){
        llOwnerSay((string)llGetFreeMemory());
    }

    touch_start(integer total_number)
    {
       string s;
       float t = 0;
       float tot = 0;
       integer i;
       integer j = 20;
       while(j--){
           i = 256;
           llResetTime();
            while (i--) {s = llJsonGetValue(jText, [0]);}
            t = llGetTime();
            llOwnerSay((string)t);
            tot += t;
        }
        llOwnerSay("Average time: " + (string)(tot/20));
    }
}

+++++++++++++++++++++++++++++++++++++++++++

//				LSL list
list List = ["1","2"]; // ["1","2","1","2"] then ["1","2","1","2","1","2","1","2"]

default
{
	state_entry(){
	llOwnerSay((string)llGetFreeMemory());
	}

    touch_start(integer total_number)
    {
	   string s;
       float t = 0;
       float tot = 0;
       integer i;
       integer j = 20;
       while(j--){
           i = 256*256; // *256 required to avoid 0 time!
           llResetTime();
            while (i--) s = llList2String(List, 0);
            t = llGetTime();
            llOwnerSay((string)(t/256));
            tot += t;
        }
        llOwnerSay("Average time: " + (string)(tot/5120)); // 256*20 = 5120
    }
}

 

// 			== Appending a Value... ==
//				Json Array
string jText = "[1,2]"; // "[1,2,1,2]" then "[1,2,1,2,1,2,1,2]"

default
{
    state_entry(){
        llOwnerSay((string)llGetFreeMemory());
    }

    touch_start(integer total_number)
    {
       string s;
       float t = 0;
       float tot = 0;
       integer i;
       integer j = 20;
       while(j--){
           i = 256;
           llResetTime();
            while (i--) {s = llJsonSetValue(jText, [JSON_APPEND], "3");}
            t = llGetTime();
            llOwnerSay((string)t);
            tot += t;
        }
        llOwnerSay("Average time: " + (string)(tot/20));
    }
}

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

//				LSL list
list List = ["1","2"]; // ["1","2","1","2"] then ["1","2","1","2","1","2","1","2"]

default
{
	state_entry(){
	llOwnerSay((string)llGetFreeMemory());
	}

    touch_start(integer total_number)
    {
	   list s;
       float t = 0;
       float tot = 0;
       integer i;
       integer j = 20;
       while(j--){
           i = 256*4; // *4 required to avoid 0 time!
           llResetTime();
            while (i--) s = List + "3";
            t = llGetTime();
            llOwnerSay((string)(t/4));
            tot += t;
        }
        llOwnerSay("Average time: " + (string)(tot/80)); // 4*20 = 80
    }
}

 

Link to comment
Share on other sites
PeterCanessa Oh

PeterCanessa Oh

Posted
Posted

Perhaps I'll keep all my processing off-world anyway then.  Not much point if it takes more memory and is slower as well.  Thankfully a lot of what I do is either trivial enough to be easily handled by LSL scripts or complex enough that it's worth bouncing off the web, letting a 'proper' programme do the work and throw the results back.

(Lol, and here's me messing about with Python at the moment, which isn't even compiled, but interpreted ^^.)

 

Link to comment
Share on other sites
LepreKhaun

LepreKhaun

Posted
  • Author
Posted

Though the time approximations are correct I must point out that the list memory requirements are deceptively low because of how they're formed, with nothing but "1" and "2" as elements. Mono does an optimization with like string elements within lists it seems, as you can discover playing with this little snippet.

// Change these one at a time and resave// 16 byte penalty for each unique element within...list List = ["A","A","A","A","A","A","A","A"];default{    state_entry()    {        llOwnerSay((string)llGetFreeMemory());    }}

 

Link to comment
Share on other sites
LepreKhaun

LepreKhaun

Posted
  • Author
Posted

Adromaw Lupindo wrote:

Sorry was I reading the OP list wrong then, I thought JSON's getfree memory had more memory(free) reported every time up there. 

Which is which?

Edit: Or did Pete mean relative to off-world processing?

You were reading it right but, because the variables are global, the list gets a "bonus" for like string elements is all  Lists of random types may be longer than their string serializations (which a json array basically is) unless one is preserving type information. And a list of vectors or rotations is subjected to float string formatting (where every float within them is written out with 5 decimal point precision, ie <1,1,1>  = "<1.00000, 1.00000, 1.00000>").

 

And, yes, the need for off-world processing/storage does not look to be threatened by Json text, for sure. 64K is a miserably small size when you need to store and work with more than just a few dozen LSL keys. Plus you have to add in the necessary user functions one has to have for the lack of native support in LSL for array and map operations.

 

With that being said, Json text is becoming the preferred data serialization format for web client/server communication and learning to work with it may enhance off-world database capabilities. And there are primitive abstract data types that can prove quite useful (such as a stack) that can be effectively implemented with Json text (see https://wiki.secondlife.com/wiki/User:LepreKhaun_Resident/Stack_Implementations_using_Json_Arrays for examples) as well as replacing multiple lists with an array of arrays in some instances.

 

You just don't want to be using Json text manipulations in time critical places. It is definitely slow...

Link to comment
Share on other sites
LepreKhaun

LepreKhaun

Posted
  • Author
Posted

PeterCanessa Oh wrote:

My brain in neutral, I looked at the timing figures and glanced at the memory-use and assumed they were also worse so reporting memory used.  They are, indeed, memory free and therefore better.  Oops.

Me bad. I should have made it clear in my first posting that I was relying on a heap optimization trick to allow my lists to get to the size I was before incurring stack-heap collision to better show the time differences. It may have been less confusing if I had left out the free memory at each step since this would not reflect what would be obtained if the variables had been local.

[ETA: Just tried the same retrevial algorithm for an item from a list as I gave in my original posting, but with the list declared locally. It refuses to compile with the list at 1k elements, which is why I used the global variable trick. Sorry for the confusion.]

Link to comment
Share on other sites
LepreKhaun

LepreKhaun

Posted
  • Author
Posted

PeterCanessa Oh wrote:

I think I'm the only one introducing the confusion.  Sorry to have mucked up your thread ^^

Oh no, not at all! Your input adds to clarifying the differences of Json text arrays and LSL lists. And one should be aware of these differences when deciding when one might be preferred over the other.

 

Though Json text may have an advantage of implementing nested data structures and can achieve a certain amount of memory savings at times, the LSL implementation carries a distinct time disadvantage. Only by weighing all considerations might one come to a decision which might be better in a specific application.

Link to comment
Share on other sites
You are about to reply to a thread that has been inactive for 3925 days.

Please take a moment to consider if this thread is worth bumping.

Please sign in to comment

You will be able to leave a comment after signing in



Sign In Now
 Share
Go to topic listing
×
×
  • Create New...