64K should be enough for anyone

[animats]

Or, notes on doing a large problem in too many little scripts.

I've been coding an NPC system to replace SL pathfinding. It's mostly working at this point. Some things I've learned.

Measuring memory

Filling the 64K memory space for a script produces a "stack-heap collision" error. So it's useful to know how much you have left. llGetFreeMemory() is helpful but will be an underestimate until after a garbage collection. So the following snippet of code may be useful:

//
//  needmem -- need at least this much free memory. Return TRUE if tight on memory.
//
integer needmem(integer minfree)
{
    integer freemem = llGetFreeMemory();                // free memory left
    if (freemem < minfree)                              // tight, but a GC might help.
    {   
        integer memlimit = llGetMemoryLimit();          // how much are we allowed?
        llSetMemoryLimit(memlimit-1);                   // reduce by 1 to force GC
        llSetMemoryLimit(memlimit);                     // set it back
        freemem = llGetFreeMemory();                    // get free memory left after GC, hopefully larger.
    }
    if (freemem < minfree)                              // if still too little memory
    {   
        return(TRUE);
    }
    return(FALSE);                                      // no problem
}  

This will reliably tell you if you're running low on memory, without forcing too many unneeded garbage collections. Keeping about 3000-4000 bytes free at the place where the program uses the most memory is a good idea. Remember that you always need enough free space for the biggest link message anyone can send you.

Link messages

Link messages are how multiple scripts talk to each other. They go into the event queue with other events for the script, and there's a limit of 64 events on the event queue. More than that, and events are "silently dropped". So scripts can't be busy for too long, or their event queue will fill up. Big jobs may have to be divided into incremental steps run from a timer event. This would be much easier if LSL had objects.

Again, you get the link messages from everything in your prim, whether you want them or not. Remember that lists are copied when passed as a function parameter, so discard the irrelevant ones in the link message event before doubling their size with a function call.

Packaging up data as JSON is useful. LSL has built in JSON marshaling, so you don't have to parse anything character by character.

llMessageLinked(LINK_THIS, PATHPLANPREPPED, llList2Json(JSON_OBJECT,
                ["target",gPathprepTarget,"points", llList2Json(JSON_ARRAY,pts)]),"");

The llList2Json function does all the type conversions for you, and handles all escape characters needed. Note that the array of vectors, pts, is automatically converted to a JSON array.
 

    link_message(integer status, integer num, string jsonstr, key id)
    {   if (num == PATHPLANPREPPED)                                     // if request for a planning job
        {   key target = (key)llJsonGetValue(jsonstr,["target"]);  // get target if pursue
            list points = llJson2List(llJsonGetValue(jsonstr,["points"]));     // the preprocessed static path points
            integer len = llGetListLength(points);
            integer i;
            list gpts;                                             // points as a list of vectors
            for (i=0; i<len; i++) { gPts += (vector)llList2String(points,i);}   // convert to list of vectors
            ...

Here's the receive side. llJsonGetValue will extract one value from the JSON string. That value comes in as a string, and has to be cast to the desired type. JSON arrays come out as a list of strings, and each value has to be converted to the desired type. That's what the FOR loop is doing. This is kind of clunky, but at least you're not parsing strings in LSL.

[animats]

With enough 64K programs...

The end result uses about 3x as much memory as one big program, took about 4x as long to write and debug, and is slower. About a quarter of each program consists of the code to communicate with the other programs.

I'd really like to be able to get more script memory, even if it cost some land impact. Doing it this way is just too hard.

 

[Nova Convair]

Good idea to use json to send lists. Didn't need that so far but I'll try to remember that.

About the scripts - yes - of course but LSL was never meant to run big tasks. If you give more memory and features ppl would simply abuse it and even try to generate bitcoins if that would be even slightly possible.

Scripts are mainly not made by responsible developers here.

And for LL there always is the question: what do that cost and what do they get out of it?

[Rolig Loon]

Like you, Nova, I have not seen much practical need for JSON so far, but the idea of using it to send lists in a LM is an interesting thought.  I'll have to play with it.  Thank you, @animats.

You're right that LSL was not designed to run big tasks, or even small tasks that have to crunch a lot of numbers. Within its limitations, though, we can do some rather complicated jobs.  I find myself outsourcing some kinds of subtasks -- ones that involve a lot of list handling, for example -- to extra scripts and then passing data back and forth among them as link messages.  There's an increased risk of race conditions as a result, but if I take care to verify which steps are already completed and which are waiting, the extra scripts provide more free memory to work with and can help avoid some truly complicated logic.

I'm not quite sure whether you meant

8 hours ago, Nova Convair said:

Scripts are mainly not made by responsible developers here.

to emphasize the point that developers (that is, non-scripters) are not writing their own scripts or whether you meant that developers who write their own scripts are irresponsible.  I somehow doubt that you meant the latter, but then I am still puzzled by the word "responsible."   There has always been a bit of a divide between those who create visible objects (mesh or prim artists) and those who make them come alive with scripts.  Although there are great examples of creators who can do both, we typically have different skill sets.  Especially as we create complicated structures that demand tricky scripting, it's important to talk to each other throughout development.  What we (and Linden Lab) get out of it is a world that is not only more enticing to the end user but also makes more efficient use of resources.  That is truly "responsible" at all levels. 

[NeoBokrug Elytis]

What about llGetUsedMemory() or llScriptProfiler() ?

[animats]

On 11/4/2019 at 10:14 PM, NeoBokrug Elytis said:

What about llGetUsedMemory() or llScriptProfiler() ?

I'm already using llGetFreeMemory(). See above.

Here's what I'd like to have added in LSL. These would help prevent stack-heap collisions caused by big data coming in from outside the script. A few LSL events have no incoming size limit. They should.

Proposed changes. (Not that I expect them to be implemented, but it's worth documenting.)

New LSL call: llMessageLinkedFilter(list nums, integer maxstr, integer maxid)

Call this with a list of integers for "nums", and then messages for llMessageLinked are only queued if their "num" parameter is on the list. This allows filtering incoming messages by message type before they go onto the queue. Right now, if you have 10 scripts, and you send a message with llMessageLinked, all 10 scripts are awakened, get the message, and most of them discard it immediately. This creates O(n^2) overhead on the number of scripts, which is a bad thing. Also, it's possible for a script to be crashed by a message that's too big for it, even though the script would discard that message immediately. The script can't discard it before it blows the memory limit.

The "maxstr" parameter is the maximum length of the "str" parameter of an incoming message. Longer messages are truncated. Length means the same thing as the value from  llStringLength. If you want a limit of 100 characters, set this to 101, and check the string length. If it's 101, it's oversize and an error.

The "maxstr" parameter is the maximum length of the "id" parameter of an incoming message.  Works the same way as "maxstr".

 

llGetStaticPath, new param constant GSP_WAYPOINT_LIMIT.

Usage is to add [GSP_WAYPOINT_LIMIT, nnn] to the params to llGetStaticPath. No more than nnn waypoints will be returned. This does not change the error status returned. If you use this, the length of the list returned must be checked to see if the limit was hit. Currently, there is no limit on the number of waypoints returned, so this call can crash a script with a stack-heap collision.

 

Are there any other events or calls which can return an object of unbounded size? Those are the only two I know of. HTTP replies and listens, the other two main sources of big data, have a size limit.

 

 

[NeoBokrug Elytis]

On 11/18/2019 at 12:30 PM, animats said:

I'm already using llGetFreeMemory(). See above.

...

New LSL call: llMessageLinkedFilter(list nums, integer maxstr, integer maxid)

I specifically said llGetUsedMemory(), which is different and more accurate than llGetFreeMemory().

Also I don't think having a new llMessageLinkedFilter is a good thing.  Instead why not just limit what you send with the sending script using llGetSubString(message, 0, 100)?

You can also avoid getting all 10 scripts triggered with llMessageLinked by using other prims and directing messages to specific prims, instead of LINK_SET or having all scripts in LINK_ROOT.

If you're sending llGetStaticPath over a llMesageLinked, you can use llList2List() to reduce what is returned before you send it to other scripts.

[Mollymews]

agree with Neo

as the author of the sender code as well as the receiver code we do have the responsibility of managing the volume of message data ourselves

[Qie Niangao]

12 hours ago, NeoBokrug Elytis said:

You can also avoid getting all 10 scripts triggered with llMessageLinked by using other prims and directing messages to specific prims, instead of LINK_SET or having all scripts in LINK_ROOT.

This seems like it should work as long as the sending script has omniscience of all recipient scripts, and I guess that could be established dynamically among linked scripts given an elaborate enough subscription/registration protocol.

Personally, though, I'm less comfortable distributing scripts around the linkset. For one thing, there are no such functions as llGetLinkScriptState() nor llResetOtherLinkScript(), but I suppose that's all doable with llRemoteLoadScriptPin() and enough code -- and enough patience.

My other hesitation would be that I really don't know the transmission overhead of targeting messages by link. Is it the same as would be the proposed subscription by message class? This is definitely knowable (at least as visible to scripts themselves), but I've never looked at it.

And maybe it's just me, but it seems aesthetically off-putting to have to add prims to improve computing efficiency.

On the other hand, I don't often write programs large enough to face these problems.

Also, the same untargeted broadcasting of events across scripts applies to dataserver and http events, so attacking the link message version of the problem in isolation may not be ideal.

[Profaitchikenz Haiku]

59 minutes ago, Qie Niangao said:

Personally, though, I'm less comfortable distributing scripts around the linkset.

This is an ambivalent situation: reducing the number of individual scripts seems to be something to aim for, but the whole idea of link messages is that there are scripts distributed around the link set. Trying to fit everything into a single 64K (or larger) script means link messages become superfluous.

 

The thought that has occurred to me is, where link children just need simple script functionality, why not compile them in LSL so that they only require 16K, leaving the more complex scripts to be compiled in mono? 16K allows space for a limited set of link messages to react to. I'm not sure if this is feasible but it could reduce the overhead on the servers.

Edited November 21, 2019 by Profaitchikenz Haiku

[NeoBokrug Elytis]

5 hours ago, Qie Niangao said:

And maybe it's just me, but it seems aesthetically off-putting to have to add prims to improve computing efficiency.

That's where I use a transparent 1 triangle mesh shape, if I need a large system with a lot of scripts.  But I strive to smash everything into one or two scripts where I can.

We met in passing once @animats at a Linden Office hour.  You had mentioned that you too were also working on a Keyframe Pathfinding solution, so you piqued my interest.  I'm glad I'm not the only person that found LLs Pathfinding system lacking.

As others have noted, it would be more beneficial to try to cram as much as you can into one script.  Obviously things like a maze solver would probably still need their own script, that alone is rather big.  However the majority of your logic flow above can probably be squished into one script.  It *IS* harder to manage, but you'll benefit from a significant speedup when not needing to pass data with link messages. 

My only advice to save script memory is to avoid using strings and use integers where possible.  If you have the same logic blocks used over and over - declare them as a function.  Avoid declaring variables if you're only going to use them once in an event, instead inline the function calls.  Everything DOES get ugly trying to save memory, but that's how it be.  I have worked on my own non-havok pathfinding solution as well.  I have managed to fit everything you mentioned above (sans maze solver) into one script, with another single auxiliary script as a very robust llDialog menu to control tons of variables for the end user.  I don't use a true maze solver, and instead use a basic logic tree.  So it can be done.  But that IS accounting for all the parcel flag problems and much more.

 

Edited November 21, 2019 by NeoBokrug Elytis

[NeoBokrug Elytis]

5 hours ago, Qie Niangao said:

This seems like it should work as long as the sending script has omniscience of all recipient scripts, and I guess that could be established dynamically among linked scripts given an elaborate enough subscription/registration protocol.

That's a bit overkill.  Just remember which scripts are in which links, and you're good.

[Qie Niangao]

3 hours ago, NeoBokrug Elytis said:

That's a bit overkill.  Just remember which scripts are in which links, and you're good.

Right, if the project is self-contained. I was thinking of how a script -- a server, say -- could interact with other scripts that aren't known in advance, or might even come and go, written and managed by other parties. I'm not proposing to devise a protocol for such a thing without some application in mind, but I'd find it reassuring to think it possible, if it's ever needed.