"Linkset Data" is coming.

[Love Zhaoying]

1 hour ago, primerib1 said:

I think it's because the garbage collector triggered late. So the temporary lists etc you have within an event handler lingered for some time before they get freed.

And remember because all values are practically immutable, adding something to a list means you actually end up with 2 lists: Original list and the list with the added item. At least for awhile until GC gets triggered and the original list is destroyed.

And if you're quickly adding lots of items into a list (e.g. in a loop), you will quickly consume memory in a triangular way (1, 3, 6, 10, 15, 21, and so on... f(n) = sum(1..n) ).

And this begs the questions:

When does the GC get triggered?

Is it possible for us to trigger the GC? Maybe by switching to a different state then switching back?

It doesn't use lists, at all. Just JSON.

Now that I've converted it to LSD, it still shows a surprising minimum memory free, even though it starts with 40k. It only keeps 1 "large" string in memory at a time now, and re-reads from LSD for every JSON call after the initial write.

But, unless I keep using larger and larger JSON, it should not get the stack/heap collusion faults for awhile. I'll check further into where the memory is going after the next series of improvements / feature changes. 

The only event handler is link_message(). Guess I need to make sure that is writing directly to LSD and clearing the incoming string.

Basically, this script is a "best case scenario" for never running out of memory. I think it only has some very shallow recursion, which of course I can refactor to use iteration. 

[Love Zhaoying]

1 hour ago, primerib1 said:

When does the GC get triggered?

Sorry, forgot to answer your questions. Don't know - midst of processing. I was able to restart the failing one and it word almost always work on the next run; if not, then after 2 more runs. As in my last response, I'll probably not revisit the pre-LSD version.

1 hour ago, primerib1 said:

Is it possible for us to trigger the GC? Maybe by switching to a different state then switching back?

Nope, only 1 state! Sure, I could have forced it to happen more often with bigger strings but, since it would fail randomly, forcing a failure by increasing memory doesn't prove why it failed only sometimes with the same data.

I think your "late GC" guess sounds most likely!

Edited January 21, 2023 by Love Zhaoying

[Frionil Fang]

1 hour ago, primerib1 said:

When does the GC get triggered?

I find that a minimal sleep to ensure execution frame ends will clear the "lingering memory" left behind by previous values of variables.

28 minutes ago, Love Zhaoying said:

It only keeps 1 "large" string in memory at a time now, and re-reads from LSD for every JSON call after the initial write.

But, unless I keep using larger and larger JSON, it should not get the stack/heap collusion faults for awhile. I'll check further into where the memory is going after the next series of improvements / feature changes. 

And because of above, this is not quite true. If you have a huge chunk of JSON string in memory and re-read it from linkset data, the old string may linger and stack-heap collide. This example script will provide an easy way to replicate and avoid the problem:

default {
    state_entry() {
        string dummy = "0123456789012345678901234567890123456789"; // 80 bytes
        integer i;
        for(i = 0; i < 9; ++i) // 80 * 2^9 uses up roughly 40k
            dummy += dummy;
        llLinksetDataWrite("0", dummy);
        llSleep(0.001); // ensure execution frame ends
        for(i = 0; i < 10; ++i) {
            dummy = llLinksetDataRead("0"); // spam-reread the string, this will crash almost certainly
            // llSleep(0.001); // uncomment to ensure execution frame ends between reads to stop crashing
        }
    }
}

 

[Love Zhaoying]

6 minutes ago, Frionil Fang said:

I find that a minimal sleep to ensure execution frame ends will clear the "lingering memory" left behind by previous values of variables.

And because of above, this is not quite true. If you have a huge chunk of JSON string in memory and re-read it from linkset data, the old string may linger and stack-heap collide. This example script will provide an easy way to replicate and avoid the problem:

default {
    state_entry() {
        string dummy = "0123456789012345678901234567890123456789"; // 80 bytes
        integer i;
        for(i = 0; i < 9; ++i) // 80 * 2^9 uses up roughly 40k
            dummy += dummy;
        llLinksetDataWrite("0", dummy);
        llSleep(0.001); // ensure execution frame ends
        for(i = 0; i < 10; ++i) {
            dummy = llLinksetDataRead("0"); // spam-reread the string, this will crash almost certainly
            // llSleep(0.001); // uncomment to ensure execution frame ends between reads to stop crashing
        }
    }
}

 

Hmm, interesting.

I forgot that llSleep() helps/forces GC.

You'd think GC would also get triggered during quiescent time, between events.

[Frionil Fang]

Maybe in your case somethiing else is too memory intensive right after reading the new string and being "kind of over the limit" for a while, dunno. I haven't investigated further once I figured out a few well placed single-frame sleeps fixed the extremely memory constrained script I had crashing because of it.

It's kind of funny that scripts that are too efficient and run fast crash, but ones that take their time and stall don't... modern LSL problems.

[primerib1]

6 minutes ago, Frionil Fang said:

Maybe in your case somethiing else is too memory intensive right after reading the new string and being "kind of over the limit" for a while, dunno.

Can also stack growth.

Variables -- and lingering values -- eat up heap.

Juuuust reaching the limit, but not yet crossing it.

Then you make a function call and the stack grows because of the things that need to be pushed into stack...

Voila! Stack-Heap collision.

EDIT: Oh yeah I just remembered that LSL is "pass by value" ... so if you call a function with a list as a parameter ... well, the list is copied into the stack ...

So if you call, say llGetSubString() with a loooong string ...

Edited January 21, 2023 by primerib1

[Quistess Alpha]

6 hours ago, primerib1 said:

Is it possible for us to trigger the GC?

I think I got from @animats  that you can force the garbage collection by (ab)using llSetMemoryLimit() (add and sub 1 from the limit)

[Love Zhaoying]

4 hours ago, primerib1 said:

Can also stack growth.

Variables -- and lingering values -- eat up heap.

Juuuust reaching the limit, but not yet crossing it.

Then you make a function call and the stack grows because of the things that need to be pushed into stack...

Voila! Stack-Heap collision.

EDIT: Oh yeah I just remembered that LSL is "pass by value" ... so if you call a function with a list as a parameter ... well, the list is copied into the stack ...

So if you call, say llGetSubString() with a loooong string ...

Looks like none of these should be a problem with the current script (at least, since converted to LSD)!

- No global variables of any size except 1 now, with LSD (there were several before)

- Not calling functions in any "deep nest" or recursively, so the stack should not get eaten

- Not using lists, except for passing very short lists as calls to pass "specifier lists" for llJson calls (not nested, not recursive)

- Not using llSubString()

So, my problems should now "go away" and I can add an llSleep() if needed to see if the final memory goes up any.

Weird that quiescing and waiting for the next link_message() event doesn't effectively trigger GC like llSleep() does!

Thanks, everyone!

[primerib1]

1 hour ago, Love Zhaoying said:

So, my problems should now "go away" and I can add an llSleep() if needed to see if the final memory goes up any.

You should add llSleep(0.01) on strategic places anyways.

I just added them to my "Outfit System" scripts 😉

[animats]

5 hours ago, Quistess Alpha said:

I think I got from @animats  that you can force the garbage collection by (ab)using llSetMemoryLimit() (add and sub 1 from the limit)

//
//  pathneedmem -- need at least this much free memory. Return TRUE if tight on memory.
//
integer pathneedmem(integer minfree)
{
    integer freemem = llGetFreeMemory();                // free memory left
    if (freemem < minfree)                              // tight, but a GC might help.
    {   pathMsg(PATH_MSG_WARN, "Possibly low on memory. Free mem: " + (string)freemem + ". Forcing GC.");
        integer memlimit = llGetMemoryLimit();          // how much are we allowed?
        llSetMemoryLimit(memlimit-1);                   // reduce by 1 to force GC
        llSetMemoryLimit(memlimit);                     // set it back
        freemem = llGetFreeMemory();                    // get free memory left after GC, hopefully larger.
    }
    if (freemem < minfree)                              // if still too little memory
    {   pathMsg(PATH_MSG_WARN, "Low on memory after GC. Free mem: " + (string)freemem);
        return(TRUE);
    }
    return(FALSE);                                      // no problem
}        

That's the code I use in my NPCs. The llGetFreeMemory call is cheap but a garbage collection is expensive. So you don't want to force a GC unless you have to.

It's useful if some external event or LSL call can return a lot of data. My NPCs say hello to new people in the area. A large number of avatars passing through could result in a large list returned from llGetAgentList. So there is a minimum amount of free memory required before calling that call. Other transient overloads include incoming link messages and paths from llGetStaticPath.

This is only useful if you have a strategy for reducing memory consumption when under stress. The NPC remembers who's been around for 15 minutes or so to avoid re-greeting people who left the parcel briefly and returned. The recent visitors list gets trimmed if memory gets tight.

I also use this in the path planner, which is planning a few moves ahead. If the path planner gets tight on space, it stops planning ahead. The NPC then stops and stands, arms folded, while the planner runs and figures out the next move. (That wait also happens in some other situations, such as script overload or moving avatars cutting in front of the NPCs.)

Another useful trick: you can read DEBUG_CHANNEL from LSL and detect out of memory script crashes. Debug messages are local chat, so things that crash when you're not around do so silently. You have to read DEBUG_CHANNEL from a different prim than the one with the problem, because scripts don't get listen events for their own output. If you read DEBUG_CHANNEL from a script, you can send the errors some place else, to a server, IM, or email. I put the internal logging in a circular buffer, and if something bad happens, it's all packed up into an email and mailed to me.

All of this is a huge pain. It took months of work to make the NPCs work reliably without memory overflow, and now I don't change the programs because the retest time is too long. In summary, unless you absolutely have to run close to memory limits, don't.

Edited January 21, 2023 by animats
Plural

[Qie Niangao]

I stumbled on the llSetMemoryLimit() trigger for garbage collection too, churning some TPS report to HTTP as fast as possible, but I don't think I ever tried llSleep(0.0) which should skip to the next frame. Maybe I can eventually find that code and see if that llSleep works when the GC is needed immediately (as opposed to merely slowing things just enough that unforced GC keeps up).

[Wulfie Reanimator]

6 minutes ago, Qie Niangao said:

I don't think I ever tried llSleep(0.0) which should skip to the next frame

As long as the value is greater than zero! Otherwise llSleep doesn't do anything.

default
{
    state_entry()
    {
        llOwnerSay("loop");
        integer i = 5;
        while (i--) {
            // Prints the same time multiple times without sleep.
            llOwnerSay((string)llGetTime());
            llSleep(0.0);
        }
    }
}

 

[Love Zhaoying]

My attempt to fix my memory-restricted script didn't work so...I am completely changing when I write to LSD.

Everything was fine, until I tried to use a feature I hadn't used since conversion to LSD!

Instead of actually "merging" data while processing and writing out the data to LSD at the end, I am just writing the data to LSD as I discover it is needed.

This should do it! If I get constraints later, I can just change the Schema to have things in smaller pieces per Scope/Schema.  Since this piece is the Schema processor, I'm in control.

 

[Love Zhaoying]

2 hours ago, Love Zhaoying said:

My attempt to fix my memory-restricted script didn't work so...I am completely changing when I write to LSD.

Everything was fine, until I tried to use a feature I hadn't used since conversion to LSD!

Instead of actually "merging" data while processing and writing out the data to LSD at the end, I am just writing the data to LSD as I discover it is needed.

This should do it! If I get constraints later, I can just change the Schema to have things in smaller pieces per Scope/Schema.  Since this piece is the Schema processor, I'm in control.

 

Ok, I spent many hours getting this to work.  (Refactoring the code to process completely differently took more effort than I thought.)

Result, the script starts with 40k available memory. 

The script finishes with 28k available, after all cleanup and llSleep(0.01), etc.  There SHOULD actually be more memory available at the end, after all the cleanup.. but I'll take it!

[Kadah Coba]

On 1/20/2023 at 11:09 PM, primerib1 said:

Indeed. In script-space you can represent them as 4 integers then use strided list.

But since we're in the LSD thread, there's an additional limitation: You have to convert that into string as LSD can only store strings.

So all the "packing" discussion in this thread is just us discussing about the most efficient, reversible way of storing 4 integers in LSD.

If you simply stringify the integers, you can end up with 47 characters per encoded UUID when you're unlucky... (4 * 11 chars [if your integer is *very negative*] + 3 separators)

(Example of 'very negative' integer: -1564371818 = 11 characters)

I think what I meant was, once you get data down to being a list of ints, encoding generally becomes easier to manage. x3

I really wish there was a way to transmit binary data between scripts, and if LSD could have supprted lists (TL;DR I got is that it would have required a much bigger project, eg. wouldn't have happened). All the stringization we have to do because code has to be divided up amongst multiple scripts is painful, and all of that eats up so much resources that could be doing more with less. Larger scripts would be great, but there's a lot that needs to change on the backend before that could happen...

If the data is going to be mostly large ints, IMO llIntegerToBase64 is still the best option for speed and code size, the fixed 6 char length makes indexing a lot simpler. I used this method for storing large amounts of binary data via media faces (something which could be directly ported to LSD).

Currently kicking around a few ideas for non-fixed length encoding with arbitrary lists. Code for fixed structures would be more efficient, but a pain to have to customize for each data exchange type.

[Anna Salyx]

Add a feature request.  Not sure if it was already requested or similar, and my searching was coming up with nothing. This is for getting a count of keys matching a regex pattern. 

https://jira.secondlife.com/browse/BUG-233242

 

[Love Zhaoying]

5 hours ago, Kadah Coba said:

if LSD could have supprted lists

Using JSON with LSD works out the same, at least for me..

[Quistess Alpha]

11 hours ago, Anna Salyx said:

Add a feature request.  Not sure if it was already requested or similar,

https://jira.secondlife.com/browse/BUG-232800

"unactionable"

[Anna Salyx]

<comment redacted>

EDIT:  Oh, I thought that on my suggestion at first.  

Edited January 24, 2023 by Anna Salyx

[Love Zhaoying]

14 hours ago, Anna Salyx said:

This is for getting a count of keys matching a regex pattern. 

https://jira.secondlife.com/browse/BUG-233242

Guess I thought the "Find Keys" function already did that, oops!

[Kadah Coba]

18 hours ago, Love Zhaoying said:

Using JSON with LSD works out the same, at least for me..

I found out last night that the JSON functions are stupidly fast. By an order of magnitude, this is the fastest way to convert lists to/from strings while keeping types. Will have some benchmarks on this "eventually".

9 hours ago, Quistess Alpha said:

https://jira.secondlife.com/browse/BUG-232800

"unactionable"

Having support for lists naively in LSD, or other places outside of an individual script runtime, isn't currently possible for technical reasons. It would be nice, but we're gonna have to wait for a bunch of other changes first.

[Love Zhaoying]

3 minutes ago, Kadah Coba said:

18 hours ago, Love Zhaoying said:

Using JSON with LSD works out the same, at least for me..

I found out last night that the JSON functions are stupidly fast. By an order of magnitude, this is the fastest way to convert lists to/from strings while keeping types. Will have some benchmarks on this "eventually".

Great!

Yeah, I think JSON in LSL is generally a couple orders of magnitude better than lists for most things - not exaggerating!

 

[Kadah Coba]

29 minutes ago, Love Zhaoying said:

Great!

Yeah, I think JSON in LSL is generally a couple orders of magnitude better than lists for most things - not exaggerating!

 

Native binary stored JSON within LSL would be nice, the stringized we have now is very memory inefficient. Its fine for save/storing, comms to other scripts and for interfacing with external services, but will eat a lot of mem if uses much within a script as a list.

[Love Zhaoying]

1 minute ago, Kadah Coba said:

Native binary stored JSON within LSL would be nice, the stringized we have now is very memory inefficient. Its fine for save/storing, comms to other scripts and for interfacing with external services, but will eat a lot of mem if uses much within a script as a list.

But..what if..your list is..a list of..strings?

Dun dun dunnnnn!

(Assuming you didn't need a list of numbers in the first place.)

Edited January 25, 2023 by Love Zhaoying

[Kadah Coba]

5 minutes ago, Love Zhaoying said:

But..what if..your list is..a list of..strings?

Dun dun dunnnnn!

(Assuming you didn't need a list of numbers in the first place.)

Its possibly about as bad as lists in that example, though lists have the upside that duplicate values are stored as references, though that is kind of an edge case to make happen. There might also be some referencing happening on retrieved values, I can't remember, its been years since I tested this.