"Linkset Data" is coming.

[Love Zhaoying]

1 minute ago, Frionil Fang said:

Maybe it should be thought as more of a cache rather than a file, it could be invalidated at any time.

I like this!

In fact - if it were only semi - persistent, say until the script restarts - that would STILL be awesome!

Speaking of - sorry, but I may have missed this:

If you take a linkset into inventory and re-rez it, assuming it is "no-copy" (should always get the same UUID): Would its linkset data still exist? 

[Qie Niangao]

2 hours ago, Love Zhaoying said:

I like this!

In fact - if it were only semi - persistent, say until the script restarts - that would STILL be awesome!

Speaking of - sorry, but I may have missed this:

If you take a linkset into inventory and re-rez it, assuming it is "no-copy" (should always get the same UUID): Would its linkset data still exist? 

There were very early reports that this didn't work, but it was claimed to be fixed so I just tested to make sure, and the data persists with the object. Not only no-copy, either; in fact, the same data is in shift-drag copies of a linkset.

(And most of the applications I have in mind for this depend on it not being ephemeral, instead persisting like Experience KVP—or a link's Description field. I'd consider it a sadly wasted opportunity if it could only function as a cache between script restarts.)

[Love Zhaoying]

2 minutes ago, Qie Niangao said:

Not only no-copy, either; in fact, the same data is in shift-drag copies of a linkset.

Hmm...having the same data persist for a copy does not work for some of my use-cases!

Let's say the data is the "save state" for a "game". 

Making copies would mean the user could "cheat" by having a "restart point" with the old copies.

[Fenix Eldritch]

Thank you Quistess, Kadah, and Wulfie. After a bit of sleep, rereading your examples and those on the wiki page are making more sense now.

15 hours ago, Wulfie Reanimator said:

Are newlines allowed for keys?

It would seem so!

llLinksetDataWrite("\n","testdata");
string regex = ".";
llOwnerSay("regex result: '"+llList2CSV(llLinksetDataFindKeys(regex, 0, 10))+"'");
llOwnerSay("data: "+llLinksetDataRead("\n"));

returns

regex result: '
'
data: testdata

As I go back through the other examples, I've been jotting down examples of what work. Sharing them here for anyone interested:

sample keys: 
firstkey01, firstkey02, firstkey03, firstkey04, secondkey01, secondkey02, secondkey03, secondkey02, AAC, ABC, ACC, ADC, A9C, A.C, AAACC

.       matches any single character, including newline
\       escapes a special character (in [] only?)   ex: A[\.]C matches A.C only and not AAC,ABC,ACC...etc.

Logic and Grouping
|       OR opperand             ex: (first|second)key.. matches all the firstkeyxx and secondkeyxx entries
()      capturing group         ex: A(A|D)C matches AAC and ADC

Qunatifier (placed after an element)
+       matches preceeding element once or more     ex: B+C+ will match BCC
*       matches preceeding element zero or more     ex: A*B*C* will match AAACC
?       matches preceeding element zero or once     ex: B?C* will match BCC, but B?C? will not.
{3}     matches preceeding element exactly 3 times  ex: A{3}B*C{2} will match AAACC
{3,}    matches preceeding element 3 or more times  ex: A{1,}B*C{2} will match AAACC
{2,4}   matches preceeding element 2 to 4 times     ex: A{1,4}B{0}C{2} will match AAACC, but A{1,2} wouldn't (too many As)

Classes (can chain multiple ranges together)
[]      matches a single character within           ex: A[ABCD]C matches AAC, ABC, ACC, ADC
[a-z]   matches any single chr in range             ex: A[A-D]C matches  AAC, ABC, ACC, ADC
[ -~]   matches any single printable ASCII char
[0-9]   matches any single number in range          ex: firstkey[0-9]{2} matches the firstkey01 through firstkey04 entries
[^]     matches anything NOT specified in []        ex: A[^B-D] would match AAC and exclude ABC/ACC/ADC

Anchor and Boundaries
^       matches beginning of string                 ex: ^second.* matches all of the secondkeyxx entries
$       matches ending of string                    ex: .*C matches anything that endes in "C"

Inline Identifiers
(?i)    case insensitive                            ex: (?i)abc will match ABC and abc

 

Edited October 13, 2022 by Fenix Eldritch
typos

[Love Zhaoying]

I wonder if the data is indexed in such a way that the RegEx expressions are "efficient"? Or, is it effectively doing a "table scan" -- reading all the rows in the LinkSet data and comparing each against whether it matches the RegEx.

[Rider Linden]

6 minutes ago, Love Zhaoying said:

I wonder if the data is indexed in such a way that the RegEx expressions are "efficient"? Or, is it effectively doing a "table scan" -- reading all the rows in the LinkSet data and comparing each against whether it matches the RegEx.

It's a table scan. But the table is generally small enough (few thousand records?) that it shouldn't be too large a hit.

[Rider Linden]

32 minutes ago, Love Zhaoying said:

Hmm...having the same data persist for a copy does not work for some of my use-cases!

Let's say the data is the "save state" for a "game". 

Making copies would mean the user could "cheat" by having a "restart point" with the old copies.

You can address this by removing keys that you would rather not have persist in the script's default state_entry handler.

[Love Zhaoying]

13 minutes ago, Rider Linden said:

You can address this by removing keys that you would rather not have persist in the script's default state_entry handler.

Thanks for addressing my use-cases! 🙂 

[Jenna Felton]

I could not test it yet (was just lucky to find in the BUG-232751 that this feature is live in Mauve in Aditi) but in that Jira is said

Quote

The llLinksetDataListKeys and llLinksetDataFindKeys functions both sort the list of keys before returning them to your script.

Is it important to sort the keys in these functions? Because when I am just interested in the list of the keys without needs for order, the sorting is an overhead. When sorting not used internally, maybe it is better to drop it and when you need the keys sorted, you just wrap the output in llListSort?

Edit: Ok, after I know the parameters, sorting is indeed needed. The functions return keys out of a certain range and for this you need a proper order to make the output deterministic.

Edited October 13, 2022 by Jenna Felton

[Jenna Felton]

On 10/13/2022 at 12:06 AM, Kadah Coba said:

As somebody that will link different things together when building out my region... or even as someone that would make things that could be linked later to random things by end users...

How about each object has their own LSD store, which can be accessed from other links, and maybe has some optional method to restrict access, possibly via a pin. That would resolve the issue of unwanted collisions when linking things that could have incompatibilities when they are trying to share the same LSD store, and things that can support sharing can just always use LINK_ROOT and handle the merging themselves via the on link event. If there is some issue with each object being able to have an LSD store, make a limit of how many their can be per link set and either prevent linking (preferable) or block/wipe the additional ones being added.

I quite like this suggestion. Maybe the LSD would be attached not to the root link but to every prim in linkset? 64kB would be the alotment per object (linkset) and not per prim. To access the LSD a script would first open the LSD via the function

integer llLinksetDataOpen(integer link, integer writemode, string password);

The function would allow opening the LSD in any prim in the linkset, while using LINK_THIS would allow access to the LSD of the same prim, when really needs and allow to link the prims without the scripts damage the LSD from an other prim after linking. Delinking a prim would take the attached LSD with the prim, as well.

The return value (LSD index) would address the LSD the script has opened (0 = FALSE means the script provided wrong password and can not use data functions). When you want to have access to LSD from multiple prims at same time, e.g. to add values from different stores, then this LSD index need to be provided in the data accessing functions.

It would be also good to have the way to lock the LSD from changing by different scripts. For example when a script opened the LSD in write mode, no other script can do this and is delayed until the LSD is closed via

llLinksetDataClose(integer index);

As alternative, instead of using the number, returned by llLinksetDataOpen, the data function and llLinksetDataClose would accept the link number as reference of the LSD, since no link can have two LSD attached.

Edit

I not realized yesterday, letting scripts lock multiple LSD stores can lead to a circular lock where two or more scripts try to lock the same 2 or more stores in different order and lock out themselves.

To avoid this, the scripts should be able only to lock different stores in the order of the LSD-associated link numbers. Or not delay when a script tries to open a locked store but receive an error response so the script can lift the lock on an other store and try again in a random while.

Or we drop hard-locking completely (because it should ensure that two scripts not change the same store at same time destroying the data), but instead use the soft-locking in the manner of the Experience Tools:

integer llLinksetDataUpdate(integer store, string name, string value, integer checked, string original_value);

This will not prevent two scripts damage the same LSD store but they can detect if a store is in use and with some discipline on the scripter's end refrain from overwriting.

Edited October 14, 2022 by Jenna Felton

[Elisabeth Ohmai]

Everything nice with LSD, but what about commercial NO-MOD scripts? With description field customer can to set initial values for script without permission to edit the script itself. We need UI for view/edit LSD, or this is not full replacement for description field.

[Coffee Pancake]

2 minutes ago, Elisabeth Ohmai said:

Everything nice with LSD, but what about commercial NO-MOD scripts? With description field customer can to set initial values for script without permission to edit the script itself. We need UI for view/edit LSD, or this is not full replacement for description field.

This wont replace notecards or descriptions as ways to pass a script settings, it just gives a script somewhere to store a decent amount of data.

My concern is more about this enabling anti consumer practices .. scripts that only work in prims with secret data in them, objects that can't be truly reset, weird DRM, products with a baked in limited number of uses, .. etc. Of course all this was possible before with an external server to store data, but that did create something of a barrier to entry for most of the evil. 

[Quistess Alpha]

1 hour ago, Coffee Pancake said:

scripts that only work in prims with secret data in them, objects that can't be truly reset, weird DRM, products with a baked in limited number of uses, .. etc. Of course all this was possible before with an external server to store data

I agree that all those things are bad, and that LSD ~Might make them easier buuut:

llGetObjectCreator() ; and the fact that no-mod scripts in no-mod no-copy objects can't be reset (As far as I recall) has those evil niches filled even without an expensive server.

[Love Zhaoying]

46 minutes ago, Quistess Alpha said:

no-mod scripts in no-mod no-copy objects can't be reset

TIL!

[Qie Niangao]

4 hours ago, Elisabeth Ohmai said:

Everything nice with LSD, but what about commercial NO-MOD scripts? With description field customer can to set initial values for script without permission to edit the script itself. We need UI for view/edit LSD, or this is not full replacement for description field.

Right, but a page or so ago we were promised that the viewer interface to Linkset Data only needed a simple read-only table and here under the tent I see the camel nose of feature creep. (I'd just note that ideally a scripter would provide some more helpful configuration UI than having to edit Description fields or Linkset Data directly, using the idiosyncratic "language" of that script's configuration parameters. Worst case, give the poor user a setup HUD, or at the very least a llTextBox with a clue-giving prompt.)

To other "secret data" concerns, again, Linkset Data cannot be secret in modifiable objects, regardless of the script's permissions. It can be encrypted, much as sekrit texture UUIDs are disguised in readable notecards, but encrypted or not it's visible to all by a script—trivially, for the read-only table the pre-creeping feature specified.

(Needless to say, I would so much rather leave any immediate UI for this feature to third party scripters or TPV devs or whomever, and devote Lab development to refining control of script access to these data which will define the limits of its utility for the lifespan of Second Life.)

What drives me slightly crazy here is that this is perceived as some scary new hiding place for evil scripters to work their wiles. Nobody fusses about a script's global variables being invisible, yet the completely exposed and user-editable Linkset Data is somehow anti-consumer. 

[Extrude Ragu]

To be honest I would have been quite happy with just an extra hidden string field like the description, the async stuff just adds a layer of complexity IMO , but I suppose its a whole lot better than nothing, hopefully this will bring about the end of the days of notecard configurations

[Kadah Coba]

On 10/12/2022 at 6:32 PM, Wulfie Reanimator said:

Genuine question: For the things we would expect to be able to link together (like sim scenery or clothing), how often do we imagine them to be using object-shared persistent data storage, instead of in-memory storage (or something in the description)? Are key-conflicts (and loss of data) in a situation like this something that's likely to become a common problem?

Considering this is possibly one of the most generally useful things to be added to LSL is a very very long time... I expect its gonna see a lot of use in new stuff as it gets know about.

Most obvious use that I envision causing conflicts either with keys or overruns is pose systems. Take a decorative item that uses LSD for texture info then link it to a sitting apparatus with a pose system and was already using nearly 64KB of data and you've suddenly have unpredictable data loss.

On 10/12/2022 at 6:55 PM, Quistess Alpha said:

The main use-case problem I'm envisioning is texture-change clothing. But, I think with the improved efficiency of utf-8 over utf-16, assuming you don't have any name-conflicts, I'd expect to be able to link 5+ really fat ones together before memory overflow, and that's assuming they don't make any effort to compress the keys (with utf-8, I'd think you could get a UUID down to 18~20 bytes? vs theoretical minimum of 16.)

(back of the envelope: 14+36 ~= 50 characters for texture name+key, *200 texture options = 10,000 bytes, or 1/6 of the local database.)

LSL string don't support every unicode character (would be nice to have a byte array type...). From the ASCII block, all but null work, which is kinda of annoying for this.

BASE127 works, which is typically doing to be 18-19 bytes per key (less if there are many leading zeros, but that's very uncommon). I think that's the best achievable without compression. BASE64 uses 24 bytes per key. The trade-off of BASE127 over BASE64 is that it takes much more time and uses more byte code.

Here's the result of a quick benchmark on a slightly laggy region. Each test is encoding and decoding 100 random keys and storing the encoded keys to a list in script mem.

[03:39] base: 127 Start: 13934 Current: 19596 Change: 5662 SPMax: 20948 Size Bound: 18 - 19 Runtime: 12.400760
[03:39] base: 127 Start: 13934 Current: 19614 Change: 5680 SPMax: 20902 Size Bound: 18 - 19 Runtime: 11.949040
[03:40] base: 127 Start: 13656 Current: 19576 Change: 5920 SPMax: 20850 Size Bound: 17 - 19 Runtime: 11.935850
[03:40] base: 127 Start: 13656 Current: 19586 Change: 5930 SPMax: 20796 Size Bound: 18 - 19 Runtime: 9.685167
[03:41] base: 127 Start: 13656 Current: 19602 Change: 5946 SPMax: 20762 Size Bound: 18 - 19 Runtime: 10.044070
[03:41] base: 127 Start: 13656 Current: 19552 Change: 5896 SPMax: 20832 Size Bound: 18 - 19 Runtime: 11.290470

[03:41] base: 64 Start: 10862 Current: 17574 Change: 6712 SPMax: 17574 Size Bound: 24 - 24 Runtime: 2.442940
[03:42] base: 64 Start: 10584 Current: 17526 Change: 6942 SPMax: 17526 Size Bound: 24 - 24 Runtime: 2.270569
[03:42] base: 64 Start: 10584 Current: 17526 Change: 6942 SPMax: 17526 Size Bound: 24 - 24 Runtime: 2.274925
[03:42] base: 64 Start: 10584 Current: 17526 Change: 6942 SPMax: 17526 Size Bound: 24 - 24 Runtime: 2.372391
[03:42] base: 64 Start: 10584 Current: 17526 Change: 6942 SPMax: 17526 Size Bound: 24 - 24 Runtime: 2.252426
[03:42] base: 64 Start: 10584 Current: 17526 Change: 6942 SPMax: 17526 Size Bound: 24 - 24 Runtime: 2.262833

 

On 10/13/2022 at 5:12 AM, Qie Niangao said:

Right, so for my first big LinksetData project I'll claim llChar(0xDF) —looks like "×"—as my namespace prefix. Nobody collide with it, okay? 😛 Just one extra byte per row.

And I promise never to call llLinksetDataReset(), so nobody else ever call it either, okay? 😛 Because you can never know when somebody might have your script in an object that gets linked with one that has my scripts inside.

Instead we'll all just call llLinksetDataDelete() one row at a time for keys found in batches until llLinksetDataFindKeys() gets no results for our personal namespace prefix. Because we'll all just agree to do that, okay?

(Am I the only one who sees a problem here?)

 

Yeah... this could be like the whole "reserved" channel mess again.

Would really prefer each prim/link being able to have its own store so we can just avoid the future mess and headaches mergers. Plus more storage is always more better.

23 hours ago, Frionil Fang said:

This is strictly my personal feeling on linkset data:

Trying to safeguard against what every other user/scripter might do seems like an unreasonable expectation, and from my point of view linkset data is meant to be a "sane", spacious replacement for hacks like storing data in description/other string fields of an object, not a secure remote server replacement.

Maybe it should be thought as more of a cache rather than a file, it could be invalidated at any time.

Personally, I'll be very happy to have what's basically a free 64 kB of data storage for my projects, since they tend to be single-script and not expected to allow linking to random things.

Edited to add: I understand that now would be a good time to expand the spec to have more than just the "sane bigger replacement", but the extra complexity... would we ever actually get it then?

Its about as persistent as script mem, if the user has mod on the object, they can reset any script, and also deal with whatever that breaks. The extra new thing on top of that is object mod also also allows read/write assess, which some people are getting really hung up on.The lack of not being able to have mod and block other scripts from messing with the LSD store is not a blocker for me (beyond that it will likely result in more no-mod/no-buy objects in the future).

I welcome the 64KB of shared storage. As-is now, its totally usable for me.

Buuuut.... If adding features is still on the table, I'll take, in order of importance to me:

1. Per link stores.
   • So we can allow for sidestepping the mess merges and conflicts can cause.
   • More storage is always welcome.
   • Would need link param on the functions to specify the store.
   • I could add memory submissive scripts and get at least 32KB of storage per script. Pretty sure LSD stores are less impactful than twice as many scripts. :p
2. Pin/keyed access.
   • Avoid having to make objects no-mod/no-buy, and/or using time expensive encryption, to prevent undesired access/tampering to stored data.
   • Could be a dedicated KVP entry in the store with each script needing to call an unlock function for the read/write/find functions return successfully.
   • Per key encryption would work too, which is something we could already.
3. Ability to access from other objects.
   • Would mostly want pin/key for this.
   • While long-distance access would be cool, but that's just Exp KVP at that point.
   • Simply the same rules as llGetObjectDetails would be good to me.
4. And I can't think of anything else.
   • A viewer-side bower is defiantly getting in to feature creep. Would rather have that for Exp KVP, but if we want it that badly, we could make such ourselves. I could likely make something that sync's KVP stores to/from a Google Sheet if someone want to fund my time.

8 hours ago, Elisabeth Ohmai said:

Everything nice with LSD, but what about commercial NO-MOD scripts? With description field customer can to set initial values for script without permission to edit the script itself. We need UI for view/edit LSD, or this is not full replacement for description field.

Use the mem savings to make a menu system or notecard reader. Description config has always been rather user-unfriendly for anything more than something simple. Nothing is taking away using descriptions in that manner in the future, so you can keep using them if you want. :p

I'm starting to wonder if I missed all the requests back in the day for a viewer front end list browser for llList.

1 hour ago, Extrude Ragu said:

To be honest I would have been quite happy with just an extra hidden string field like the description, the async stuff just adds a layer of complexity IMO , but I suppose its a whole lot better than nothing, hopefully this will bring about the end of the days of notecard configurations

Its not async though. You can use it similar to llList2String and llString2List.

On scripts that use a lot of mem for storage, they could use the savings to have a better menu system.

Edited October 14, 2022 by Kadah Coba

[Extrude Ragu]

1 minute ago, Kadah Coba said:

Its not async though. You can use it similar to llList2String and llString2List.

 

Oh yeah, just re-read, saw the event and assumed it was async, my bad

[Gabriele Graves]

3 minutes ago, Kadah Coba said:

Would really prefer each prim/link being able to have its own store so we can just avoid the future mess and headaches mergers. Plus more storage is always more better.

I like this and was thinking along the same lines.  It would make more sense, be much easier to use and less potential problems.

[Kadah Coba]

2 hours ago, Qie Niangao said:

Right, but a page or so ago we were promised that the viewer interface to Linkset Data only needed a simple read-only table and here under the tent I see the camel nose of feature creep. (I'd just note that ideally a scripter would provide some more helpful configuration UI than having to edit Description fields or Linkset Data directly, using the idiosyncratic "language" of that script's configuration parameters. Worst case, give the poor user a setup HUD, or at the very least a llTextBox with a clue-giving prompt.)

To other "secret data" concerns, again, Linkset Data cannot be secret in modifiable objects, regardless of the script's permissions. It can be encrypted, much as sekrit texture UUIDs are disguised in readable notecards, but encrypted or not it's visible to all by a script—trivially, for the read-only table the pre-creeping feature specified.

(Needless to say, I would so much rather leave any immediate UI for this feature to third party scripters or TPV devs or whomever, and devote Lab development to refining control of script access to these data which will define the limits of its utility for the lifespan of Second Life.)

What drives me slightly crazy here is that this is perceived as some scary new hiding place for evil scripters to work their wiles. Nobody fusses about a script's global variables being invisible, yet the completely exposed and user-editable Linkset Data is somehow anti-consumer. 

This. Its been a really weird hangup in this thread for some reason.

If that many users really must have this kind of crutch, fund/bribe/convince one of us FOSS devs to make a front-end. I'm pretty sure a viewer component is well outside of the scope for this project, and I'd greatly prefer LL dev time going to expanding LSD or other LSL functionality first and those are things we cannot affect ourselves (plus I'm not allowed to add server features without permission anymore).

[Love Zhaoying]

28 minutes ago, Kadah Coba said:

1. Per link stores.

Yes! Basically, letting each object (whether prim/mesh) have its own store would mean any scripts in those individual links NEEDING their own store wouldn't need to use the "our shared store, my key schema, my data, please don't delete all the data!" approach. It kinda-makes-sense why this new feature would be restricted to the "linkset" model, but..why?

[Fenix Eldritch]

Oh my god, it just clicked with me that y'all were abbreviating LinksetData as LSD.

45 minutes ago, Kadah Coba said:

Pin/keyed access.

That did come up during this week's server meeting (and in fact commanded a large portion of the conversation). I believe Rider said he was going to experiment in adding the feature, with a lean towards an additional set of read/write/delete functions that required an integer pin (that way you wouldn't lock out the entire store). Some suggested using a string for a passphrase, but if that happens, Rider said he'd count those against the store size.

Edited October 14, 2022 by Fenix Eldritch

[Gabriele Graves]

Don't know what some people are going on about.  Nobody made any promises as far as I can see.  There are no "weird hang-ups" unless imagined.  All I see is everyone having an opinion about what they would like, would find useful and think others would also find useful.

It's distasteful to have to read reductive b/s like that, it diminishes the contributions of others just because some disagree.

Edited October 14, 2022 by Gabriele Graves

[Love Zhaoying]

4 minutes ago, Fenix Eldritch said:

Oh my god, it just clicked with me that y'all were abbreviating LinksetData as LSD.

iknowrite? Least-Significant-Digit huh?!? Either that or the other thing!

[Kadah Coba]

19 minutes ago, Fenix Eldritch said:

That did come up during this week's server meeting (and in fact commanded a large portion of the conversation). I believe Rider said he was going to experiment in adding the feature, with a lean towards an additional set of read/write/delete functions that required an integer pin (that way you wouldn't lock out the entire store). Some suggested using a string for a passphrase, but if that happens, Rider said he'd count those against the store size.

Passphrases counting against the store was what I was expecting.

Per row locks would be ideal, so that's good to hear.