"Linkset Data" is coming.

[Coffee Pancake]

24 minutes ago, Kadah Coba said:

I use llOwnerSay and added an option to FS to route all llOwnerSay to the object's own script panel, but not the main panel too. Its a lot better than spamming everybody with debugs while also not making your local chat completely unusable. 😛

KADAH !! OMG.

[Fenix Eldritch]

33 minutes ago, Kadah Coba said:

Its a lot better than spamming everybody with debugs

I don't think that's the case anymore. I recently tested in-world on the official viewer and people nearby (also using the official viewer) didn't report seeing anything sent to DEBUG_CHANNEL from my scripted objects. Wulfie did a similar test with FS and saw the same result.

 

This was mentioned briefly at the server meeting and I don't think anyone commented on it (conversation was deep on something else at the time) :

Quote

Rider Linden: One thing that I've not seen mentioned much on the forum is llLinksetDataFindKeys()
Rider Linden: The search parameter on that accepts full regex.

But... I've been having a heck of a time getting it to work. I made a bunch of entries with keys named firstkey01, firstkey02, etc. and am struggling to get anything with a basic regex returned.

default
{
    state_entry()
    {
        llLinksetDataReset();
        
        llLinksetDataWrite("firstkey01","testdata");
        llLinksetDataWrite("firstkey02","testdata");
        llLinksetDataWrite("firstkey03","testdata");
        llLinksetDataWrite("firstkey04","testdata");
        
        llLinksetDataWrite("secondkey01","testdata");
        llLinksetDataWrite("secondkey02","testdata");
        llLinksetDataWrite("secondkey03","testdata");
        llLinksetDataWrite("secondkey04","testdata");
        
        llOwnerSay("LinksetData stats:\n"
            +(string)llLinksetDataCountKeys()+" keys used, "
            +(string)llLinksetDataAvailable()+" bytes available");
        
        //llOwnerSay("result: '"+llList2CSV(llLinksetDataListKeys(0, 10))+"'");
        
        string regex = "/key/g";
        llOwnerSay("regex result: '"+llList2CSV(llLinksetDataFindKeys(regex, 0, 10))+"'");
    }
}

Admittedly, I'm not very experienced in regex, but from the online aids I've found, a regex of "/key/g" should find all instances of the string "key" amongst the KVP key names. I've tried other examples, but am always getting an empty result. Am I missing something really basic? (probably)

[Love Zhaoying]

6 minutes ago, Fenix Eldritch said:

Admittedly, I'm not very experienced in regex, but from the online aids I've found, a regex of "/key/g" should find all instances of the string "key" amongst the KVP key names. I've tried other examples, but am always getting an empty result. Am I missing something really basic? (probably)

I despise RegEx, but this is awesome!! Thanks for the update!

[Quistess Alpha]

30 minutes ago, Fenix Eldritch said:

Admittedly, I'm not very experienced in regex, but from the online aids I've found, a regex of "/key/g" should find all instances of the string "key" amongst the KVP key names. I've tried other examples, but am always getting an empty result. Am I missing something really basic? (probably)

"/key/g" looks like a sed string, not regex.

I'm not really a regex expert either, but I'd try "*key*" to find anything with the string "key" in it.

ETA: looking at Wikipedia, if that doesn't work, try ".*key.*"

https://en.wikipedia.org/wiki/Regular_expression

Edited October 12, 2022 by Quistess Alpha

[Quistess Alpha]

25 minutes ago, Fenix Eldritch said:

Rider Linden: One thing that I've not seen mentioned much on the forum is llLinksetDataFindKeys()
Rider Linden: The search parameter on that accepts full regex.

@Rider Linden

Are there plans to add that functionality to ordinary experience-based KVP? That functionality is vital for most of the things I would want to use KVP for, and its non-existance is one of the main reasons I don't use KVP very often.

[Kadah Coba]

52 minutes ago, Coffee Pancake said:

KADAH !! OMG.

Not sure if that's a possitive response, but if ya want to use it, the setting should still be in the main prefs panel (unless FS removed it as they still love to mess with hiding prefs for some reason, lol), just search for llOwnerSay, or its still in the script settings panel, the same one where you would set your preproc include path.

33 minutes ago, Fenix Eldritch said:

I don't think that's the case anymore. I recently tested in-world on the official viewer and people nearby (also using the official viewer) didn't report seeing anything sent to DEBUG_CHANNEL from my scripted objects. Wulfie did a similar test with FS and saw the same result.

For sure it and llOwnSay spam yourself still though. llOwnerSay has less overheard and throttling, so generally I prefer it debugging.

33 minutes ago, Fenix Eldritch said:

This was mentioned briefly at the server meeting and I don't think anyone commented on it (conversation was deep on something else at the time) :

But... I've been having a heck of a time getting it to work. I made a bunch of entries with keys named firstkey01, firstkey02, etc. and am struggling to get anything with a basic regex returned.

Admittedly, I'm not very experienced in regex, but from the online aids I've found, a regex of "/key/g" should find all instances of the string "key" amongst the KVP key names. I've tried other examples, but am always getting an empty result. Am I missing something really basic? (probably)

Hold up, we have an LSL function that can use regex finally? That's a first. I wish we could get a function do regex on strings directly (I'd also love an llFormat too, just saying).

15 minutes ago, Quistess Alpha said:

"/key/g" look like a sed string, not regex.

I'm not really a regex expert either, but I'd try "*key*" to find anything with the string "key" in it.

ETA: looking at Wikipedia, if that doesn't work, try ".*key.*"

https://en.wikipedia.org/wiki/Regular_expression

That looks like a full expression, but almost nothing I have used ever needs/uses the starting / or end /g (which I think means global).

.* is greedy match any or none of any length.

If you are trying to match JUST a UUID, then try:

[a-zA-Z0-9]{8}-[a-zA-Z0-9]{4}-[a-zA-Z0-9]{4}-[a-zA-Z0-9]{4}-[a-zA-Z0-9]{12}

If there is ALWAYS extra data before or after, add .+, if there is OPTIONALLY extra data, use .*

https://regexr.com/ is good site to test expression on.

Edited October 12, 2022 by Kadah Coba
Spelling

[Fenix Eldritch]

Hmm, from the multiple examples I looked at they all just referred to it as "regex".

"*key*" and ".*key*." were also among the permutations I tried but also to no effect.Trying some of the examples on the wikipedia page too, but I'm not finding any success...

Edit: https://regexr.com/ was one of the online resources I also tried. It implicitly fills in the leading / and trailing /g, but I've tried with and without them as well. Also tried just searching for the two number characters with \d\d and while that checks out on something like regexr, I still can't get it right for llLinksetDataFindKeys...

Edited October 12, 2022 by Fenix Eldritch

[Anna Salyx]

1 hour ago, Kadah Coba said:

 

As somebody that will link different things together when building out my region... or even as someone that would make things that could be linked later to random things by end users...

How about each object has their own LSD store, which can be accessed from other links, and maybe has some optional method to restrict access, possibly via a pin. That would resolve the issue of unwanted collisions when linking things that could have incompatibilities when they are trying to share the same LSD store, and things that can support sharing can just always use LINK_ROOT and handle the merging themselves via the on link event. If there is some issue with each object being able to have an LSD store, make a limit of how many their can be per link set and either prevent linking (preferable) or block/wipe the additional ones being added.

 

I would like to echo and up-vote this.  I go out of my way sometimes, when I can, to get +MOD items to wear so I can link different parts together.  Since with rigged mesh everything 'goes to wear it belongs' when worn I can make a full outfit by linking shoes, (pants/skirts), undies (if needed), tops, gloves, and anything else i can so I just have one attach point used.  Having a way to resolve conflicts and keep from having one script when linked in from completely breaking things is something to strive for.  I've not thought deeply enough into the issue to offer up ideas on the hows, I'm just, at the moment, shining an extra light on this potential issue

(and as a side note, I'd really love if somehow linking/delinking could be decoupled from the MOD permissions.  I'd be happy even with it being linking only as in  "if an object is no-mod you can add links but not remove links and you can only link +copy items."  But this might just remain wishful dreaming.)

[Quistess Alpha]

1 hour ago, Fenix Eldritch said:

"*key*" and ".*key*." were also among the permutations I tried but also to no effect.

Just got a sec to do a sanity test in-world and it seems to work like the wikipedia overview of POSTIX regular expressions says it should:

llLinksetDataWrite("A","1234");
llLinksetDataWrite("AB","1234");
llLinksetDataWrite("CAC","1234");
llLinksetDataWrite("DAD","1234");

llOwnerSay(llList2CSV(llLinksetDataFindKeys("A",0,10)));
// returns A only
llOwnerSay(llList2CSV(llLinksetDataFindKeys(".*A.*",0,10)));
// returns all 4 keys.
llOwnerSay(llList2CSV(llLinksetDataFindKeys("A.*",0,10)));
// returns A, AB
llOwnerSay(llList2CSV(llLinksetDataFindKeys(".*D",0,10)));
// returns DAD

since "A.*" doesn't find CAC or DAD, that makes the "^" beginning of line selector redundant. In a normal regex system, I'd expect in-string searches to be the default, but it seems in this implementation, ".*" at the beginning and end is required for an in-string search.

ETA: bracket expressions and number-ranges (EX. "[A-Z]{2,3}" to match 2 or 3 capital letters) also seem to work, but back-references do not.

Edited October 13, 2022 by Quistess Alpha

[Wulfie Reanimator]

Regex expert here, I'm happy to see a little bit of it in LSL. 😊

 

1 hour ago, Fenix Eldritch said:

"*key*" and ".*key*." were also among the permutations I tried but also to no effect.

The second expression was almost correct. It should've been ".*key.*"

The dot character matches any character, and the asterisk repeats the previous token any number of times (including none). So ".*" matches anything, and the above expression will match strings like "key" or "akey" or "some keys".

 

It would be nice if the wiki had an explanation for how the input expression is used for the full expression (like whether the surrounding "/" are required or optional, flags used, etc).

Are newlines allowed for keys?

Edited October 13, 2022 by Wulfie Reanimator

[Qie Niangao]

2 hours ago, Kadah Coba said:

As somebody that will link different things together when building out my region... or even as someone that would make things that could be linked later to random things by end users...

How about each object has their own LSD store, which can be accessed from other links, and maybe has some optional method to restrict access, possibly via a pin. That would resolve the issue of unwanted collisions when linking things that could have incompatibilities when they are trying to share the same LSD store, and things that can support sharing can just always use LINK_ROOT and handle the merging themselves via the on link event. If there is some issue with each object being able to have an LSD store, make a limit of how many their can be per link set and either prevent linking (preferable) or block/wipe the additional ones being added.

Assuming you mean each link having its own store (rather than each "object"), yeah, this would be a tidy finesse. Otherwise the implementation is going to face a morass of different ways linking can screw the pooch, all to solve a "problem" that's a clean solution to a bigger problem: how to partition local KVP space in an object for use by multiple scripts.

To whatever extent scripts need their own KVP and need to access other scripts' KVPs in the same linkset, well, that's a link message away (or nearer, if scripts ever get to read KVPs in other objects/links).

I wonder if this KVP is a region-local implementation, or if it's an object-specific partitioning of the grid-wide KVP of Experiences. If the latter, then making it region-specific would be pretty artificial… although objects come and go at the region level, so that either complicates a grid-wide partitioning, or reveals that it's all really region-local.

In any case, I don't see as much need to share the event, suggested somewhere above, as there is to share access to the KVP store itself.

[Wulfie Reanimator]

3 hours ago, Kadah Coba said:

As somebody that will link different things together when building out my region... or even as someone that would make things that could be linked later to random things by end users...

2 hours ago, Anna Salyx said:

I go out of my way sometimes, when I can, to get +MOD items to wear so I can link different parts together.  Since with rigged mesh everything 'goes to wear it belongs' when worn I can make a full outfit by linking shoes, (pants/skirts), undies (if needed), tops, gloves, and anything else

Genuine question: For the things we would expect to be able to link together (like sim scenery or clothing), how often do we imagine them to be using object-shared persistent data storage, instead of in-memory storage (or something in the description)? Are key-conflicts (and loss of data) in a situation like this something that's likely to become a common problem?

Edited October 13, 2022 by Wulfie Reanimator

[Quistess Alpha]

12 minutes ago, Wulfie Reanimator said:

For the things we would expect to be able to link together (like sim scenery or clothing), how often do we imagine them to be using object-shared persistent data storage, instead of in-memory storage (or something in the description)?

The main use-case problem I'm envisioning is texture-change clothing. But, I think with the improved efficiency of utf-8 over utf-16, assuming you don't have any name-conflicts, I'd expect to be able to link 5+ really fat ones together before memory overflow, and that's assuming they don't make any effort to compress the keys (with utf-8, I'd think you could get a UUID down to 18~20 bytes? vs theoretical minimum of 16.)

(back of the envelope: 14+36 ~= 50 characters for texture name+key, *200 texture options = 10,000 bytes, or 1/6 of the local database.)

[Anna Salyx]

2 hours ago, Wulfie Reanimator said:

Genuine question: For the things we would expect to be able to link together (like sim scenery or clothing), how often do we imagine them to be using object-shared persistent data storage, instead of in-memory storage (or something in the description)? Are key-conflicts (and loss of data) in a situation like this something that's likely to become a common problem?

Honestly, at first very little.  Going forward, I can't say, but  I can see it being useful if it's adopted by some of the body makers to throw auto alpha configuration into in a KVP instead of relying on prim description or script memory to hold that detail so if people change descriptions or reset scripts that piece data is not lost.  I see a lot of use in gadgets and HUDS and the like (and that's going to be my  personal focus) but little in realm of wearable or general landscape.  But I also know that there are some very clever scripters out there who might have a clearer vision of how it can be leveraged for such items.  Bringing the idea of it up now while It's in the early days of figuring out how it's going to work lets bigger brains than mine to think ahead and decide if it might be issue enough to bother with or not.  It might not ever be deemed a big enough issue to expend resources on. But better the discussion now rather than bug reports/features requests later maybe?

[Qie Niangao]

6 hours ago, Wulfie Reanimator said:

Genuine question: For the things we would expect to be able to link together (like sim scenery or clothing), how often do we imagine them to be using object-shared persistent data storage, instead of in-memory storage (or something in the description)? Are key-conflicts (and loss of data) in a situation like this something that's likely to become a common problem?

First, persisting data in the description has always been problematic, partially because the description field (especially of root) has legitimate use even in totally unscripted products, so scripts were always intruding on that use—but worse, they were always intruding on each other when they used more than some  "fair share" of the field length or used mutually confusing expressions. So I think migration away from that will be very swift as scripts update.

Second, it's not key collisions I'm most concerned about, it's the ability to efficiently reset the KVP storage one script is using without wiping out something stored there by another unknown script. Maybe an application's entries could be selectively wiped if its keys were all prefaced by a "unique-enough" identifier found in a row-by-row table scan disguised as a regex query. But that's hideous both in space and processing efficiency.

Also, we've all been twisting ourselves in knots for years, working around memory limitations, and this is real relief. Folks are gonna use it.

Most SL products have multiple scripts from different sources, pieced together by creators who will have Trouble when those scripts start mysteriously stepping on each other months after product is released.

[Love Zhaoying]

Hmm! 2 different scripts both using linkset storage for the same linkset sounds potentially problematic. For instance, they could be unrelated scripts from different creators that use similar key structures. 

[Zi Ree]

With these things it's always good to come up with a good prefix for your keys that make them unique. So don't just use "position" as key but "myscript_position", where "myscript" can be anything that acts like a namespace for this specific script.

[Love Zhaoying]

28 minutes ago, Zi Ree said:

With these things it's always good to come up with a good prefix for your keys that make them unique. So don't just use "position" as key but "myscript_position", where "myscript" can be anything that acts like a namespace for this specific script.

That's how I'd do it, or if I have a suite of related scripts, use a prefix that the scripts hold in common.

[Qie Niangao]

Right, so for my first big LinksetData project I'll claim llChar(0xDF) —looks like "×"—as my namespace prefix. Nobody collide with it, okay? 😛 Just one extra byte per row.

And I promise never to call llLinksetDataReset(), so nobody else ever call it either, okay? 😛 Because you can never know when somebody might have your script in an object that gets linked with one that has my scripts inside.

Instead we'll all just call llLinksetDataDelete() one row at a time for keys found in batches until llLinksetDataFindKeys() gets no results for our personal namespace prefix. Because we'll all just agree to do that, okay?

(Am I the only one who sees a problem here?)

 

[Love Zhaoying]

3 minutes ago, Qie Niangao said:

(Am I the only one who sees a problem here?)

Not really - if someone writes a script that delete other's rows - or uses a "somewhat non-unique prefix" - it is on that scripter.  

Unless you are suggesting, by way of slight irony / sarcasm, that no prefix is needed at all for our keys? Go for it!

Personally, if someone puts other people's scripts in the same objects with my scripts, then I am not responsible for the results. Their scripts could start/stop my scripts at random, steal all the script time, send random linked messages that trigger my scripts, etc. etc. etc.! (Let alone garbage up / mangle / delete my linkset data.)  Heck, their scripts could even *gasp* change the object description, which would kill any "old" scripts that depend on it. Buyer Beware!

[Wulfie Reanimator]

14 minutes ago, Love Zhaoying said:

Personally, if someone puts other people's scripts in the same objects with my scripts, then I am not responsible for the results. Their scripts could start/stop my scripts at random, steal all the script time, send random linked messages that trigger my scripts, etc. etc. etc.! (Let alone garbage up / mangle / delete my linkset data.)  Heck, their scripts could even *gasp* change the object description, which would kill any "old" scripts that depend on it. Buyer Beware!

This is kind of what I was thinking while I was typing my previous question. Usually linked linksets don't play nicely if they're scripted, because the script usually makes some assumptions about the object it's in (like being in the root...).

I would think anybody who is used to linking stuff together is aware that these issues can happen. That's why we have copies (and get automatic copies when rezzing stuff anyway).

Edited October 13, 2022 by Wulfie Reanimator

[Love Zhaoying]

6 minutes ago, Wulfie Reanimator said:

This is kind of what I was thinking while I was typing my previous question. Usually linked linksets don't play nicely if they're scripted, because the script usually makes some assumptions about the object it's in (like being in the root...).

I would think anybody who is used to linking stuff together is aware that these issues can happen. That's why we have copies (and get automatic copies when rezzing stuff anyway).

Oooh! That brings up a new one! 

Customer: "All my data is gone!"

Support: "Did you happen to rez a copy or use a redelivery?"

[Qie Niangao]

Okay, maybe I'm trying to over-engineer this.

Still, though, there's this innocent-looking library function, llLinksetDataReset(), that will unwittingly annihilate any other linkset data that happens to be used by a script anywhere in the linkset. (Even llResetOtherScript() is scoped to the individual link.)

On the plus side, the victim script can detect that an unexpected linkset_data event transpired and try to recover until it happens again.

[Love Zhaoying]

13 minutes ago, Qie Niangao said:

Okay, maybe I'm trying to over-engineer this.

Still, though, there's this innocent-looking library function, llLinksetDataReset(), that will unwittingly annihilate any other linkset data that happens to be used by a script anywhere in the linkset. (Even llResetOtherScript() is scoped to the individual link.)

On the plus side, the victim script can detect that an unexpected linkset_data event transpired and try to recover until it happens again.

That would be a nice function to put in a "script bomb", give it to your enemies to put in their important objects.

Otherwise, prolly don't call it.

[Frionil Fang]

This is strictly my personal feeling on linkset data:

Trying to safeguard against what every other user/scripter might do seems like an unreasonable expectation, and from my point of view linkset data is meant to be a "sane", spacious replacement for hacks like storing data in description/other string fields of an object, not a secure remote server replacement.

Maybe it should be thought as more of a cache rather than a file, it could be invalidated at any time.

Personally, I'll be very happy to have what's basically a free 64 kB of data storage for my projects, since they tend to be single-script and not expected to allow linking to random things.

Edited to add: I understand that now would be a good time to expand the spec to have more than just the "sane bigger replacement", but the extra complexity... would we ever actually get it then?

Edited October 13, 2022 by Frionil Fang