"Linkset Data" is coming.

[Kadah Coba]

47 minutes ago, Rider Linden said:

Protecting a key incurs a 32byte overhead in addition to the value and key length. 

That is a reasonable trade-off. Awesome

[Lucia Nightfire]

I somehow caused Mauve to crash messing around with an empty pass string.

IDK what I was doing.

WIll look into it again later.

Too tired.

😴🛌💤

[Quistess Alpha]

5 hours ago, Rider Linden said:

Documentation for the LSL functions is up on the wiki but not yet linked to the main functions page. You can find it here:

https://wiki.secondlife.com/wiki/Category:LSL_LinksetData

Could you add a caveat to llLinksetDataListKeys that it doesn't default to in-string searches (search for "A" doesn't find "CAT") or, more preferably, modify the functionality such that it ~Does find in-string matches without specific qualification? (I would like ".*A.*" and "A" to find the same set of keys. "^A$" is sufficient to find exactly the key "A" if it exists.)

Edited October 15, 2022 by Quistess Alpha

[Wulfie Reanimator]

4 hours ago, Quistess Alpha said:

I would like ".*A.*" and "A" to find the same set of keys. "^A$" is sufficient to find exactly the key "A" if it exists.

Personally I don't like the idea of having to turn all of my searches into...
llLinksetDataFindKeys("^" + key + "$", 0, 1);
...just in case I or somebody else down the line adds a similar key in there.

It seems more intuitive to me that the expected behavior is an exact match, not "this might match something else." (And yes I understand the in-string search behavior is a thing, but I'm not sure we have the same context/intent here.)

Edit: I take it back, already forgot about the Read function. 😅

Edited October 15, 2022 by Wulfie Reanimator

[Quistess Alpha]

1 hour ago, Wulfie Reanimator said:

Personally I don't like the idea of having to turn all of my searches into...
llLinksetDataFindKeys("^" + key + "$", 0, 1);
...just in case I or somebody else down the line adds a similar key in there.

If you expect a single key to maybe have data, couldn't you just use llLinksetDataRead() ?

If I'm searching for something, I would expect the possibility of multiple matches.

Edited October 15, 2022 by Quistess Alpha

[Kadah Coba]

Did some more testing, here's what Unicode doesn't work since I'm guessing there's at least a few here who are gonna try doing something silly at some point.

U+0: Null. Cannot be in LSL strings, ever.

U+25C8 and up don't work in keys. That's an odd point for it to stop, mid way in to the Geometric Shapes block.

U+FFFE and U+FFFEF both return U+FFFD, which you shouldn't use either (is the "replacement character"). Generally just don't use the whole Specials block (U+FFF0..U+FFFF)

U+D800..U+DFFF are the high and low surrogates. Its UTF-8, you can't use those values on their own. Doesn't work in LSL strings anyway.

U+110000 and up don't work in LSL strings. Unicode reasons.

If you're going to try to encode data to save space, you're better off using BASE64 for speed or BASE127 for maximum packing. Higher bases will actually use more space due to UTF-8.

So if you need a special character for key prefixes or whatever, it appears anything from U+1 to U+25C7 are usable.

 

I do love the speed of LSD. Its nice to have something new that will be useful but has is also not been kneecapped with highly restrictive bottlenecking.

Exp KVPs could have been handy, but between the land scope requirements and async, it really makes it hard to find applications for it. Despite having access to 4 or 5 exp keys, I have yet to have some use for exp KVP. When the land scope isn't much of a problem, the async then makes http an option to consider. x3

2 hours ago, Quistess Alpha said:

If you expect a single key to maybe have data, couldn't you just use llLinksetDataRead() ?

If I'm searching for something, I would expect the possibility of multiple matches.

I was gonna say. There seems like there is zero utility in doing searches for an exact key match when you can just try reading the key directly.

[Tapple Gao]

I haven't seen it mentioned in the thread already. But, if I have a lot of keys, script reset seems like it would DEFINITELY overflow the 64-buffered event queue with the linkset_data event. I plan to have scripts that sets around 200 keys during a single script time slice, either on reset, or on finishing a notecard read. I ran into this problem with update-notification linked messages already, until I reworked them to support batches of updates in one message.

This would lead to some desync issues when another script needed to react to a value change, but, the queue started dropping events. It could be alleviated with 1 or 2 functions, and 1 new action type to linkset_data:

• llLinksetDataBatchBegin()
• llLinksetDataBatchEnd()

or

• llLinksetDataBatch(integer bool)

this would start/stop supressing linkset_data events. Turning batch mode off would trigger linkset_data(LINKSETDATA_BATCH_UPDATE) in all scripts, that signals them to rescan what keys they must react to

For safety, batch mode should maybe end automatically if it's still active when a script finishes it's time slice. Ending batch mode when it's already over should probably be a no-op

Edited October 15, 2022 by Tapple Gao

[Tapple Gao]

I think reading a key that doesn't exist currently causes a sim crash. put this script in a prim and click it to crash Mauve:

default {
    touch_start(integer total_number) {
        integer count = (integer)llLinksetDataRead("count");
        llLinksetDataWrite("count", (string)(count+1));
    }
}

 

[itoibo]

Wow nice find! We finally have some persistent storage!

[VirgyStella]

On 10/9/2022 at 11:54 AM, Lucia Nightfire said:

So something I never expected would come is coming soon.

"Linkset Data"

(...)

Thoughts?

Just one tough: FINALLY!

I can't count how much time I desired something like this!

[Lucia Nightfire]

1 hour ago, Tapple Gao said:

I haven't seen it mentioned in the thread already. But, if I have a lot of keys, script reset seems like it would DEFINITELY overflow the 64-buffered event queue with the linkset_data event. I plan to have scripts that sets around 200 keys during a single script time slice, either on reset, or on finishing a notecard read. I ran into this problem with update-notification linked messages already, until I reworked them to support batches of updates in one message.

This would lead to some desync issues when another script needed to react to a value change, but, the queue started dropping events. It could be alleviated with 1 or 2 functions, and 1 new action type to linkset_data:

• llLinksetDataBatchBegin()
• llLinksetDataBatchEnd()

or

• llLinksetDataBatch(integer bool)

this would start/stop supressing linkset_data events. Turning batch mode off would trigger linkset_data(LINKSETDATA_BATCH_UPDATE) in all scripts, that signals them to rescan what keys they must react to

For safety, batch mode should maybe end automatically if it's still active when a script finishes it's time slice. Ending batch mode when it's already over should probably be a no-op

FWIW, usage of the linkset_data() event is not required.

[Tapple Gao]

39 minutes ago, Lucia Nightfire said:

FWIW, usage of the linkset_data() event is not required.

no. but it would almost entirely get rid of my need for link messages and parsing in my biggest script, vastly simplifying it

Edited October 15, 2022 by Tapple Gao

[Tapple Gao]

1 hour ago, Tapple Gao said:

I haven't seen it mentioned in the thread already. But, if I have a lot of keys, script reset seems like it would DEFINITELY overflow the 64-buffered event queue with the linkset_data event. I plan to have scripts that sets around 200 keys during a single script time slice, either on reset, or on finishing a notecard read. I ran into this problem with update-notification linked messages already, until I reworked them to support batches of updates in one message.

This would lead to some desync issues when another script needed to react to a value change, but, the queue started dropping events. It could be alleviated with 1 or 2 functions, and 1 new action type to linkset_data:

• llLinksetDataBatchBegin()
• llLinksetDataBatchEnd()

or

• llLinksetDataBatch(integer bool)

this would start/stop supressing linkset_data events. Turning batch mode off would trigger linkset_data(LINKSETDATA_BATCH_UPDATE) in all scripts, that signals them to rescan what keys they must react to

For safety, batch mode should maybe end automatically if it's still active when a script finishes it's time slice. Ending batch mode when it's already over should probably be a no-op

Just tested. It's trivial to fill up the event queue with linkset_data events:

integer count = 0;

default {
    state_entry() {
        integer i;
        for (i = 1; i <= 200; i++) {
            llLinksetDataWrite("a", (string)i);
        }
        llMessageLinked(LINK_SET, 0, "Nobody will ever see this", "");
    }

//*
    linkset_data(integer action, string k, string v) {
        ++count;
        llSetText((string)count + ": " + v, <1, 1, 0>, 1);
    }
//*/

    link_message(integer sender, integer i, string msg, key id) {
        llOwnerSay(msg);
    }
}

The link message is only received if I comment out linkset_data event

Edited October 15, 2022 by Tapple Gao

[Fenix Eldritch]

I observed that the three new protected functions can all work with non-protected keys if the password parameter is empty - in which case they essentially function like the original non-protected variants.

So if for example you try to create a new key with llLinksetDataWriteProtected, and leave the password field blank, you just get a regular kvp.

You cannot update/convert an existing non-protected key to be protected or vice-versa (obviously). Nor can you change a protected key's password.

Attempting to read a key that does not exist (protected or otherwise) will crash the region - as others have noted already.

[Jenna Felton]

18 hours ago, Rider Linden said:

llLinksetDataWriteProtected(string name, string value, string pass);
llLinksetDataReadProtected(string name, string pass);
llLinksetDataDeleteProtected(string name, string pass);

Thank you Rider, having protected keys is cool, but I'd like to have a way to get access to protected keys to some degree without having a password. Because there are applications extensible via plugins, be it scripts installed into the application or (never seen but possible) linked to core prim. The core script and the plugin script can be by different creators.

Now the core script may want to maintain some keywords while plugin scripts would have only read access to them unless the developer or the core script made the write or deletion password known to the plugin script.

I would suggest to create protected key like this:

llLinksetDataCreateProtected(string name, string pass, integer level);

while level defines if the key is read protected, write protected, delete protected.

• Write protection inherits read protection (read protected key needs password to be changed)
• Delete protection inherits read and write protection.
• Write protected keys can be read without password but not changed
• Delete protected keys can be read and changed without password but not deleted.

Edit

Better to separate creation of the key and actual writing because this will avoid to change the protection level every time you change the content. After the key is created it can be changed via llLinksetDataWriteProtected as defined.

 

 

 

Edited October 15, 2022 by Jenna Felton

[Jenna Felton]

On 10/14/2022 at 6:43 PM, Jenna Felton said:

I have created a test script which allows to reset the LSD, store and read keys of it, and it remembers the number of keys it has stored. My Test Plan which I'd like to see to pass at every point is this:

...

 Will continue with the test tomorrow.

I've tested out my 'crash plan', all points passed. I tried some of the forced errors (Disconnection, llError, Bad Memory, Driver Crash) gave up testing all because same result. Crashing as soon as possible after adding keys to the LSD store or after teleport did not lose the content of the store or the new keywords. However, I could not force a crash which would lose changes to the script itself, and thus can not say if the LSD store will be more persistent than the linkset and scripts in there. But I have big hopes about this.

Edited October 15, 2022 by Jenna Felton

[Jenna Felton]

23 hours ago, Fenix Eldritch said:

I suppose in a way it makes sense to have the datastore be lost if you move into a region that doesn't support it. The region has no understanding of the feature so not only can it not access it, but when you move to the next region it's not going to be to serialize it back up to send along to the destination. Even if the new destination does support the feature, as soon as you arrive in the non supporting region, the association is lost.

I was actually expecting that when the server finds some data of an object it can not handle, this data remains attached to the object and is passed over to the next region because it probably can handle the data.

This way when LL deploys e.g. a server that for some reason forgot how to handle materials, walking into this region, while wearing a super expensive super rare gacha suit full of materials will not loose the materials, you can safely visit the region. But would the server drop the unknown properties, your suit is done.

Loosing the unknown object properties brings extra pressure on LL to not deploy servers to Agni before all bugs that may destroy worn asset are fixed. Actually such bugs should not deployed to Agni anyways but they can happen. Passing over unknown object properties, like the LSD store, would reduce the danger of these bugs.

[Jym Nova]

20 hours ago, Rider Linden said:

Documentation for the LSL functions is up on the wiki but not yet linked to the main functions page. You can find it here:

https://wiki.secondlife.com/wiki/Category:LSL_LinksetData

On this roll I've added three new functions

llLinksetDataWriteProtected(string name, string value, string pass);
llLinksetDataReadProtected(string name, string pass);
llLinksetDataDeleteProtected(string name, string pass);

These functions create "protected" keys in the datastore. In order to update, read or delete a protected key/value pair you must provide the same pass string that you supplied when it was originally created. Protecting a key incurs a 32byte overhead in addition to the value and key length. 

Writing to a protected key without the correct pass string will fail and return LINKSETDATA_EPROTECT.  Reading a protected key without the pass string returns an empty string, and attempting to delete one fails silently. 

In the linkset_data event when writing a protected key, the value parameter will always be an empty string. 

Protected keys are removed by a call to llLinksetDataReset.

This is great to an extent, I was going to say, I read through the first 4-5 pages and didn't see my concern(s).
Of course I'm not seeing the details of the 3 new added functions either.

I strongly disagree with "Protected keys are removed by a call to llLinksetDataReset."
Maybe negate protected keys & values by adding: llLinksetDataResetProtected(string pass);

#1 is Security, should I decide to distribute a "modifiable" object, anyone can read the data.
`llLinksetDataRead` which appears you have managed a solution: `llLinksetDataReadProtected`
What about: llLinksetDataListKeys? are protected keys negated or still fed in to the list?
> Not good if the data is meant to be private.
(I foresee this being great for HUD's though, since they will almost always be no modify)
(I don't care if they want to count how many keys I have or know how much data is available)

#2 is Persistence, should I decide to distribute a "modifiable" object, anyone can wipe the data.
> Not good for modifiable objects if you want persistent data.
`llLinksetDataReset` currently would wipe all data, even the protected keys.

Edited October 15, 2022 by Jym Nova

[Jym Nova]

5 hours ago, itoibo said:

Wow nice find! We finally have some persistent storage!

Not 100% unless your "object/linkset" is no modify for next owner.

[Fenix Eldritch]

18 minutes ago, Jym Nova said:

What about: llLinksetDataListKeys? are protected keys negated or still fed in to the list?
> Not good if the data is meant to be private.

Yes, llLinksetDataListKeys and llLinksetDataFindKeys will return all key names within the supplied range/regex - even protected keys. However the data itself will remain private because in order to do anything else to it (read, write, update, individually delete) you need to supply the password.

25 minutes ago, Jym Nova said:

I strongly disagree with "Protected keys are removed by a call to llLinksetDataReset."
Maybe negate protected values and add: llLinksetDataResetProtected(string pass);

I must disagree. We need to have a way to scrub data stores in the event they become full of stale entries for which the current users or scripts don't have the passwords for anymore. I believe llLinksetDataReset is purposefully destructive as a means to get a clean slate.

[Jym Nova]

13 minutes ago, Fenix Eldritch said:

I must disagree. We need to have a way to scrub data stores in the event they become full of stale entries for which the current users or scripts don't have the passwords for anymore. I believe llLinksetDataReset is purposefully destructive as a means to get a clean slate.

Don't use a password you're going to forget....
(you're probably going to have a variable in your script with the password anyway, what's the harm?)

Or maybe some other alternative that disallows "end-users" to possibly corrupt a modifiable object if they decide to add a script with llLinksetDataReset.

like: llLinksetDataResetProtected(llGetInventoryCreator(llGetScriptName()));

That's a terrible example, but something that won't let an end-user wipe your data.

I still stand by my original suggestion: llLinksetDataResetProtected(string pass);

Edited October 15, 2022 by Jym Nova

[Fenix Eldritch]

2 minutes ago, Jym Nova said:

Don't use a password you're going to forget....

That's missing my point.

I chose the word "stale" deliberately. Stale in the sense that the entries no longer serve a purpose for the current scripts contained within the linkset. It is entirely possible to have a scenario where a person buys an object and then removes the scripts. If they put their own scripts into it for a different purpose, then whatever residual linkset data that was there prior is now taking up valuable space. We need a way to remove everything.

[Jym Nova]

2 minutes ago, Fenix Eldritch said:

That's missing my point.

I chose the word "stale" deliberately. Stale in the sense that the entries no longer serve a purpose for the current scripts contained within the linkset. It is entirely possible to have a scenario where a person buys an object and then removes the scripts. If they put their own scripts into it for a different purpose, then whatever residual linkset data that was there prior is now taking up valuable space. We need a way to remove everything.

And you're missing my point:
We also need a way to protect everything.

Edited October 15, 2022 by Jym Nova

[Jym Nova]

5 minutes ago, Fenix Eldritch said:

That's missing my point.

I do get your point, mostly, however; if your use case would allow for stale kvp to be removed then you'd also have no reason to use:

llLinksetDataWriteProtected(string name, string value, string pass);
llLinksetDataReadProtected(string name, string pass);
llLinksetDataDeleteProtected(string name, string pass);

[Fenix Eldritch]

I don't think I agree with that assessment. If I make a scripted object that uses sensitive linksetdata, of course I will use the protected functions. This prevents the data from being read, updated, or casually deleted by other scripts. That gives me a basic safety net and if nothing else, ensures the data stays private. But if my script is completely removed form the object, I see no reason to keep my old sensitive kvps around. In that case it's doing nothing but taking up space.

My belief is that if it's a modify-enable object, then the end user should be able to reset the store - even to my own scripts' detriment. Because we don't know if the user intends to even keep my script.

If I didn't want end users resetting the data store, I'd make the object no-modify.

Edited to add: I think my main concern here is the prospect of a datastore getting filled with obsolete junk entries from previous owners over time. Because if we can't remove them when they are genuinely no longer need, the store will be rendered unusable for that item.

Edited October 15, 2022 by Fenix Eldritch
additional thought