"Linkset Data" is coming.

[Jenna Felton]

I think the feature as it is will cover a wide range of applications. Now when it is released we will be able to write the applications and then when we get idea for applications which the LSD store in this definition can not or can hardly support, feature requests for enchanting it will follow.

For example I can imagine a function that would wipe out a bunch of keys, protected with the same password. This way a plugin script which is leaving the build would not remove each key separately and overflow the event queue. This can be added after the release.

Maybe I am wrong but I think the project is in a phase when it would only accept the last minute changes such that without them the LSD store definition would break future applications. But maybe I just wish it life before Christmas

Edited October 18, 2022 by Jenna Felton

[Kadah Coba]

6 hours ago, Wulfie Reanimator said:

It wouldn't be ideal for scripts to be able to hog the entire data store, either maliciously or by accident (such as changing and/or forgetting what password you used, or accidentally creating protected keys and forgetting to remove them). The protected keys shouldn't be permanent, so a function like llLinksetDataReset should be able to get rid of all data.

The protected write/delete functions have their use for managing individual keys. 🙂

It should also be noted that making a new copy doesn't get rid of any keys, those get copied too.

If I have an mod object I want to reuse without its scripts, it would be nice if part of the LSD store isn't locked off from an old script. And if there is some other function like llLinksetDataResetButAlsoLocked(), then that function will probably just get used instead of llLinksetDataReset() and its the same problem.

5 hours ago, Lucia Nightfire said:

This allows a mod product's end-user to add their own links/scripts/functionality with their own storage with their own passwords and keep existing scripts "happy" while also preventing existing scripts from accidentally/intentionally breaking things without having to go "all-in" and reset data.

Granted, an application's protocol could have still been set up to wipe all data naturally/occasionally and without being nefarious or to satisfy a previous owner's control.

It's like multiple people living in a dorm, respecting each other's space and property and not kicking each other or their property out. Coexistence.

It's also like the "Good fences make good neighbors." proverb.

This is the same type of thing as the existing issue with scripts with incomparable link messages.

"But just put those scripts in another link", sure as long as they use link_this, but then the LSD store conflicts could also be avoided by child links being able to have their own stores.

[Zi Ree]

On 10/13/2022 at 6:30 PM, Rider Linden said:

It's a table scan. But the table is generally small enough (few thousand records?) that it shouldn't be too large a hit.

 

@Rider Linden Is there a chance to get Search and Regex also for Experience KVP storage? It would make things SO much better! See also: https://community.secondlife.com/forums/topic/492023-linkset-data-is-coming/page/4/#comment-2520025

 

Edited October 19, 2022 by Zi Ree
typo

[Rider Linden]

14 hours ago, Zi Ree said:

 

@Rider Linden Is there a chance to get Search and Regex also for Experience KVP storage? It would make things SO much better! See also: https://community.secondlife.com/forums/topic/492023-linkset-data-is-coming/page/4/#comment-2520025

 

Zi, please open a feature request JIRA, we're doing some work in the the area of Experience KVP soon.

[Zi Ree]

Any chance to get:

list llLinksetDataFindData( string regex, integer first, integer count );

to search for data?

 

[Zi Ree]

25 minutes ago, Rider Linden said:

Zi, please open a feature request JIRA, we're doing some work in the the area of Experience KVP soon.

Done! Thank you!

https://jira.secondlife.com/browse/BUG-232779

[Kyrah Abattoir]

I'm still not sure on having delete protection on key value pairs, If my scripts are trying to clear datastore space by removing crud that it can't identify, it shouldn't require wiping its own records in the process.

[Lucia Nightfire]

4 hours ago, Kyrah Abattoir said:

I'm still not sure on having delete protection on key value pairs, If my scripts are trying to clear datastore space by removing crud that it can't identify, it shouldn't require wiping its own records in the process.

I think the argument is too one-sided.

Would you, in-turn, want other creator scripts you chose to leave intact in your objects or were unaware that existed in your objects to also delete your own protected keys as well?

There are two scenarios where this will typically be an issue:

1. Other-creator objects already containing other-creator scripts being linked together and said scripts being maintained for the sake of functionality/convenience or because the object owner lacks the skill/knowledge to achieve the same functionality with their own scripts.
2. Users directly placing other-creator, no-mod scripts in their content for the sake of functionality/convenience or because the object owner lacks the skill/knowledge to achieve the same functionality with their own scripts.

Without including delete protection, the protected keys feature is incomplete and would only lead to an increase in no-mod content, in-turn, making protected keys redundant.

There is no 100% best way to deal with it. The nuclear option of wiping all data is a "hopeful" deterrent to allow for good actors to work together in the same environment.

[Kadah Coba]

If we're gonna have delete protect on locked, can we have optional protect on non-locked? I'm seeing a use-case where LSD is used for cache and reset is used to quickly clear all the keys in the cache portion. That would be possible with locks though, would just have to document the pass somewhere for keys to function as public. Not sure if all the search functions (can) work on locked keys however, as that could be a hindrance.

I'd still enjoy LSD stores on child links, not just the root, to completely side-step the merge and conflict issues. But that's me and I'll probably use that for more novel uses, like massive L2 cache. x3

 

Gonna keep hoping for remote (same sim) sync read access. The immediate use-case we have is allowing for greater flexibility for low-latency interaction in games.

Right now we use object descriptions, which is painfully limited to 127 bytes. The main thing is that this does not require a back and forth comm between the object and HUD, nor using sensors in the game world objects. Many objects types don't need any scripts for this to work, just the correctly set desc string, and the ones that do, don't need to have a lot of duplicate game logic.

The HUD does 2 short ray traces and if the object is different from last, reads the description. A down trace sees what the avatar is standing on, this is for stuff like footstep sounds based on ground type.

A trace along the mouselook camera forward vector gets what the being directly looked at for a wide array of custom interactions. Things like "look at ladder, press action key to grab, use ladder like every video game since there have been ladders in first person games" and the ladder can be scriptless, so adding more only has LI cost.

Yeah... you could somethings with touch and/or collision instead, but then every obj needs logic to handle when its ok to be touched, or the player HUDs have to deal with even more incoming comms processing and filter out stuff like when a player is trying to shoot/cast but their cursor was over a touchable obj across a room. Collision is good for game event triggers, using it for the things like footstep sounds would make it not worth doing. For the most part, using chat comms for everything adds a lot of latency and failure points (ie. 2 generals problem, etc), there's a lot of cases where only one side needs to know or do anything and having to needless make that a two-way conversion handshake is painful.

Video demoing some of this. That game is NSFW, though there is nothing above PG in the video.

 

[Jym Nova]

2 hours ago, Kadah Coba said:

If we're gonna have delete protect on locked, can we have optional protect on non-locked? I'm seeing a use-case where LSD is used for cache and reset is used to quickly clear all the keys in the cache portion. That would be possible with locks though, would just have to document the pass somewhere for keys to function as public. Not sure if all the search functions (can) work on locked keys however, as that could be a hindrance.

For a use case such as cache, you could simply just store those values like normal, ( llLinksetDataWrite ), and for your functional values use protected ( llLinksetDataWriteProtected ). Then just run a loop to clear the "cache".

clearCache(){
    list my_list = llLinksetDataListKeys(0,-1);
    integer i = 0;
    for(i; i < llGetListLength(my_list); i++;){
        llLinksetDataDelete(llList2String(my_list,-1)); //using "-1" for hopefully obvious reasons
    }
}

It's not perfect, but it'll get the job done.

(Of course I would hope that if protected keys are returned they would silently fail with this example to not stall the script)

Edited October 21, 2022 by Jym Nova

[Jenwen Walpole]

i am trying to test the LinksetData fuctions in the beta grid, but it seems thee fuctions are unrecognised and will not save.

do i need a beta viewer or soming to test this?

Edited October 21, 2022 by Jenwen Walpole

[Fenix Eldritch]

Currently it's only usable on the handful of beta grid regions that are running the appropriate server version:

• Blake Sea - Arabian
• Blake Sea - Atlantic
• Blake Sea - Beagle
• Blake Sea - Binnacle
• Blake Sea - Black
• Gothlauth
• Jigglypuff
• Mauve
• Moonberry
• Sapas
• Smithereens

Edited October 21, 2022 by Fenix Eldritch

[Rider Linden]

2 minutes ago, Jenwen Walpole said:

i am trying to test the LinksetData fuctions in the beta grid, but it seems thee fuctions are unrecognised and will not save.

do i need a beta viewer or soming to test this?

It is only active on a few of the beta grid regions.  Try testing on either [Mauve] or [Jigglypuff]

[Jym Nova]

6 minutes ago, Jenwen Walpole said:

i am trying to test the LinksetData fuctions in the beta grid, but it seems thee fuctions are unrecognised and will not save.

do i need a beta viewer or soming to test this?

It's only live on certain regions:

Aditi Mainland regions:

• Blake Sea - Arabian
• Blake Sea - Atlantic
• Blake Sea - Beagle
• Blake Sea - Binnacle
• Blake Sea - Black
• Gothlauth
• Jigglypuff
• Mauve
• Moonberry
• Sapas
• Smithereens

[Jenwen Walpole]

awsum thanks!

[Jenwen Walpole]

wow ... it is a LOT faster then i expected.

back of the matchbook testing ot me 1,000,000 copies of a 10 char linkset data to a local var in 0.085416 milliseconds and just for fun, becase it was so fast i did the same copy but from global to local var in 0.000156 milliseconds just to see the comparison.  ymmv

This is going to repalce a lotta link message stuff i am doing and config file handling.  PLUS the extra fuctionality of persistant data and more var space. .. my head is spinning wiht ideas

Edited October 21, 2022 by Jenwen Walpole

[Kadah Coba]

9 hours ago, Jym Nova said:

For a use case such as cache, you could simply just store those values like normal, ( llLinksetDataWrite ), and for your functional values use protected ( llLinksetDataWriteProtected ). Then just run a loop to clear the "cache".

clearCache(){
    list my_list = llLinksetDataListKeys(0,-1);
    integer i = 0;
    for(i; i < llGetListLength(my_list); i++;){
        llLinksetDataDelete(llList2String(my_list,-1)); //using "-1" for hopefully obvious reasons
    }
}

It's not perfect, but it'll get the job done.

(Of course I would hope that if protected keys are returned they would silently fail with this example to not stall the script)

Ignoring that my_list could easily be larger than the current free stack, for stuff I might do, a for loop of 10k+ iterations would not be ideal. 

Locked with public key would work though.

4 hours ago, Jenwen Walpole said:

wow ... it is a LOT faster then i expected.

back of the matchbook testing ot me 1,000,000 copies of a 10 char linkset data to a local var in 0.085416 milliseconds and just for fun, becase it was so fast i did the same copy but from global to local var in 0.000156 milliseconds just to see the comparison.  ymmv

This is going to repalce a lotta link message stuff i am doing and config file handling.  PLUS the extra fuctionality of persistant data and more var space. .. my head is spinning wiht ideas

I did some benchmarks on earlier pages, though I think I may have only posted the chart some of that on the Discord.

Value length in bytes (from 8 to 1024) verses write+read cycles per second. Each point measured 2,000 iterations. Consistently there is was a odd-even pattern (might be the regular LSL time share kicking in), so for 2 of the 4 runs, did an extra 8byte run at the start (that was discarded) to see if the odd-even pattern was unrelated to the value size, which it was.

I did not explicitly check (yet), but this might be pretty close to using lists.

[Jenwen Walpole]

great work kahda,

one of the things i am curious about is if the amount of space  effects performance at all, though i suspect it will have no impact.  I mgiht check that anyway.

i also noticed perormance variations run over run. when doing them conscutivly .. i just asumed that it was the luck of the draw on the slicing and other thigns takign up scipt time.

I see the iops go down as expected as the data size goes up..but the bps goes up.  it be interesting to graph for the bytes per sec or kbps.  I .just did a rough calcualtion for  and if my calulations are right you were about 10x higher data thoughput at 1024 over 8 bits, even though the number of operations/itterations droped to 1/10th

 

 

Edited October 22, 2022 by Jenwen Walpole

[Kadah Coba]

2 hours ago, Jenwen Walpole said:

great work kahda,

one of the things i am curious about is if the amount of space  effects performance at all, though i suspect it will have no impact.  I mgiht check that anyway.

i also noticed perormance variations run over run. when doing them conscutivly .. i just asumed that it was the luck of the draw on the slicing and other thigns takign up scipt time.

I see the iops go down as expected as the data size goes up..but the bps goes up.  it be interesting to graph for the bytes per sec or kbps.  I .just did a rough calcualtion for  and if my calulations are right you were about 10x higher data thoughput at 1024 over 8 bits, even though the number of operations/itterations droped to 1/10th

 

 

I would think worst case impact would be less than one additional idle script.

That very likely just script-time variance. I ran more and longer ones and the averages were pretty much in the same ranges.

I have no idea what the internal structure of the store is like, but there seems like there may be some sort of difference after/around 64 bytes. There's going to be many trade-offs to using longer single values: more storage for values, higher access times, more script mem, more script time in parsing/serializing, etc.

Any case, still far less punishing/limited that Exp KVP. LSD has far more usage potential than just a local KVP store, it can pretty much function as a list of strings on its own stack.

[Jym Nova]

12 hours ago, Kadah Coba said:

Ignoring that my_list could easily be larger than the current free stack, for stuff I might do, a for loop of 10k+ iterations would not be ideal. 

True, just adapt it:

instead of grabbing them all
list my_list = llLinksetDataListKeys(0,-1);
grab segments:
list my_list = llLinksetDataListKeys(0,20);
//llLinksetDataListKeys(integer first, integer count);

[Casper Warden]

This is an absolutely outstanding feature, and something that has been long overdue. CasperVend and CasperLet will be updated to make use of the feature on launch - no longer any need to store keys in the description of the object

[Casper Warden]

Hmm.. that is, if it survives a shift + drag copy...

[Fenix Eldritch]

9 minutes ago, Casper Warden said:

Hmm.. that is, if it survives a shift + drag copy...

It does, as per the wiki:

Quote

This data is attached to the root prim in the linkset and survives transfer, script reset and object copying, it does not count against the memory usage of any script.

Note that the data does not survive moving the object into a region running a server version that doesn't yet support the feature. In that scenario the data will be lost and does not return even if you move the object back into a supporting region. This won't be an issue once the feature is fully deployed across the grid but until then, be wary of rezzing or moving objects with LSD in unsupported regions.

[Tapple Gao]

On 10/15/2022 at 9:43 AM, Tapple Gao said:

I haven't seen it mentioned in the thread already. But, if I have a lot of keys, script reset seems like it would DEFINITELY overflow the 64-buffered event queue with the linkset_data event. I plan to have scripts that sets around 200 keys during a single script time slice, either on reset, or on finishing a notecard read. I ran into this problem with update-notification linked messages already, until I reworked them to support batches of updates in one message.

@Rider LindenI moved this batch notification proposal to https://jira.secondlife.com/browse/BUG-232792

Edited October 22, 2022 by Tapple Gao

[Jenwen Walpole]

 

1 hour ago, Casper Warden said:

Hmm.. that is, if it survives a shift + drag copy...

i've tested this and I can confirm that the data does survive a shift + drag copy