"Linkset Data" is coming.

[Love Zhaoying]

The way I would handle the llLSDChaos() ("OMG this resulted in callng the LSD change events every time!1!!") scenario is:

1. Before calling that new LSD function that triggers LSD change events you don't want, set a "Semaphore"; a variable for all scripts that says "Ignore LSD Events until you hear otherwise".

- This could be done through by setting an LSD value "flag" that you DO want your scripts to see and react to.  

- After detecting this Semaphore "flag" is "set", all LSD change events should immediately exit until they detect it is "cleared".

2. After calling the new llLSDChaos() function is complete, set the same Semaphore to "all clear".

- This would cause all of the LSD change events to fire as with any other LSD change.

- The events would detect the "Semaphore is clear" and then stop ignoring LSD change events.

 

[Bleuhazenfurfle]

On 7/15/2023 at 3:29 PM, Bleuhazenfurfle said:

… and perhaps using a handle (I'd think an integer, llListen style, rather than a key) to link the two.  There is existing precedent for returning several values as CSV, and returning a handle gets around having to pass the regex itself in the event (which won't work with CSV — the next option would be JSON, but that's horrible and a handle should do fine).

 I realise I'd missed explaining something there.  What other scripts see.  The handle is great within the script doing the deletion, but not so much use to any of the others.  However, fitting the regex in there without a new event is going to be ugly (there are ways, but none of them are very nice).  Another alternative, of course, is to just send the regex alone, without the counts at all — though doing it that way, I think I'd prefer just sending through a free string parameter.

3 hours ago, Love Zhaoying said:

The way I would handle the llLSDChaos() ("OMG this resulted in callng the LSD change events every time!1!!") scenario is:

1. Before calling that new LSD function that triggers LSD change events you don't want, set a "Semaphore"; a variable for all scripts that says "Ignore LSD Events until you hear otherwise".

Ewww!  icky.  evil.  nope.  It wouldn't be toooo bad if there was such a flag that only inhibited that specific script from sending linkset_data events…  It would cover (badly) a few other cases of flooding other scripts with LSD nonsense, too…  But it still doesn't address the problem of what if they DO need to know?  Paging the delete into smaller chunks at least gives them the option — as long as they're not running behind.

[Gayngel]

Linkset data really needs an index or FIFO system to be able to iterate over entries and do things like delete the oldest entries or loop through all keys and delete a key if it evaluates to a UUID, etc. It's very frustrating not being able to edit keys with an unknown regex pattern... or even know if all the keys with the pattern are returned for that matter.

Edited July 23, 2023 by Gayngel

[Bleuhazenfurfle]

54 minutes ago, Gayngel said:

Linkset data really needs an index or FIFO system to be able to iterate over entries and do things like delete the oldest entries or loop through all keys and delete a key if it evaluates to a UUID, etc. It's very frustrating not being able to edit keys with an unknown regex pattern... or even know if all they keys with the pattern are returned for that matter.

FIFO ordering is what you want for forming a cache — and usually implemented using a timestamp at the front of the key.  (Also, BUG-233965)

Though, I don't get most of what you're saying there.  I imagine it's along the lines of what I've said in a couple Jira comments, and ranted about in Scripts several times, among other things; The offset integer is just plain broken in the presence of other scripts…  If another script adds items, you'll get duplicates, and if another script deletes items, you'll miss some, and while you can track linkset_data to try to adjust your offset as you go, that still suffers from races.  Alpha sorting the keys is actually better for this, IF, we could use a key as the index rather than an integer offset.  DB's regularly use the last key found as their index (before they added cursors), for a reason.  (You'll still miss items added at or before your current cursor, but it's completely stable otherwise.)

[Gayngel]

8 hours ago, Bleuhazenfurfle said:

FIFO ordering is what you want for forming a cache — and usually implemented using a timestamp at the front of the key.  (Also, BUG-233965)

Though, I don't get most of what you're saying there.  I imagine it's along the lines of what I've said in a couple Jira comments, and ranted about in Scripts several times, among other things; The offset integer is just plain broken in the presence of other scripts…  If another script adds items, you'll get duplicates, and if another script deletes items, you'll miss some, and while you can track linkset_data to try to adjust your offset as you go, that still suffers from races.  Alpha sorting the keys is actually better for this, IF, we could use a key as the index rather than an integer offset.  DB's regularly use the last key found as their index (before they added cursors), for a reason.  (You'll still miss items added at or before your current cursor, but it's completely stable otherwise.)

I get what you mean about racing but that can be mitigated by only editing LSD in one script and/or through control flow.

All I want is something like this:

 

// linkset data contains the following kv pairs

// "a9ba2797-81af-429d-9833-51127ad5593c", "1"
// "apple, 0"
// "4ab1ec43-b2c0-4ded-8154-bb3b9cc9cbec", "1"
// "banana, 0"

integer tot = llLinkSetDataCountKeys();

integer i;

for(i = tot; i > -1; -- i)
{
  

  if(llLinkSetDataType(i,DATA_TYPE_KEY))  // imaginary function that casts key found by index and then evaluate it to check if it is a valid key
   
  llLinksetDataIdxDelete(i); //imaginary function that deletes a key by its index 

}

 

Far simpler than having to add and search a pattern... and with a large database, having to search several times over to make sure all keys are caught.

Edit: llLinksetDataDeleteFound might work for my purposes though once it arrives... if keys that are not to be deleted are made protected and keys that are to be deleted are unprotected. Or just delete all keys containing a pattern.

Edit edit: Some llLinksetDataArray functions would be nice actually. I've made a feature request: https://jira.secondlife.com/browse/BUG-234166

Edited July 23, 2023 by Gayngel

[Bleuhazenfurfle]

What I wanted right from the start of LSD (before it was even out for people to try), was a function whereby you give it a key, and it gives you the next one back (or an empty string).  That simple.  (Well, plus a variant that applies BUG-233078…  Can also do reverse versions…)  You won't miss any, it won't foul up on deletions or additions, and it's absolutely trivial to use in a loop.

Offset indexes are just simply utterly broken for a shared access database.  You can wave it off as "only editing LSD in one script", but that's a huge part of LSD right there.  And we don't HAVE flow control, either.  There are ways to fake database locking, but they're cludgy and fragile, and remarkably hard to get right — plus all the scripts have to agree to abide by them.  And your full-index approach, requires the ENTIRE LSD store be effectively locked even during simple searching and reading.  The present method, where you find a bunch of keys lets you index within that set to your hearts content using a simple llList2String, the problem being that list can quite readily exceed your scripts total memory.  But indexing off that snapshot list, or iteration without a list, both protect you from everything going to hell in a hand-basket, just because another script decides to add or remove a few items.

Which is where BUG-233819 comes in (either with or without the part referring to Henri, if those issues could be figured out).  Or, another option I put forward AGES ago (and several times since), of the same sort of thing, but llDetected style…  That has the added benefit that they can keep the snapshot list as a simple array of pointers to the existing key string data (or maybe even the entire items themselves, depending on how it's implemented), and only convert the key and/OR value names as needed (if you never actually care about the keys, they never need to even convert them for the script) — though hopefully without that exceedingly stupid 16 entry limit…

For offset indexes to be useful, you need to be able to either lock the store from anyone else accessing it (won't happen as a built in thing, because that would add blocking semantics), or snapshot the set of keys currently in effect (which means potentially allocating large arrays).  Iteration doesn't have those problems — its problem is a lack of random-access, but that's not generally an actual problem for most uses.  An option for locking, could be to request a lock, and have it come through as an event, during which all other scripts get an error response from LSD access — but that would break existing scripts (of which there are already a lot), so also not going to happen.  Oh, I did at one point also offer a solution for snapshotting creating large arrays, too…  Implementing versioning in the store.  You just version the store when needed any time an update is performed, and collapse old versions as their associated events conclude.  A given version can also be shared by any number of scripts, if no edits have been performed in the meantime.

In the current setting, indexing the LSD store, is just plain broken.  At least it's presently only an issue for searches.  Could you imagine searching for an item, getting it's index, and then a frame delay happens, and when you go to update or delete that item, it's the totally wrong item…?!?

Edited July 23, 2023 by Bleuhazenfurfle

[Wulfie Reanimator]

7 hours ago, Gayngel said:

Linkset data really needs an index or FIFO system to be able to iterate over entries and do things like delete the oldest entries or loop through all keys and delete a key if it evaluates to a UUID, etc. It's very frustrating not being able to edit keys with an unknown regex pattern... or even know if all the keys with the pattern are returned for that matter.

5 hours ago, Gayngel said:

Far simpler than having to add and search a pattern... and with a large database, having to search several times over to make sure all keys are caught.

What am I missing? Couldn't you either get all keys with llLinksetDataListKeys(0,-1) and iterate through that (explicitly typecasting keys), or search for the valid UUID pattern which can be very simple?

[a-zA-Z0-9]{8}-[a-zA-Z0-9]{4}-[a-zA-Z0-9]{4}-[a-zA-Z0-9]{4}-[a-zA-Z0-9]{12}

Why would you need to search multiple times?

Edited July 23, 2023 by Wulfie Reanimator

[Gayngel]

2 hours ago, Wulfie Reanimator said:

What am I missing? Couldn't you either get all keys with llLinksetDataListKeys(0,-1) and iterate through that (explicitly typecasting keys)

Because the list to too large for script memory so have to do it in stages.
 

2 hours ago, Wulfie Reanimator said:

or search for the valid UUID pattern which can be very simple?

[a-zA-Z0-9]{8}-[a-zA-Z0-9]{4}-[a-zA-Z0-9]{4}-[a-zA-Z0-9]{4}-[a-zA-Z0-9]{12}

 

For someone who doesn't understand regex and/or are proficient at it, it is far simpler to use indexes. I might use that pattern now though. Thanks Wulfie.

[Love Zhaoying]

7 minutes ago, Gayngel said:

Because the list to too large for script memory so have to do it in stages.

Why is getting the keys in stages bad? (Sorry if I misinterpreted you.)

Edited July 23, 2023 by Love Zhaoying

[Gayngel]

1 minute ago, Love Zhaoying said:

Why is getting the keys in stages bad? 

It's not bad per se, just not as efficient as getting them all at once. 

[Love Zhaoying]

I use prefixes for everything, so I'm only finding or deleting the "category" of LSD data I want, and indexes as needed. Any "searches to get a KVP", I know the exact key or variant I want.

So except for the "show me all the data in X category" or "delete all the keys in X category", there's no pain really. (Loop and get n keys, then get n more.)

Example: If the key is really a UUID, I give it a prefix ("SCHEMA_" for the current project). Yes, it adds a little data but these particular KVP's only have a "while interpreting data" / "while running a code instance" life.

Edited July 23, 2023 by Love Zhaoying

[Gayngel]

12 minutes ago, Love Zhaoying said:

So except for the "show me all the data in X category" or "delete all the keys in X category", there's no pain really. (Loop and get n keys, then get n more.)

Well that's where the new functions llLinksetDataCountFound & llLinksetDataDeleteFound will be useful.

[Love Zhaoying]

3 minutes ago, Gayngel said:

Well that's where the new functions llLinksetDataCountFound & llLinksetDataDeleteFound will be useful.

Yep, and since I don't use the LSD event...I have no issue with the planned functions currently.

[Gayngel]

4 hours ago, Bleuhazenfurfle said:

 

Which is where BUG-233819 comes in (either with or without the part referring to Henri, if those issues could be figured out).  Or, another option I put forward AGES ago (and several times since), of the same sort of thing, but llDetected style…  That has the added benefit that they can keep the snapshot list as a simple array of pointers to the existing key string data (or maybe even the entire items themselves, depending on how it's implemented), and only convert the key and/OR value names as needed (if you never actually care about the keys, they never need to even convert them for the script) — though hopefully without that exceedingly stupid 16 entry limit…

 

Pointers would be WONDERFUL!

[Quistess Alpha]

4 hours ago, Bleuhazenfurfle said:

shared access database.

for pretty much anything I do, a single script is sufficient, but I imagine for a system with enough complexity, you could probably implement some sort of 'no-write' flag while the complex read function is working. that might not be ~perfectly immune form certain race conditions, but it would probably be good enough for most applications.

[Beach Nightfire]

At Last, A system that will allow me to get rid of "lists". Mainly as they don't clear when using list name = [];  So much debugging time spent tracking that stack error.

At least LSD seems to delete when you tell it to.

[Love Zhaoying]

9 minutes ago, Beach Nightfire said:

At Last, A system that will allow me to get rid of "lists". Mainly as they don't clear when using list name = [];  So much debugging time spent tracking that stack error.

At least LSD seems to delete when you tell it to.

Yes! Using LSD + JSON, I never use lists anymore unless I have to.

[primerib1]

On 1/13/2023 at 12:48 AM, primerib1 said:

Btw I've created a ticket requesting for the addition of yet another LSD function:

llLinksetDataDeleteKeys()

Ticket: https://jira.secondlife.com/projects/BUG/issues/BUG-233195

I AM SO HAPPY

THIS TICKET HAS BEEN REALIZED

https://twitter.com/signal_linden/status/1687240521268289536

Edit: Signal Linden's confirmation that it was the result of my ticket:

https://twitter.com/signal_linden/status/1687275052008669184

Edited August 4, 2023 by primerib1

[primerib1]

On 7/13/2023 at 5:39 AM, Love Zhaoying said:

Thank you for this! 

Luckily, I don't use the  linkset_data() event, at all. (Maybe someday.)

But, every major script I write still uses LinksetData..

LinksetDataDeleteFound() sounds much better than the loop I have to use now whenever I want to delete a RegEx matched set of keys..granted, that loop becomes a simple function. But, adding that function to every script is silly.

Please post an update when these new functions are "live"!

IT'S LIVE