"Linkset Data" is coming.

[primerib1]

11 hours ago, Anna Salyx said:

One comment here.  LSD is a prim property independent of any script.  Much like hover text, texture animation, particles, and etc, LSD persists if you remove the script. So deleting/resetting the data store will have zero effect on "script load" as is your stated purpose/concern.  I don't know how much overhead is involved when objects with any of the "extended" properties set are moved from region host to region host, or how much of a concern that truly is, but from a script / script load perspective, that particular overhead is, I believe, non-existent.

Ah thanks for the clarification.

Yeah, I might be using the term "script load" too broadly ... in any case, usage of LSD means there's additional 64 KiB of info to carry over to the next sim, for the purpose of re-rezzing the avi after crossing.

So I wonder if not having any LSD -- or clearing up LSD either using Reset() or Delete() -- can reduce the amount of data being transferred between sims...

[primerib1]

I've seen your comment @Lucia Nightfire, that's a great idea!

So I have edited my ticket.

I'm still not sure how to best implement a similar function for protected keys, though. So I explicitly limit the ticket to only request deletion for unprotected keys ... but changing the signature so it returns some helpful information.

To handle mass-deletion of protected keys, we will need @Rider Lindento pipe up 😁 mostly because maybe due to how protection is implemented, the function might not be as simple as my proposed function...

Edited January 13, 2023 by primerib1

[Love Zhaoying]

11 minutes ago, primerib1 said:

Ah thanks for the clarification.

Yeah, I might be using the term "script load" too broadly ... in any case, usage of LSD means there's additional 64 KiB of info to carry over to the next sim, for the purpose of re-rezzing the avi after crossing.

So I wonder if not having any LSD -- or clearing up LSD either using Reset() or Delete() -- can reduce the amount of data being transferred between sims...

It may be the case that with everything else associated with you, your avatar, attachments, etc. that 64k is tiny. A few textures worth of data. Just to put things in perspective..

[primerib1]

A different question:

If I do a write to a Protected Key, will linkset_data() event also contains the value that I write to that Protected Key?

Or will the value be "" and I have to do a llLinksetDataReadProtected() call to see the newly written value?

EDIT: Oh, I will receive an empty string according to this. Okay.

Edited January 13, 2023 by primerib1

[Wulfie Reanimator]

6 minutes ago, primerib1 said:

A different question:

If I do a write to a Protected Key, will linkset_data() event also contains the value that I write to that Protected Key?

Or will the value be "" and I have to do a llLinksetDataReadProtected() call to see the newly written value?

The value will be an empty string in the event, which is also mentioned as a caveat on the wiki page:
https://wiki.secondlife.com/wiki/LlLinksetDataWrite

[Love Zhaoying]

So far, I don't have a use-case for protected keys (in my actual project).

Or "Finding Keys" via RegEx.

Hopefully will have a reason before all the LSD functions become "set in stone"..

[primerib1]

9 minutes ago, Wulfie Reanimator said:

The value will be an empty string in the event, which is also mentioned as a caveat on the wiki page:
https://wiki.secondlife.com/wiki/LlLinksetDataWrite

Yup I managed to ninja your answer 😉

[primerib1]

9 minutes ago, Love Zhaoying said:

So far, I don't have a use-case for protected keys (in my actual project).

Or "Finding Keys" via RegEx.

Hopefully will have a reason before all the LSD functions become "set in stone"..

It will probably be useful when you're starting to use LSD as a database...

"db-<packed_uuid>-lastvisit"
"db-<packed_uuid>-lastscore"
"db-<packed_uuid>-hiscore"

Then you want everyone's hiscore:

list keys_hi = llLinksetDataFindKeys(".*-hiscore");
integer c = keys_hi - [];
integer i
for(;i<c;++i) {
    // Do stuff
}

[Love Zhaoying]

15 minutes ago, primerib1 said:

It will probably be useful when you're starting to use LSD as a database...

"db-<packed_uuid>-lastvisit"
"db-<packed_uuid>-lastscore"
"db-<packed_uuid>-hiscore"

Then you want everyone's hiscore:

list keys_hi = llLinksetDataFindKeys(".*-hiscore");
integer c = keys_hi - [];
integer i
for(;i<c;++i) {
    // Do stuff
}

So far, I'm using (a "slot" plus) UUID's for top-level key's. Then, any related data would be under that.

So to use your example, if the "<packed_uuid>" represented a "specific game (name of the game), etc."+"player", then under that would be all that player's stats for that game.

But I'm not storing scores yet..so we''ll see what schema makes the most sense for my overall schemas!

Thanks!

 

 

Edited January 13, 2023 by Love Zhaoying

[Quistess Alpha]

18 minutes ago, primerib1 said:

llLinksetDataFindKeys(".*-hiscore");

FWIW, despite my very earlier comment that you couldn't (because LSD regex wasn't finished then), you can use '$' as an 'end of key' specifier:

llLinksetDataFindKeys("-hiscore$");

similarly '^' as a beginning of key specifier ("^db-<packed_UUID>")

PS. what key-packing method are you thinking of? because utf-8 is different/ more efficient than utf-16, @Mollymews's key-compression compresses a 36-character key to 27~25 bytes (variable depending on key) ; a llBase64 on each of the 4 encoded integers compresses to 24 bytes. The theoretical minimum is around 20~21 bytes.

 

Edited January 13, 2023 by Quistess Alpha

[Love Zhaoying]

26 minutes ago, Quistess Alpha said:

PS. what key-packing method are you thinking of? because utf-8 is different/ more efficient than utf-16, @Mollymews's key-compression compresses a 36-character key to 27~25 bytes (variable depending on key) ; a llBase64 on each of the 4 encoded integers compresses to 24 bytes. The theoretical minimum is around 20~21 bytes.

I'm still not really convinced that for most use-cases, some portion of the UUID wouldn't be just fine. Assume the data is not to be "global", just relevant to one user and/or their friends. For instance: first group + last group of digits from the full UUID. ETA: But I am bad at teh maths, so I don't know the probability of conflicts / collisions for my use-case.

Edited January 13, 2023 by Love Zhaoying

[Quistess Alpha]

1 minute ago, Love Zhaoying said:

first group + last group of digits from the full UUID

yeah, llDeleteSubString(llIntegerToBase64(llHash(key)),-2,-1); is good enough in practice, but the problem is the failure case is persistent. if some random person has 2 friends who magically fall in the same box, they're going to have a confusing bug and it won't fix itself no-matter what they do. I think a few bytes is probably worth not accidentally confusing someone if you distribute your toy widely. if you're just collecting data on passers-by to your sim or something, confusing 2 people probably isn't the end of the world.

[Love Zhaoying]

7 minutes ago, Quistess Alpha said:

yeah, llDeleteSubString(llIntegerToBase64(llHash(key)),-2,-1); is good enough in practice, but the problem is the failure case is persistent. if some random person has 2 friends who magically fall in the same box, they're going to have a confusing bug and it won't fix itself no-matter what they do. I think a few bytes is probably worth not accidentally confusing someone if you distribute your toy widely. if you're just collecting data on passers-by to your sim or something, confusing 2 people probably isn't the end of the world.

..plus: If I chose to use the user's GUID as part of the top-level key, that's the only occurrence in the data at all. If not tracking 100's of avatars, just friends who "play", etc.

[Love Zhaoying]

FYI, I'm finally seeing some performance issues after adding a level or two of processing.

I don't have a "performance trace" utility built yet..but this is a good reason to do it.  In this case, to track how long certain functions take in "ticks". 

Anyway, for immediate mitigation / experimentation, I plan to move some data manipulation OUT of LSD back to normal variables for processing.  If that seems to help, then it means the LSD is adding some overhead.

But I can also add "LSD performance tracing" pretty easily since I am only making the LSD calls in TWO places (!) in the entire program.  So, I can compare the amount of ticks/time spent in those two calls, to the ticks/time spent overall.

[primerib1]

12 minutes ago, Love Zhaoying said:

FYI, I'm finally seeing some performance issues after adding a level or two of processing.

I don't have a "performance trace" utility built yet..but this is a good reason to do it.  In this case, to track how long certain functions take in "ticks". 

Anyway, for immediate mitigation / experimentation, I plan to move some data manipulation OUT of LSD back to normal variables for processing.  If that seems to help, then it means the LSD is adding some overhead.

But I can also add "LSD performance tracing" pretty easily since I am only making the LSD calls in TWO places (!) in the entire program.  So, I can compare the amount of ticks/time spent in those two calls, to the ticks/time spent overall.

Interesting...

Back when LSD was just being rolled out, @Kadah Cobadid a test, and LSD performs really well : 

 

[Love Zhaoying]

5 minutes ago, primerib1 said:

Interesting...

Back when LSD was just being rolled out, @Kadah Cobadid a test, and LSD performs really well : 

Yep, so I won't be surprised if it is the REST of my code/logic.  A few too many hoops and calls and checks to / within its own functions, for instance.

[Lucia Nightfire]

6 hours ago, primerib1 said:

integer c = keys_hi - [];

🤨

Does that actually compile anywhere?

Doesn't for me.

[Wulfie Reanimator]

10 minutes ago, Lucia Nightfire said:

🤨

Does that actually compile anywhere?

Doesn't for me.

They may be thinking of using != to substitute llGetListLength:

integer length = [1,1,1] != []; // 3, the length of the non-empty list
integer difference = [1,1,1] != [123]; // 2, subtracts length of A from B

 

[Love Zhaoying]

I've got "round 1" of Timing analysis logging - TO LSD! (Kind of..Meta..)

So, once I get some more time with that, I'll be able to see if it's my logic that made the application suddenly perform worse, or just general bad mojo.

Logging timing of an LSD application..TO LSD.. <== On topic!

[Love Zhaoying]

I did it! Now I've got "Profiling" saved in LSD for counting each of my function calls, and amount of script time total spent in each call.

Next, since I have control of not just each function but every command / action in each function, I can add profiling for count and time spent in each command/action.

I'm not using events except link_message(). I'm not profiling individual LSD calls. I'm profiling each function in my JSON parser/interpreter. Adding command/action profiling just means doing the same thing for each command/action the parser calls.

This is like, if you could profile an LSL program to find out which command is taking the most time, and how many times you call each command.

Who said programming was difficult?

I'm curious what other "seemingly impossible" things others are doing with LSD.

[Anna Salyx]

12 hours ago, primerib1 said:

Ah thanks for the clarification.

Yeah, I might be using the term "script load" too broadly ... in any case, usage of LSD means there's additional 64 KiB of info to carry over to the next sim, for the purpose of re-rezzing the avi after crossing.

So I wonder if not having any LSD -- or clearing up LSD either using Reset() or Delete() -- can reduce the amount of data being transferred between sims...

The key differences, in my own understanding, is that the script is a "living" process. When a script moves from sim to sim it must be registered on the new host as a running process.  It's byte code loaded (if needed) and it's current stack and event queue applied, and finally given place in the queue do it's thing. Where as a mesh/prim object by itself is just a static thing and all that needs to be done is to provide the receiving sim a packed (I assume) copy of it's current properties alongside it's asset ID. The client/viewer renders it and Bob's your uncle.  Yes there is going to be some overhead in the LSD store, but that store is not an action item that requires VM registration and time slices. 

And as LZ pointed out (below), in the scheme of things it's not *that* much really.   If you're carrying 38 attachments each chock full to brim with LSD keys, maybe then it'll have an impact, but if all you've got is a small set of objects, 1 to 3 maybe, each with only a handful of keys moving around with you, that might not be even noticed. 

 

11 hours ago, Love Zhaoying said:

It may be the case that with everything else associated with you, your avatar, attachments, etc. that 64k is tiny. A few textures worth of data. Just to put things in perspective..

if I'm wrong on my admittedly limited info assessment on how things move from region to region, I'll be happy corrected so I can learn

[Kadah Coba]

FYI,  we have an actual feature request filed for remote LSD access: https://jira.secondlife.com/browse/BUG-233201

 

9 hours ago, Love Zhaoying said:

Yep, so I won't be surprised if it is the REST of my code/logic.  A few too many hoops and calls and checks to / within its own functions, for instance.

All the string processing that's required in LSL is really painful. I would not be surprised if your script is getting random sleeps during that. I was working on some data packing code a couple weeks back and randomly some cycles will just take 10x longer. That behavior will only be worse the more loaded a region is, which is extra FUN. :p

[Love Zhaoying]

7 minutes ago, Kadah Coba said:

All the string processing that's required in LSL is really painful. I would not be surprised if your script is getting random sleeps during that. I was working on some data packing code a couple weeks back and randomly some cycles will just take 10x longer. That behavior will only be worse the more loaded a region is, which is extra FUN. 😛

String processing is about the most efficient I can get. That and using Integers and bitflags (the next step) to reduce string comparisons for the interpreter. I think "extra" JSON calls is my problem, my new profiling will reveal all!

[primerib1]

7 hours ago, Wulfie Reanimator said:

They may be thinking of using != to substitute llGetListLength:

Ack you're right.

Blame it on not enough coffee 🤣

[primerib1]

3 hours ago, Anna Salyx said:

The key differences, in my own understanding, is that the script is a "living" process. When a script moves from sim to sim it must be registered on the new host as a running process.  It's byte code loaded (if needed) and it's current stack and event queue applied, and finally given place in the queue do it's thing. Where as a mesh/prim object by itself is just a static thing and all that needs to be done is to provide the receiving sim a packed (I assume) copy of it's current properties alongside it's asset ID. The client/viewer renders it and Bob's your uncle.  Yes there is going to be some overhead in the LSD store, but that store is not an action item that requires VM registration and time slices.

That's a nice explanation, thanks!

3 hours ago, Anna Salyx said:

If you're carrying 38 attachments each chock full to brim with LSD keys, maybe then it'll have an impact, but if all you've got is a small set of objects, 1 to 3 maybe, each with only a handful of keys moving around with you, that might not be even noticed. 

Hmmm... Come to think of it... based on the previous discussions, and depending on the architecture, the size of LSD Store (that is, the number of KVP within) might not matter.

*) If implemented as, say, microservice, then size has no impact

*) If implemented as a blob, size might be static: 64 KiB preallocated regardless of how many KVP inside

*) If implemented as actual hashtable/dict that gets composed into item object, then number of KVP does matter

Edited January 14, 2023 by primerib1