Jump to content
Sign in to follow this  
LindaB Helendale

Experience Key Value memory usage

Recommended Posts

I digged out the memory usage and memory allocation for experience key-value storage, and here are the results.

Should this info be in the wiki and if, where? Comments are welcome.

 

Strings use UTF-8 encoding

The data is stored with UTF-8 encoding, with basic 7-bit ascii characters one  byte, extended ascii and few others two bytes and fancy characters (ℒℬ⇄ ⇈㐦 㐧) three bytes. (See http://wiki.secondlife.com/wiki/Combined_Library#Byte_Length_of_UTF-8_Encoded_String for checking the string length in UTF-8) 

The limits are in bytes, for example the Value part can fit 4095 7-bit ascii, and 2047 two-byte and 1365 three-byte unicode characters.

Memory allocation

There's pretty good amount of memory available (thanks LL), but it's always good to be aware of memory allocation, like overhead and fragmentation issues.

The record overhead is 125 bytes. Each record needs 125 bytes + length of the Key in UTF-8 + length of the Value  in UTF-8.

Memory for the record is allocated in blocks. Key and Value share the same block, so only the total length of Key + Value affect the block size. (Key 100 bytes  + Value 400 bytes uses the same amount of memory as Key400 bytes + Value 100 bytes).

The memory block sizes are 

Block size   Payload  Added size

240          115      240
496          371      256
1008         883      512
2032         1907     1024
4080         3955     2048
8176         8051     4096

For longer records there's a chance for considrable fragmentation;  if the key+value won't fit in the payload of 4080 block (3955 bytes) the record is allocated  8176 bytes (out of which only 1011 + 4095 bytes are available storage due to llCreateKeyValue limits on the Key and Value lengths).

Each record has separate memory block, so the total memory usage is the sum the record block sizes. For example if one key-value has key+value length 884 bytes if gets 2032 bytes allocation (with the payload of 1008 block 883) and if another key-value has 5 bytes  it gets 240 bytes block and the total usage is 1248 bytes.

 

 

 

EDITS:
- had a typo on the 3rd column of the block 496 bytes. (Was 512, should be 256)
- added explanation for the usable storage limit due to key and value size limits

Share this post


Link to post
Share on other sites

well thats abit much.its there in it's current form simply on the off chance someone would need to consult it

projects with the LSL stuff rarely reach the kind of complexity that such things would be required,as there arent that many people capable of coding something that complex on here.

Share this post


Link to post
Share on other sites

I simply don't understand the entire "Memory allocation" section. One problem is that the numbers in the "Added size" column don't seem to correspond to any of the examples in the narrative, at least not in a way obvious enough for me to track.

If I had the time or inclination, I'd investigate whether the observed memory sizes might be affected by alignment, typical of tracking memory block allocations.

Anyway, the logical place for this, I'd suggest, is with all the rest of the detailed (but somewhat speculative) information on the LSL Script Memory page of the wiki

Share this post


Link to post
Share on other sites

Hi again, Linda --

Thanks for following up further on this bit of research. Yes, that's a bit much to post on the normal wiki page, but it's exactly the sort of deep notes that are often posted in the Discussion page of a function.  I'd suggest putting them at https://wiki.secondlife.com/w/index.php?title=Talk:LlKeysKeyValue&action=edit&redlink=1 , which is empty so far.

Posting them here makes them easily searchable too, of course, like your earlier thread.

Share this post


Link to post
Share on other sites


MishkaKatyusha wrote:

[ .... ]

projects with the LSL stuff rarely reach the kind of complexity that such things would be required,as there arent that many people capable of coding something that complex on here.

To the contrary, there are quite a number of excellent LSL scripters who look for this sort of information to understand the limitations on their own complex scripts.  Everything we know about how far we can stretch the usefulness of LSL has been discovered by people who take the time to do careful experiments. :smileywink:

Share this post


Link to post
Share on other sites

 

My bad if the table is not clear.  The two first columns are what a scripter may need. The left column tells how much key-value quota is charged for storing a key-value that fits in the payload (the second column).

The third column is just nice to know -stuff, listing the chunks added to the previous block size to make the next size, thus the memory is added in increments [240, 256, 512, 1024, 2048, 4096] bytes.

Oh *blush* I just realized a typo there on the second block size, no wonder the numbers didn't make sense.  

Share this post


Link to post
Share on other sites

The key-value record is allocated 240 bytes even if the key and value were 1 byte each.

It's a bit odd number, perhaps there's additional 16 byte header that is not counted in the experience key-value quota, but we get charged for 240 bytes.

 

Share this post


Link to post
Share on other sites

Thanks Linda, that completely clarifies that third "Added size" column. I'm still hitting a wall, though, trying to understand one other comment: "(out of which only 1011 + 4095 bytes are available storage)." It's very possible I'm just being dense, but try as I might, I'm not seeing whence those numbers derive.

 

Share this post


Link to post
Share on other sites

quite a number maybe yes.but not by a logical count at all

even a cursory glance,taking in things as a whole,indicates that the number of starting,or low level scripters,or just plain bad scripters far outweighs the number of people with enough expertise to be able to use this

which coincidentally leads me to agree with your assertion "thats abit much for a normal wiki page" because basically my assertion,and that one you made,could be construed to say the same thing really

im not saying this stuff isnt useful,or that experimenters arent useful,come to think of it,i was agreeing with that post of yours i just highlighted before you made it o.O convergent evolution perhaps?

 

back on topic,by all means put this in a sub-section like that,the people who are concerned with such things will know where to look for it if this is done

Share this post


Link to post
Share on other sites

hi Qie,

the usable storage upper limit comes from the llCreateKeyValue limits 
(see http://wiki.secondlife.com/wiki/LlCreateKeyValue)
the key is limited to 1011 bytes and the value to 4095 bytes.

I added the explanation to the original post, thanks for pointing out that it may not be clear unless you are familiar with the Key-Value functions.

 

Share this post


Link to post
Share on other sites

Oh yes, thank you. Those numbers would have been clear in some contexts where this information would be used, but they weren't obvious in isolation.  Now it all makes perfect sense even without that context top-of-mind. Thanks again.

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Sign in to follow this  

×
×
  • Create New...