llGetNotecardLineSync()

[Bleuhazenfurfle]

16 hours ago, Cain Maven said:

Understood -- but in those rare cases an error (say NAK, heh) could be returned. This would be no different from other synchronous functions, and handling that edge case would probably not be any harder or more complex than the timout and error reporting you suggest; it may even be simpler.

Actually, it is.  You're talking about a fair chunk of additional complexity they'd have to add.  As pointed out by others, the sim doesn't know if the notecard even exists, it has to send off a message to the asset server to get it.  The asset server will hopefully respond either with the notecard, or a negative.  But it may not respond at all.  Networks sometimes lose packets, the asset server could be down.  Or it might just take a freakishly long time to get the response.

Pretty sure the current implementation cares about none of that.  It sends the request, and then forgets about it ("fire-and-forget", the way the internet was designed to work).  The only one who still remembers (maybe), is you (but clearly not Anna Salyx), by hanging onto that key it gave you.  To wait, and generate a NAK, means adding a timeout, and then likely switching over to guaranteed message delivery (or adding in retry and falloff logic), and deciding on timeouts, and the cost of all that and then cleaning everything up again in the very common case of the response coming back normally and none of that extra stuff was even needed.

This is the benefit of fire-and-forget; there's no set-up (apart from generating a key), clean-up, or on-going costs (apart from sending that key across the network twice).  Especially when the client (you) don't actually care that much (sometimes happens) — and when you do, you know better than the sim how much you care, and can take steps accordingly.

 

15 hours ago, Love Zhaoying said:

I'm hoping that the initial async call guarantees the NC is cached for subsequent sync calls. Like..why wouldn't it be?

Unless a sim restart occurs in the middle of reading..or someone changes the NC in the middle of reading..those are both edge cases that would result in unpredictable results anyway.

As I was explaining earlier, it depends a lot on the caching mechanism used, how much stuff is fighting for that cache space, and whether your script gets paused during it's looping.  Caches are often (at least partially) optimistically drawn from the systems memory, meaning the cache shrinks if the memory is needed for other tasks.  As an example; a bunch of script lag caused by a lot of scripts using a lot of memory, would both shrink the cache, and introduce longer pauses into your scripts (because all those other scripts can't all fit into the same timeslice, so you find yourself waiting a couple frames in the run queue each time), and if many of them are causing other notecards to be loaded into that reduced cache…  And now you're causing your notecard to be loaded into that same cache, over and over…  I'm sure you can see where that ends up.  (I don't think this is a likely scenario any more, memory has grown over the years, but it's still an example.)

And yes, there's that other stuff that can happen, too.  We also know that one physical machine generally runs more than one region, and we really don't know how much head room each region is given, or whether it's even fixed (even VM's often have a mix of fixed and shared memory).  And are these caches shared across all the regions on that one box?  (Probably, because balancing multiple caches on the same machine is kinda daft.)

But the results seem to suggest the notecards remain cached a nice long time, so it should be very much uncommon — but, uncommon isn't the same as never, so…

 

4 hours ago, elleevelyn said:

when the cache is available then use it, when it isn't then continue getting data with standard llGetNotecardLine and dataserver event

And for the most part, it really is as simple as that.  Sprinkle a bit of llGetNotecardLineSync into your existing notecard code, and you're done.  If you don't mind that initial 0.1s delay (and I know people want their scripts to run FAST, but I'd suggest that it's rare that faster in this way is actually beneficial), then that's really all you need to do to make full use of this — the bonus speed, same reliability, all for barely any additional effort.  (And getting rid of that 0.1s initial delay typically just means wrapping the line handling in a function.)

[elleevelyn]

2 hours ago, Bleuhazenfurfle said:

And for the most part, it really is as simple as that.  Sprinkle a bit of llGetNotecardLineSync into your existing notecard code, and you're done.  If you don't mind that initial 0.1s delay (and I know people want their scripts to run FAST, but I'd suggest that it's rare that faster in this way is actually beneficial), then that's really all you need to do to make full use of this — the bonus speed, same reliability, all for barely any additional effort.  (And getting rid of that 0.1s initial delay typically just means wrapping the line handling in a function.)

pretty much yes. The more I look at this then I can only see issues trying to deal with NAK in a custom loop

yes can do retry but what do we do when the retry constantly returns NAK ? Working with the assumption tnat this won't happen is basically gambling. And I am myself not into writing gambling code for a data retrieval program. If we break out of the retry loop then what ? Do we save the line numbers we missed and go back and get them later ?

If try to mitigate with llSleep then how long do we sleep it for ? If we do sleep it then none of the other events are going to fire while the script is sleeping

is all super messy are custom loops

[Cain Maven]

5 hours ago, Bleuhazenfurfle said:

Actually, it is.  You're talking about a fair chunk of additional complexity they'd have to add. 

We can all speculate. My rule of thumb is that it's better that complexity is handled once on the back end than thousand of times in the client -- with varying degrees of success. But reasonable people can disagree.

[Qie Niangao]

4 hours ago, elleevelyn said:

pretty much yes. The more I look at this then I can only see issues trying to deal with NAK in a custom loop

yes can do retry but what do we do when the retry constantly returns NAK ? Working with the assumption tnat this won't happen is basically gambling. And I am myself not into writing gambling code for a data retrieval program. If we break out of the retry loop then what ? Do we save the line numbers we missed and go back and get them later ?

If try to mitigate with llSleep then how long do we sleep it for ? If we do sleep it then none of the other events are going to fire while the script is sleeping

The question "then what ?" is indeed apt. There's no rule that we must use an inescapable loop of llSleep()s, we can instead count down retries and give up with a timer to try again after a while, letting the rest of the script function (if possible, devoid of notecard data). And that's what's needed for the async version, too: if the card won't cache, it won't read into dataserver either. Unless a call to llGetNotecardLine has a timeout on getting a response, it's gambling too.

On the other hand, I'm not sure looped retries are really necessary if there's provision for timer-based recovery. My hunch is a single call to -Line or -Lines (with that built-in 0.1s delay) will almost always (re-)cache the notecard unless there's something sick enough to warrant timer-based recovery. At least in casual beta grid testing of this little script on ancient notecards that aren't cached, the first click keeps getting results. That might be different after deployment on a busy region.

string NC = "4dc0958c-e299-2408-9ba0-a773d9d9a673";
integer lineNumber;

default
{
    touch_start(integer total_number)
    {
        string data = llGetNotecardLineSync(NC, lineNumber);
        if (NAK == data)
        {
            llOwnerSay("(FYI, confirming this notecard is uncached)");
            llGetNumberOfNotecardLines(NC);
            data = llGetNotecardLineSync(NC, lineNumber);
            if (NAK == data)
            {
                // set up recovery from missing notecard response (start timer probably)
                return;
            }
        }
        if (EOF == data)
        {
            llOwnerSay("notecard exhausted or includes non-text content");
            return;
        }
        // Use the line data. What's done here would also need doing upon a successful recovery
        llOwnerSay("line "+(string)lineNumber+": "+data);
        lineNumber++;
    }
}

This example reflects my interest in sync notecard reading as a substitute for linkset data in some applications, so it's doing "line at a time" access rather than "suck in the whole notecard in one gulp".

In this silly example, "recovery" would be clicking again, but so far that hasn't been necessary.

[Love Zhaoying]

9 minutes ago, Qie Niangao said:

The question "then what ?" is indeed apt. There's no rule that we must use an inescapable loop of llSleep()s, we can instead count down retries and give up with a timer to try again after a while, letting the rest of the script function (if possible, devoid of notecard data). And that's what's needed for the async version, too: if the card won't cache, it won't read into dataserver either. Unless a call to llGetNotecardLine has a timeout on getting a response, it's gambling too.

On the other hand, I'm not sure looped retries are really necessary if there's provision for timer-based recovery. My hunch is a single call to -Line or -Lines (with that built-in 0.1s delay) will almost always (re-)cache the notecard unless there's something sick enough to warrant timer-based recovery. At least in casual beta grid testing of this little script on ancient notecards that aren't cached, the first click keeps getting results. That might be different after deployment on a busy region.

string NC = "4dc0958c-e299-2408-9ba0-a773d9d9a673";
integer lineNumber;

default
{
    touch_start(integer total_number)
    {
        string data = llGetNotecardLineSync(NC, lineNumber);
        if (NAK == data)
        {
            llOwnerSay("(FYI, confirming this notecard is uncached)");
            llGetNumberOfNotecardLines(NC);
            data = llGetNotecardLineSync(NC, lineNumber);
            if (NAK == data)
            {
                // set up recovery from missing notecard response (start timer probably)
                return;
            }
        }
        if (EOF == data)
        {
            llOwnerSay("notecard exhausted or includes non-text content");
            return;
        }
        // Use the line data. What's done here would also need doing upon a successful recovery
        llOwnerSay("line "+(string)lineNumber+": "+data);
        lineNumber++;
    }
}

This example reflects my interest in sync notecard reading as a substitute for linkset data in some applications, so it's doing "line at a time" access rather than "suck in the whole notecard in one gulp".

In this silly example, "recovery" would be clicking again, but so far that hasn't been necessary.

Dum question: If a Notecard "exists" in inventory, for instance if you check that it's there by name before async read, should not the async notecard read always work? If not, and a timer is needed..IMO there are larger problems!

Help my unnerstand!

 

[Love Zhaoying]

25 minutes ago, Qie Niangao said:

This example reflects my interest in sync notecard reading as a substitute for linkset data in some applications, so it's doing "line at a time" access rather than "suck in the whole notecard in one gulp".

It will be interesting to compare the relative speed of reading NC sync, vs. reading LSD. If they are directly comparable, I will move my "Schemas" (which are read-only) to NC's, and leave my "Data" in LSD.

The limitation then would be as always, 1024 chars per NC line - requiring multiple NC reads vs. 1 LSD read. (Not a perfect trade-off even if speed is comparable.)

 

[Qie Niangao]

9 minutes ago, Love Zhaoying said:

Dum question: If a Notecard "exists" in inventory, for instance if you check that it's there by name before async read, should not the async notecard read always work? If not, and a timer is needed..IMO there are larger problems!

I haven't really tested that because I've assumed that adding the notecard to inventory would cache it and I haven't been patient enough to wait for it to be uncached (instead just grabbing UUIDs of ever dustier notecards). But I think @Anna Salyx's results above suggest the simulation can know the existence of a notecard (even a newly edited one) without having it cached. Maybe checking by name that it's in a prim's inventory does more to get it cached, but (without any testing at all) I kinda doubt it.

[Qie Niangao]

3 minutes ago, Love Zhaoying said:

It will be interesting to compare the relative speed of reading NC sync, vs. reading LSD. If they are directly comparable, I will move my "Schemas" (which are read-only) to NC's, and leave my "Data" in LSD.

The limitation then would be as always, 1024 chars per NC line - requiring multiple NC reads vs. 1 LSD read. (Not a perfect trade-off even if speed is comparable.)

That will be interesting, but I doubt it'll be that close. In the applications I'm considering, I'm going for merely reasonably speedy random access of single lines in notecard(s), to avoid LSD complexities such as handling multiple instances getting linked together (and other modifiable linkset shenanigans).

[Wulfie Reanimator]

1 hour ago, Qie Niangao said:

And that's what's needed for the async version, too: if the card won't cache, it won't read into dataserver either.

Does this actually happen, though? This sounds like a hypothetical that no one has mentioned in the past 15 years.

1 hour ago, Qie Niangao said:

I haven't really tested that because I've assumed that adding the notecard to inventory would cache it

This wasn't my experience while testing, I had to async the notecard first. (Also, the notecard stayed in cache for over 9 hours, even though I deleted the object with the notecard. 😄)

Edited January 29 by Wulfie Reanimator

[Love Zhaoying]

29 minutes ago, Wulfie Reanimator said:

1 hour ago, Qie Niangao said:

And that's what's needed for the async version, too: if the card won't cache, it won't read into dataserver either.

Does this actually happen, though? This sounds like a hypothetical that no one has mentioned in the past 15 years.

I'm assuming it's an "edge case" - could happen due to system errors, so if you're really worried about a script stalling due to dataserver() never running, a timer would be worthwhile.  I really don't want to go back and code my older scripts to handle this, but for my scripts that "load the whole NC once when the user says to load", I see the value in it.  But just ewww, since we never HAD to deal with this AFAIK.

IMO, llGetNotecardLine() / dataserver() should have been been written so dataserver() return an error (like existing EOF or new NAK) if the request cannot be honored due to "notecard does not exist", etc. although I'm sure that would require changes on the simulator side (whether or not it would break existing content is another thing.)
 

33 minutes ago, Wulfie Reanimator said:

1 hour ago, Qie Niangao said:

I haven't really tested that because I've assumed that adding the notecard to inventory would cache it

This wasn't my experience while testing, I had to async the notecard first. (Also, the notecard stayed in cache for over 9 hours, even though I deleted the object with the notecard. 😄)

Thanks to you and the others for testing, so us lazy / "busy" people can sit on our "laurels" and reap the rewards!

[Qie Niangao]

21 minutes ago, Wulfie Reanimator said:

Does this [notecard failing to read into dataserver] actually happen, though? This sounds like a hypothetical that no one has mentioned in the past 15 years.

Right, I've never seen it. (But the logic isn't crazy, is it? If the card is readable, it must be cacheable, right?)

Time will tell, though, how fast an uncached card caches, though. I'm currently betting the built-in 100 msec delay of an async call works often enough to treat the exceptions with an async timer-driven recovery rather than bothering with a synchronous retry countdown.

22 minutes ago, Wulfie Reanimator said:

This [adding a notecard to inventory caches it] wasn't my experience while testing, I had to async the notecard first. (Also, the notecard stayed in cache for over 9 hours, even though I deleted the object with the notecard. 😄)

Yeah, I tried to backpedal on that assumption because @Anna Salyx gave results that sure look as if it takes an async attempt somewhere in the region to get it cached, so that already refutes my earlier assumption. (Perhaps obvious but I had to try anyway: a NAK-responded synchronous attempt does not trigger caching.)

[Love Zhaoying]

Does the testing indicate that llGetNumberOfNotecardLines() forces a cache?

Sorry, not sure I saw that in the testing.

Of course, in "real use-cases", a lot of us don't actually need llGetNumberOfNotecardLines() - you read until you get "EOF".

 

[Qie Niangao]

1 minute ago, Love Zhaoying said:

Does the testing indicate that llGetNumberOfNotecardLines() forces a cache?

Yes, and I've been using it. [Doh, I wrote something silly here earlier]

Edited January 29 by Qie Niangao

[Anna Salyx]

1 hour ago, Qie Niangao said:

I haven't really tested that because I've assumed that adding the notecard to inventory would cache it and I haven't been patient enough to wait for it to be uncached (instead just grabbing UUIDs of ever dustier notecards). But I think @Anna Salyx's results above suggest the simulation can know the existence of a notecard (even a newly edited one) without having it cached. Maybe checking by name that it's in a prim's inventory does more to get it cached, but (without any testing at all) I kinda doubt it.

Yes, I can/will confirm that in my testing, adding an uncached notecard to item inventory does NOT cause it to become cached. It takes direct call to a the dataserver version for it to be loaded into the cache. That was very consistent in my testing. The same with editing in place and creating a distinct change. and yes, my testing was fairly blind.  Now, I could to some extra sanity checks into my test code to verify the existence of non blank NC and see if that makes a difference. 

[Love Zhaoying]

8 minutes ago, Anna Salyx said:

Yes, I can/will confirm that in my testing, adding an uncached notecard to item inventory does NOT cause it to become cached. It takes direct call to a the dataserver version for it to be loaded into the cache. That was very consistent in my testing. The same with editing in place and creating a distinct change. and yes, my testing was fairly blind.  Now, I could to some extra sanity checks into my test code to verify the existence of non blank NC and see if that makes a difference. 

Doesn't that confirm that it is always safest (and "simplest") for any script to perform the initial notecard read by calling the dataserver version?  If so, it seems we are overcomplicating things.

(Unless the script tracks whether previous reads were done, and whether the inventory changed - potentially signalling that the  notecard changed..)

 

[elleevelyn]

just because I can and I got nothing better to do😸

@OhWellDoItAgain;
string data;
integer continue;
do
{
   llGetNotecardLinesCount();
   data = llGetNotecardLineSync(notecardName, notecardLine);
   if (data == NAK)
   {  // thrash it upto 10,000 times but yanno :)
      continue = ++continue % 10000;   
   }
   else
      continue = FALSE;
     
} while (continue);

if (data == NAK)
{
   jump OhWellDoItAgain;
}

 

[Qie Niangao]

9 minutes ago, Love Zhaoying said:

Doesn't that confirm that it is always safest (and "simplest") for any script to perform the initial notecard read by calling the dataserver version?  If so, it seems we are overcomplicating things.

Especially if the script is reading the whole notecard at once, it makes sense to make an async call and then process it all inside the dataserver handler, but if it needs a random line "occasionally" it may call the sync version in hopes it gets lucky on the first try. (It could do an initial async call when it first learns the notecard identity, perhaps at state_entry—and maybe at CHANGED_REGION_START to improve the odds for the first/next sync read.)

[Love Zhaoying]

3 minutes ago, elleevelyn said:

just because I can and I got nothing better to do😸

@OhWellDoItAgain;
string data;
integer continue;
do
{
   llGetNotecardLinesCount();
   data = llGetNotecardLineSync(notecardName, notecardLine);
   if (data == NAK)
   {  // thrash it upto 10,000 times but yanno :)
      continue = ++continue % 10000;   
   }
   else
      continue = FALSE;
     
} while (continue);

if (data == NAK)
{
   jump OhWellDoItAgain;
}

 

If I understand:

- The call to llGetNotecardLinesCount() is possibly supposed to be llGetNumberOfNotecardLines().  I assume you are running this without a matching dataserver() event, and just hoping it results in a notecard cache (it will cause the script to sleep for 0.1 seconds, no matter what per the Wiki). 

- You are using "continue" both as a boolean loop flag and a loop counter.

- If somehow the loop counter reached the end, and the last data read was NAK, the last code shown is just to shown a "start over" condition.

Assuming everything else is right (except llGetNotecardLinesCount() call), then I guess my only exception to this approach is, you can't have a dataserver() event for some other reason or if you do, it had better ignore the llGetNumberOfNotecardLines() since the handle passed to dataserver() won't be expected.  

Using this method may preclude also using the "force a read once using llGetNotecardLine()" method, unless you add what's needed so there's no conflict.

I hope that understanding seems reasonable!

 

 

 

 

[Wulfie Reanimator]

Noticed really annoying (but consistent) behavior I never knew about.

If the last line of a notecard contains text but no newline, trying to read that line will return EOF instead of the text.

This happens with Sync/Async and applies even to notecards that only contain one line of text, but no newline. The line cannot be read because the result is EOF. I'm sure someone has to have filed a jira about it, but what's the current status?

[Anna Salyx]

Quote

[06:44] Notecard Syn tester v2: required notecard exists
[06:44] Notecard Syn tester v2: NC Key is: 00000000-0000-0000-0000-000000000000
[06:44] Notecard Syn tester v2: linkset records stored: 0
[06:44] Notecard Syn tester v2: 1706539469: starting read of 1000 lines
[06:44] Notecard Syn tester v2: notecard not ready at line: 0
[06:44] Notecard Syn tester v2: 1706539472: all NC lines Read
[06:44] Notecard Syn tester v2: linkset records stored: 1000

Had my testing alt create the same NC and give it to me with M/C/NT perms. 

Tested for the existence by name.

Tried to get the UUID (failed because it's not full perm, but interestingly I did not get the expected shout on the debug channel)

Still failed on the first Sync read before I did the async dataserver nudge.

[elleevelyn]

1 minute ago, Love Zhaoying said:

If I understand:

- The call to llGetNotecardLinesCount() is possibly supposed to be llGetNumberOfNotecardLines().  I assume you are running this without a matching dataserver() event, and just hoping it results in a notecard cache (it will cause the script to sleep for 0.1 seconds, no matter what per the Wiki). 

- You are using "continue" both as a boolean loop flag and a loop counter.

- If somehow the loop counter reached the end, and the last data read was NAK, the last code shown is just to shown a "start over" condition.

Assuming everything else is right (except llGetNotecardLinesCount() call), then I guess my only exception to this approach is, you can't have a dataserver() event for some other reason or if you do, it had better ignore the llGetNumberOfNotecardLines() since the handle passed to dataserver() won't be expected.  

Using this method may preclude also using the "force a read once using llGetNotecardLine()" method, unless you add what's needed so there's no conflict.

I hope that understanding seems reasonable!

yes you onto it

it just shows in a horrible way what is needed when we don't use the dataserver event in combination with llGetNotecardLineSync(). Is in the not-the-same-as-the-last algorithm class. Example of:
 

// thrash it until nextnumber is not equal to thisnumber.
// How likely is it that they will always be equal ?  
// Not likely, until the one time it is

float thisnumber = llFrand(1.0);
float nextnumber;
do
{
    nextnumber = llFrand(1.0);
} while (nextnumber == thisnumber);

 

[Frionil Fang]

2 hours ago, Love Zhaoying said:

It will be interesting to compare the relative speed of reading NC sync, vs. reading LSD. If they are directly comparable, I will move my "Schemas" (which are read-only) to NC's, and leave my "Data" in LSD.

2 hours ago, Qie Niangao said:

In the applications I'm considering, I'm going for merely reasonably speedy random access of single lines in notecard(s)

With some (very) crude testing, the speed quite comparable between the two of them.

I didn't do any fine-grained or huge dataset testing, but reading a notecard with 10 lines of 1024 bytes completes within one frame (llGetTime() returns 1/45 seconds), reading the same data from linkset (keys = line numbers, values = lines) was more fluctuating but median 2 frames.

Reading 80 lines of 128 bytes, NCsync reads completed within 2-3 frames (median 2), LSD read completed within 1-2 frames (median 2).

5 minutes ago, Wulfie Reanimator said:

If the last line of a notecard contains text but no newline, trying to read that line will return EOF instead of the text.

That's clearly a bug on the test server release since it breaks the old notecard reads too, you can very definitely read the last line of a notecard on actual production servers.

Possibly why the functions aren't available on the main grid yet even on RC servers?

[Anna Salyx]

3 minutes ago, Wulfie Reanimator said:

Noticed really annoying (but consistent) behavior I never knew about.

If the last line of a notecard contains text but no newline, trying to read that line will return EOF instead of the text.

This happens with Sync/Async and applies even to notecards that only contain one line of text, but no newline. The line cannot be read because the result is EOF. I'm sure someone has to have filed a jira about it, but what's the current status?

I saw that behavior noted somewhere else and I believe it's recognized as a proper bug. I tried a quick search in my usual places for seeing things like that but came up blank.  if I find it I'll update.

[Love Zhaoying]

5 minutes ago, Frionil Fang said:

With some (very) crude testing, the speed quite comparable between the two of them.

I didn't do any fine-grained or huge dataset testing, but reading a notecard with 10 lines of 1024 bytes completes within one frame (llGetTime() returns 1/45 seconds), reading the same data from linkset (keys = line numbers, values = lines) was more fluctuating but median 2 frames.

Reading 80 lines of 128 bytes, NCsync reads completed within 2-3 frames (median 2), LSD read completed within 1-2 frames (median 2).

Thanks!

[Lucia Nightfire]

11 hours ago, Love Zhaoying said:

If so, it seems we are overcomplicating things.

I've been thinking this for 3 pages now. 😆