LSL Documentation Generator

[agentronin]

I have been looking for the equivalent of the DoxyGen documentation generator used for C, and C++, that can be used for LSL. I found a promising one here:

https://www.naturaldocs.org/

It can be used to document code written in any language. In the case of LSL a Languages.txt configuration file must be created:

https://www.naturaldocs.org/reference/languages.txt/#adding_languages

I would like to hear from anyone who has experience using this with LSL. Also, before I create a Languages.txt file, I would like to see the Languages.txt file anyone else has already created.

[Quistess Alpha]

Since LSL doesn't have complex classes and such, and because memory limits make programs program size a good deal smaller than other languages, what kind of documentation do you really need?

with LSD, adding some extra comments about which keys are used where could be useful, and information about channel protocols could be helpful for allowing "plugins" to interface with other scripts, but I don't really see the benefit of generating that kind of info "automatically".

[agentronin]

Because LSL is multithreaded, is not Object Oriented, has an asynchronous data server, an LSL script can become very complex. The OpenCollar script is an example of how complex an LSL script can get. There is just as much a need for the ability to quickly create an HTML manual for these as there is for most any C, C++, C#, etc, program. There is a difference between reading documentation embedded as comments in the code, and being able read that documentation in an outlined, and organized, HTML, or CHM. I have very complex code I wrote that I need to start editing again. It has been a few years since I last worked on it so much has been forgotten. I have a need to have documentation in an organized HTML document to re-familiarize myself with what I did. This is much easier than reading it distributed throughout pages of code. 

I did thoroughly document. In the hope that someday DoxyGen would someday support LSL I wrote that documentation in accordance with DoxyGen standards. DoxyGen cannot generate from it because it is language sensitive. Now with Natural Docs I have means of generating this. It will mean time spent reformatting the documentation, but doing that will save me a lot of time during re-familiarization.

If someone else has already created a Natural Docs Languages.txt file I am hoping such a person will share it so I would not have to duplicate that effort. If I end up creating it, I will share it here.

[Quistess Alpha]

9 hours ago, agentronin said:

Because LSL is multithreaded

It's actually single-threaded, unless you consider each independent script a different "thread".

[Love Zhaoying]

5 minutes ago, Quistess Alpha said:

9 hours ago, agentronin said:

Because LSL is multithreaded

It's actually single-threaded, unless you consider each independent script a different "thread".

I was going to say..but didn't want to argue the point.

I've learned over the years, that script "clarity" is more about your own "code patterns" than anything else.

In my case, I use my own "code patterns" that leverage certain conventions - which make it pretty easy to tell what I am doing.  Because same convention is used in multiple scripts. Plus I add lots of comments.

It peeves me when people say, "XXX language is self-documenting", then they use things like "while (~--x)".

IMHO, you can't both "use compact, extra efficient coding styles" with C-style languages, and still expect most people to understand what you are trying to do, without verbose comments.

[Bleuhazenfurfle]

12 hours ago, Quistess Alpha said:

It's actually single-threaded, unless you consider each independent script a different "thread".

TL;DR — So is it "multithreaded"…  "kinda"?  But really no.

LSL appears to use a kind of "co-operative multitasking", more commonly referred to as "fibers" these days.  A fiber can yield control back to the script engine, allowing the next script in the queue to pick up where it left off — but there is indeed at most only one script executing at any given time.  And even more than that, there's probably a limit in that only one part of a given script can be executing at a given time (a common limitation with fibers, they tend to be fairly "granular").  In LSL, this yielding seems to come into play only in llSleep, the delays attached to many functions, and at the start of functions and loops, and maybe other similar spots.  My personal suspicion is that all of those are actually the exact same thing — essentially a call to the backing function of llSleep — and for some reason (possibly something to do with using icky actual fibers rather than much cleaner async/generators), LL seem highly reluctant to add any new yield points.  (I recently suggested one in the form of llAwaitDataserver, but I've seen others given the same response, too.)

Anyhow, sorry to belabour an off-topic, but I've seen this come up a number of times, and I thought it worth the clarification.

[Love Zhaoying]

It's just an asynchronous state machine with queued events. Nothing too fancy. 

[Bleuhazenfurfle]

2 hours ago, Love Zhaoying said:

It's just an asynchronous state machine with queued events. Nothing too fancy. 

Mmmmmmm…  mini-serving of buzzword soup…  Yummy.  Was trying not to hijack this thread, but, okay, since you bring it up…

TL;DR: I'd agree with you but for one point…  llSleep.  State machines in their own can't suspend mid-state-function (not to be confused with LSL states), so it's at least more than just that.  (It's mostly a problem with a little thing called a stack…)  And asynchronous applies only in so far that there's an event system present, it also does not address llSleep.  That one little function, is the bogeyman of your response, and pretty much all solutions are indeed quite fancy.

Interestingly, that prompted me to go get my favourite error message to slap you around with (gently, of course).  But it's changed from what I remember…  And the new one doesn't lend the same insight.  Whether that means they've switched from fibers to something else (hopefully async/gen — and may now indeed add new yield points), or not, I can not infer.

This isn't an exhaustive list, and terms get reused and mixed up often, not to mention they've changed over the years (seems like everything used to be just "multitasking"), but I'll try to lay out the important ones hopefully reasonably clearly… There is somewhat of a conceptual hierarchy here; networking -> multitasking -> threading -> fibers -> generators -> state machines -> functions.  (The ordering of generators vs. state machines is particularly arguable, but I think this ordering fits in here, and of course the whole lot is somewhat recursive with networking usually being implemented using a state machine, etc. — but a state machine usually won't be implemented using networking.)

"Fiber"s allow a normal stack-based program to essentially have multiple stacks, though "stack switching".  Those stacks are created explicitly (I've heard them referred to as "fiber boundaries"), which in this case, would almost certainly be the script as a whole (hence the granularity I mentioned).  The old error message strongly suggested this was the method in play in LSL.  This is quite fancy, and also the method I used when I wrote my first "multitasking library" some 30-odd years ago in good ol' Borland Pascal, and involves what C/C++ call a "longjump".  This idea is basically the same as "threads" (from the term multi-threadding) which are simply the OS version of fibers, and usually not co-operative (used to be, and can be, but the OS will additionally also step in periodically and suspend you, and even in some OS's do so on pretty much every kernel call), and then "processes" in turn encapsulate a group of threads together, along with other attached resources (the set of allocated memory most notably).  And that's not even mentioning multi-core systems, which seem to have taken over the definition of "multi-tasking".

Asynchronous is that the script doesn't just stop the world to wait for external tasks to finish, but rather starts the action, and makes a note of it to come back and attend to it later when it's done.  In LSL's case, that's handled by the script engine, and the insertion of a suitable event into the scripts queue, which the script engine subsequently re-invokes the script to handle at a later time.  It has no bearing on the scripts ability to suspend mid-run, as llSleep does.  The script "finishes" at the end of each event, returning control to the engine (and thus unwinding it's stack), before a new event can begin to be processed.  In this regard, it is very much a state machine, with each state being represented by an event handler (again, not to be confused with LSL states), but at no point does this address a state being suspended mid-execution.  A common implementation is that of "promises", but still requires some kind of external polling/checking mechanism, or "main loop", to which it returns (assuming nothing fancy).

Generators is a newer alternative to fibers; This idea, is about getting the compiler to do it for you — much more fancy.  One method is for the compiler to kind of turn a function inside out, effectively converting it into it's own little state machine — everyone does this themselves the manual way, every time they use an asynchronous function, and is the method often employed by compiled languages.  (This is also roughly related to how every functional language ever does it.)  I'm also most familiar with Python's version, though, in which each function invocation carries with it it's own private little perfectly sized mini-stack and "instruction pointer" (the package as a whole, called a generator), allocated on the heap along with all your other values (much less state-maciney, though) — most often employed by bytecoded languages such a Python (and perhaps Mono, though it feels more geared towards being just a brief stop on the way to compilation).  One quirk of generators (as opposed to fibers) is that yields can only occur in the bytecoded portions (not exactly, but close enough), and not within any C/compiled library functions it may call (also not exactly, but close enough — Python for example has a mechanism to allow it, but you have to do the state-machinery part yourself, essentially the same as what every LSL scripter does).  Either way, async (not to be confused with asynchronicity) is then typically implemented on top of the generator mechanism by utilising a generator yield point (quite often literally the yield keyword) to pass control back to the underlying script engine, rather than returning a value as such, as would typically be done with a generator.  The use of "async" and "await" keywords, kind of wraps up the generator so it doesn't behave as one, both providing and capturing the yeilded values for it's own use (there's nothing strictly preventing an "async generator", but that's a level of convolution that's likely to make heads explode).  In terms of granularity, the yield boundary for generators tends to be the individual function, and usually doesn't include functions it may call (hence fibers still have a place, unless you enforce a turtles all the way down policy with a little extra layering).

Hope that clears things up.  Any more questions?  🤪

Edited May 11, 2023 by Bleuhazenfurfle