Obscure question: when does the simulator send EstablishAgentCommunication to the viewer?

[animats]

One of those really obscure technical questions viewer developers sometimes need to know.

Background:

When logged into Second Life, the viewer is talking to one or more region simulators. It's always talking to the region your avatar is in , and it's usually talking to some nearby regions to see what's over there that needs to be shown on screen. This depends on draw distance.

When you first log into Second Life, your viewer starts by talking to the login server. The login server decides if you get to log in, and if you do, it returns enough information to let you talk to your initial region. This includes the region handle (which is just the X,Y location of the region), the IP address and port of the region for UDP messages, and the "seed capability", which is a URL used for requesting more URLs from the region. The "seed capability" URL has a randomly generated field for security reasons and is only good for the duration of the login.

That's the beginning of how things get started.

On a teleport, there's an event, TeleportFinish, sent over the EventQueueGet HTTP connection, with similar information about the teleport destination region. (Yes, the wiki shows that as a UDP message, but it was moved to the event queue years ago.) So that's how the initial phase of simulator handoff works.

Now, once you're connected to a SL region, you need to know about nearby regions so you can see them. So the main simulator your avatar is in sends EnableSimulator events telling you about nearby regions. These contain the region handle, the IP address, and port of the nearby region. For some reason, though, they don't contain the "seed capability" for the new region. So the viewer can't connect to the new region yet. There's probably some historical reason for this.

So there's another event, EstablishAgentCommunication, which provides that info. But, in my Sharpview viewer, I'm not getting that message from SL simulators. So Sharpview can't connect to nearby regions. I get EnableSimulator events for all nearby regions, but no EstablishAgentCommunication messages. That's weird, getting one but not the other.

EstablishAgentCommunication does show up in messages from the Other Simulator. Sharpview gets the info needed to connect to neighbor regions over there. There's code in the LL viewer to handle this message. It's shown on the SL wiki as coming in with EnableSimulator. So it ought to show up.

The message sequence is documented for the Other Simulator. Probably works the same way for SL. Diagram from 2009.

Sequence of events for a teleport in the Other Simulator for a region you can see. When a teleport fails, something in the above sequence didn't happen.

This shows the process of discovering a neighbor region, making it visible, and then, later, going there via teleport. Note the EstablishAgentCommunication message. I am not receiving that from SL regions in Sharpview. Do I need to send something to get the simulator to send it?

[animats]

More info:

There's another chart on that page, simply marked "Related", which shows this:

This looks like the procedure by which an EstablishAgentCommunication gets generated. Maybe. This shows the client (viewer) making a UDP circuit connection to the neighbor, and then an EstablishAgentCommunication message being sent from the simulator that has the user's avatar (the "main agent" is the term used in code). Now, why that UDP message to the neighbor simulator should be required, or even useful, is not clear. If that's an actual constraint, then there has to be some server-side inter-server communication by which Neighbor Simulator tells Simulator that Client has talked to it. This is just some random chart that happens to appear on the wiki for the Other Simulator, so I don't want to take this as authoritative. Especially since I know for sure that the Other Simulator does not need for the viewer to send UseCircuitCode before it sends EstablishAgentCommunication.

[Henri Beauchamp]

The ”RegionHandshake” message is likely what you missed: it should be received by your viewer after connecting to a region, and your viewer should reply to it via ”RegionHandshakeReply”.

I do not see those messages in the diagrams above...

In the C++ viewers, the handler for ”RegionHandshake” is set in llstartup.cpp (like other handlers). That handler is implemented in LLWorld::processRegionHandshake() and calls LLViewerRegion::sendHandshakeReply() for the corresponding region, the latter itself sending the ”RegionHandshakeReply” message back to the simulator server, after it got the objects cache loaded for the region.

Edited September 2, 2023 by Henri Beauchamp

[Beq Janus]

The scenario generalises for non-movement region connectivity, (such as a region coming into view because draw distance increased)

I think it boils down to something like this...

The solid lines are HTTP, with the narrow arrow heads inside the groups being the messages carried as the payload of the eventqueueget

dashed lines are LLUDP, where the solid arrow heads are marked as reliable.

The example is very simplified, it assumes only one region got added, but all that happens in the case of more regions is that the eventqueue payload has an Enable/EAC for each of the new regions. I am not showing all the other noise that may well appear.

 

[animats]

OK, I get that. Have to send UseCircuitCode and do the full region handshake over UDP to the neighbor to get the EstablishAgentCommunication message from the host region..

 

 

 

[Monty Linden]

This made me look into this part of the protocol design and I with I hadn't.  You may need to send some AgentUpdate messages into the host region to drive the neighbor region into building the EAC message.  Neighbor then uses the host region as a proxy to deliver it back to the viewer.  I really need to look at this someday...

[animats]

35 minutes ago, Monty Linden said:

This made me look into this part of the protocol design and I with I hadn't. 

I get that feeling, too

39 minutes ago, Monty Linden said:

You may need to send some AgentUpdate messages into the host region to drive the neighbor region into building the EAC message.  Neighbor then uses the host region as a proxy to deliver it back to the viewer.  I really need to look at this someday...

I think it's UseCircuitCode or RegionHandshake to the neighbor that's needed, but I'm not sure yet. AgentUpdate messages to the host region were already being sent and do not trigger an EstablishAgentCommunication message from the host region providing the seed capability for the neighbor.

Beq Janus and I have been trying to figure this out. Beq produced the event diagram above, after watching what Firestorm was doing. I'd been testing against the Other Simulator, which sends EstablishAgentCommunication immediately following EnableSimulator, without requiring any additional messages from the viewer. This differs from SL simulator behavior.

(I'm trying to figure out some way to formalize this with event diagrams like Beq's above, and state machines, like the one for TCP that's in Monty Linden's profile picture.)

[Monty Linden]

5 hours ago, animats said:

I think it's UseCircuitCode or RegionHandshake to the neighbor that's needed, but I'm not sure yet.

They're needed but they may not be enough.  If you find the EAC message delayed or still missing, try driving it with a few AgentUpdates after UCC and RH.  The Opensim version of this has got to be better than what I'm looking at.

[animats]

Thanks. I'll be trying that soon.

Meanwhile, I'm trying to document parts I already have working. Here's initial login:

If something above is wrong, please let me know. I'm going to put these charts on a wiki, or Github, or something, for the benefit of TPV developers and others who need such details.
(These are drawn using a free tool at "sequencediagram.org", if you want to draw some.)

A full set of charts would include:

• Login (above)
• Adding a neighbor region (above)
• Dropping a neighbor region from view.
• Teleport to distant region, with all-new neighbors
• Teleport to nearby region which is already a visible neighbor.
• Region crossing, non-corner.
• Region crossing, corner, i.e. a double region crossing.

You know, all the stuff that gives trouble.

[Beq Janus]

Stepping aside from the discussions slightly.

@Monty Linden I've been using sequencediagram.org (or a tool that it is based heavily upon websequencediagrams.com) for years to capture these types of charts in my RL work. Because the charts are source-driven, they are perfect for keeping in version control, but we also might want to think about refreshing some of the wiki.secondlife.com pages that haven't been updated since the days of the original AWG. 

[Monty Linden]

I'm all for better docs.  (I'm also a proponent of *any* docs.)  Sequence diagrams would be nice but they also have weaknesses.  They tend to show happy paths and miss recovery transitions.  Not all ordering shown in sequence diagrams is necessarily a constraint.  Some are, some aren't, and real docs are needed for the distinction.  But having the sequence diagram to help frame that doc would be a win.  Just takes work to extract that from the frankencode.  *cough*  I'm also looking for some better options for API exchanges (particularly for http transport).  Not certain llidl is the tool.

[Love Zhaoying]

I highly recommend updocs! 
..is joke!

[Monty Linden]

Not falling for that one...

[animats]

11 hours ago, Monty Linden said:

Sequence diagrams would be nice but they also have weaknesses.  They tend to show happy paths and miss recovery transitions. 

Right. A sequence diagram plus a state machine covers that. The trouble is, this system wasn't designed in terms of state machines, and trying to document it that way is difficult. What we see in the code (at least viewer side) is "here is message A, do this", "here is message B, do that", somehow adding up to a protocol. There's no explicit variable that says "we were in state X and are now transitioning to state Y'. There are also places where it's "must have both A and B, in any order, to do X".

Event sequence diagrams with comments are a good starting point. We can look at what messages go in and out, and write diagrams accordingly. That doesn't necessarily tell us what the constraints are, but it's a start.

Formalism is needed because it's hard to get your head around how something this complicated works. Whoever designed it probably had it all in their head, but two decades of changes later, nobody really does. Sequence diagrams are a concise way to describe what's happening. They don't describe all the possibilities, but just having a clear description of the happy path is a big help.

Most of the problems involving login, teleports, and region crossings getting stuck involve one of the players (viewer, servers) waiting for another one of the players to send something. The other player isn't sending it. If you can look at logs and event charts and say "things were on the happy path up to this step, and then stalled", that provides enough info to start work on the problem. This is all about making SL's long-standing, mysterious, and unsolvable problems less mysterious. And perhaps fixing them.

17 hours ago, Beq Janus said:

We also might want to think about refreshing some of the wiki.secondlife.com pages that haven't been updated since the days of the original AWG. 

I'll have a go at that, starting by putting the diagrams above and their source code in the SL wiki.

Edit: Starting to fill this in. See https://wiki.secondlife.com/wiki/Protocol#Protocol_Sequence_Diagrams

 

Edited September 7, 2023 by animats

[Henri Beauchamp]

You might be interested in the Cool VL Viewer v1.30.2.27 (or experimental branch v1.31.0.5) I released today: I revamped the messaging logging so that the DEBUG level ”Messaging” tag logs all the messages (with the exception of the super-spammy and pretty irrelevant ”PacketAck” one) exchanged between the viewer and the server.

You may toggle the ”Messaging” debug tag when needed (before or after login/TP/region change/draw distance change), from ”Advanced” -> ”Debug tags” in the login screen menu or ”Advanced” -> ”Consoles” -> ”Debug tags” in the main (post-login) menu.

Also, I implemented optional threaded object cache reads (which toggle is ”Advanced”  -> ”Cache” -> ”Threaded object cache reads”), meaning the viewer won't block to read object cache file(s) after a ”RegionHandshake” message from the server and will keep processing other messages, replying with ”RegionHandshakeReply” only after the object cache file(s) (the 's' is for the PBR viewer branch) has(have) been read: this enlightens some interesting ”aspect” (bug ?) of the server messaging algorithm. In particular, you will see that the server sends two (!) ”RegionHandshake” messages without even waiting for a first ”RegionHandshakeReply” or some timeout, excepted on login (where only one such message is sent by the login region server, unsurprisingly and as should ”normally” be the case)...

Edited September 9, 2023 by Henri Beauchamp

[animats]

Back to connecting to a region.

OK, so I have to do more before I get to find out the seed capability. The headache is that if the viewer sends an AgentUpdate and a RegionHandshakeReply to the new  region, it should start getting ObjectUpdate messages. But it can't start processing those fully until it has a seed capability and the neighbor region's capabilities, because it can't start the asset fetches without those. So those have to be deferred until the capabilities are fetched. I wasn't expecting that constraint, and it's a pain, involving a defer queue.

Can I avoid getting object updates from the new region before I've received the seed capability? If I have to handle the hard case above, so be it. But do I have to?

The way the Other Simulator does this is much cleaner. You do all the communication with the current region before you start talking to the neighbor region. There, you get EstablishAgentCommunication from the current region before doing anything with the neighbor region.

(Why does EstablishAgentCommunication even exist, anyway? All it contains is the IP address and port, which was already delivered with EnableSimulator, and the seed capability. Just putting the seed capability in EnableSimulator would have been much simpler. Sending them at different points in the sequence hints that it sent to prevent the viewer from doing something too soon. If so, what? Starting the EventQueueGet poller too early?)

Edited September 9, 2023 by animats

[Monty Linden]

Pulling out the true story is going to require reverse engineering and documentation on our part.  Henri's logging is something that should probably have been available in the SL viewer from day 1.  But here we are.  And as always, never mistake anecdotes for a contract.  You might see the same pattern ten times in a row but that doesn't necessarily mean there's a temporal constraint.  Especially in our code.

You may not need quite as much deferred logic as hinted at.  Once you send the UCC into the neighbor, there will be a RegionHandshake in your future.  You may be able to delay the RHR (I mean, it's async anyway so pretend you're on Mars).  What I'd meant before was that after sending UCC (and probably better after RH receipt), send AgentUpdates into the Host region.  This is one of several paths that drives a private protocol between simulators that causes a neighbor to emit the EAC message, which is routed back via the host region, that you need to begin HTTP operations with the neighbor in anger.

As always, you shouldn't entirely trust me on this.  I haven't walked through all of this to verify what's accidental and what's a true constraint or dependency.  HTTP was bolted onto the old UDP message system.  It's a Frankenstein monster.  'AgentCommunication' isn't just an abstract concept in the simulators, it's a physical class LLAgentCommunication and they are tied together (also hints that the people who did this are not the ones who should have been allowed to do it).  This class 'encapsulates' all the capabilities as well as the low-level plumbing for HTTP bridging of LLMessage comms.  This thing isn't actually tied to an agent strongly.  Nor to circuits (it doesn't really know about them).  It can live on after an agent has departed.  And yet the viewer has the ability to tear down this entity via arguments to 'EventQueueGet' requests.  (This resets the 'id'/'ack' value on the query among other things.)  It's quite the thing.

[Love Zhaoying]

10 minutes ago, Monty Linden said:

Once you send the UCC into the neighbor, there will be a RegionHandshake in your future. 

 

[animats]

19 minutes ago, Monty Linden said:

And as always, never mistake anecdotes for a contract.  You might see the same pattern ten times in a row but that doesn't necessarily mean there's a temporal constraint.

Exactly. Situations like that probably underlie some fraction of login, teleport, and region crossing failures. When you see those viewer-side, they take the form of waiting for something that didn't show up. "Why" is usually a mystery, although with suitable logging, you can see where things went off the "happy path" and got stuck.

I'm glad this is getting attention.

20 minutes ago, Monty Linden said:

Once you send the UCC into the neighbor, there will be a RegionHandshake in your future.  You may be able to delay the RHR (I mean, it's async anyway so pretend you're on Mars).  What I'd meant before was that after sending UCC (and probably better after RH receipt), send AgentUpdates into the Host region. 

Ah. AgentUpdate messages are already being regularly sent to the host region simulator at this point, since that region is already live and the agent is present there. No problem there.

What starts object updates from the neighbor region? Sending it an AgentUpdate? If so, that's good, because that can be put off until  the region handshake and capability fetch exchanges are out of the way. (The chicken-and-egg problem with the first AgentUpdate seen at login and teleport doesn't apply here; we know where the camera is when connecting to a neighbor region.)

Quote

You may not need quite as much deferred logic as hinted at.

Might, might not. What else server side sends an asset UUID to the viewer and wants an asset fetch started before we know the seed capability? Land textures have UUIDs, for example. And they seem to come in before object updates start. I probably should bite the bullet here and be prepared for arbitrary delays in getting seed capabilities and the caps obtained from them. I've seen Firestorm logs with errors in capability fetches, so this isn't a theoretical risk.

All of this is much easier if it goes 1) connect up new region and get all info needed for communication, 2) tell it to start sending content. When those overlap, as they do at login and teleport, it's harder.

1 hour ago, Monty Linden said:

And yet the viewer has the ability to tear down this entity via arguments to 'EventQueueGet' requests.

Those are used for something? For a while, I was sending the string "TEST" instead of the proper LLSD message with ID number, and it still worked. Sequentially numbered events showed up. (SL has consistent event ascending numbers from 1; the Other Simulator will sometimes skip a number, and starts from some arbitrary large integer.)

The event queue has ordered reliable delivery, which is useful where order matters.

[Monty Linden]

46 minutes ago, animats said:

Those are used for something? For a while, I was sending the string "TEST" instead of the proper LLSD message with ID number, and it still worked. Sequentially numbered events showed up. (SL has consistent event ascending numbers from 1; the Other Simulator will sometimes skip a number, and starts from some arbitrary large integer.)

The event queue has ordered reliable delivery, which is useful where order matters.

Well, LLSD is meant to allow slop (c++ and python implementations particularly).  But there are arguments other than id/ack in the body.  They're not currently used in the SL viewer so documentation is diddly squat.

The whole id/ack thing is badly done.  It should have been treated more like ETag.  As it is, old events can get a new id and other sins.  I think the incremental nature is accident, not contract.  And it will reset to 0 in a region/agent pair in certain circumstances (including that viewer-initiated reset).  So, beware.  Hope to document this.

And it's not necessarily ordered-reliable.  The event queue was very deliberately designed to implement event compaction/overwrite in place and, at some point in time, it was neither ordered (between events) nor reliable.  Someone disabled that UDP-like behavior at some point and I'm not certain why and if that has contract status at this point.  In fact, it is still unreliable now as we'll drop queue overruns on the floor.  They do not get regenerated unless a higher-level application protocol times out and causes a regeneration from a new query or other re-synchronizing action.

Future hope is to homologate this in-place and as-is with minimal optional extensions that would also allow each end to monitor how badly the other side is messing up.

 

[animats]

49 minutes ago, Monty Linden said:

And it's not necessarily ordered-reliable.  The event queue was very deliberately designed to implement event compaction/overwrite in place and, at some point in time, it was neither ordered (between events) nor reliable.  Someone disabled that UDP-like behavior at some point and I'm not certain why and if that has contract status at this point.  In fact, it is still unreliable now as we'll drop queue overruns on the floor.  They do not get regenerated unless a higher-level application protocol times out and causes a regeneration from a new query or other re-synchronizing action.

Ouch. Need to think about the implications of that. With reliable UDP messages, at least you get a few retries. I'm aware that UDP message dropping under overload can occur on both sender and receiver side, and that those are usually repaired by retransmission. But if the event queue drops messages under overload, there's no similar backup.

Some messages you never want to drop. If an EnableSimulator is dropped, the viewer has no idea that it hasn't been told about a region, and will never do anything to discover it. That makes me think of those rare situations where you reach the edge of a region, and the other region isn't showing, even though it's up. That was much more common before AWS uplift.

I haven't looked at the side of communications that involves group messages and such. Beq Janus is the expert on that and might have something to say.

49 minutes ago, Monty Linden said:

It should have been treated more like ETag.

As in RSS? Right, there it's interpreted as "I've seen and processed everything through #117, send me anything from #118 and later." If you lost the TCP poll connection with data in transit, there might be loss. The sender side has to be be careful about the write completion and close status on the TCP connection for this to be airtight. The TCP protocol knows on the write side if there was a clean close with all data delivered, but whether that info makes it through the socket interface is questionable. Or whether that even still works. There's a good chance that info does not make it all the way up through the HTTP and HTTPS protocol stacks, because web servers don't care if they delivered the entire web page. So you probably do need the ability to go back one ID number.

Incidentally, on a timed out event poll, SL does not send the documented 502 status; it just closes the connection. (The Other Simulator does send a 502). Timeout occurs around 58 seconds on both SL and the Other Simulator.

Since asset sending over UDP is finally dead and removed, UDP traffic isn't that heavy. It's probably appropriate to go more for reliability and worry less about overload.

Edited September 10, 2023 by animats

[animats]

It looks like there is a state machine inside, fighting to get out. Login, connection to neighbor, and teleports all follow a pattern. So here's an attempt to express this as a state machine, to understand the constraints on what can happen when.

A region connection state machine (first draft)

IDLE state - no connection

DISCOVERY state - finding out or being told about a region to connect to

• User makes login request to login server, or
• Home region server sends EnableSimulator to viewer to display a nearby region, or
• Home region server sends TeleportFinish to viewer. (I think)

CONNECTING state - actually making the connection to the new region.

• State entry requires knowing requires knowing IP, and port of new region.
• Seed capability comes in via EstablishAgentCommunication
• Set up UDP circuit.
• Do RegionHandshake and reply.
• Fetch capabilities.
• Start event poller.

CAMERA_AIMING state - agreeing on the viewpoint.

• Viewer and server have to agree on where the camera is.
• This is the messy part where, for login and teleport we need the ObjectUpdate for the avatar so the viewer calculate the camera location, but can't get it without sending a dummy location first in an AgentUpdate to start object update sending. This state can be skipped for neighbor connections, because the camera isn't moving.

LIVE state - fully connected and operating normally.

• AgentUpdates going out from the viewer, ObjectUpdates coming in, region live, user interacting and happy.
• Enjoy SL.

FAULT state - something went wrong.

• Invalid message or timeout in DISCOVERY, CONNECTING, or CAMERA_AIMING state.
• Log problem, maybe send report, including event sequence that got us here, to a server.
• For login, go back to login page. For teleport, teleport has failed, try to continue from old location. For neighbor region connection, display a region down hole in the ground.

Discussion

This almost fits what's happening.  Almost. There are are two known ordering problems:

• The seed capability comes in later than we'd like viewer side. It would be nice to have the seed capability before the first thing that needs an asset. That would be the terrain textures in the RegionHandshake. But RegionHandshake must precede EstablishAgentCommunication (at least for SL; the Other Simulator sends it earlier). Some object updates and terrain info come in before we have a seed capability. So asset requests have to held until the region transitions to LIVE state. Not too bad. 
• Timing of RegionHandshakeReply is touchy. Deliberately delaying RegionHandshakeReply by 2 seconds seems to prevent login interest list problems. This suggests it should be sent later in the sequence. Comments on how that works server side would be helpful. 

There's an underlying model to all this. I think. What have I missed?

Can we fit region crossings into this model?

Edited September 10, 2023 by animats
Clean up whitespace

[Monty Linden]

On 9/10/2023 at 12:43 AM, animats said:

As in RSS? Right, there it's interpreted as "I've seen and processed everything through #117, send me anything from #118 and later."

No, simpler, as in HTTP.  To support very basic 'If-Match'/'If-None-Match' tests.  The one special case is where the simulator side resets and this may need to be acted upon. 

On 9/10/2023 at 12:43 AM, animats said:

Incidentally, on a timed out event poll, SL does not send the documented 502 status; it just closes the connection. (The Other Simulator does send a 502). Timeout occurs around 58 seconds on both SL and the Other Simulator.

This keeps getting worse the more I look.  So *both* viewer and simulator implement a 30-second timeout on these requests.  A guaranteed race condition.  Simulator's spans a different part of the request lifecycle than does the viewer's curl timeout.  More variability.  Simulator can send at least three different responses related to its timeout actions:

• A 200 status with empty body and no content-type header
• A 200 status with llsd content-type and a body of:  '<llsd><undef /></llsd>'
• A 499 status with html content-type and a body I haven't looked at but it's about 400 bytes.

Between viewer and simulator is an apache stack that adds a layer of interpretation, time dependencies, and other factors to something that's already a hodge-podge.

If a 502 is returned anywhere, it's probably an accident.

[Henri Beauchamp]

1 hour ago, Monty Linden said:

This keeps getting worse the more I look.  So *both* viewer and simulator implement a 30-second timeout on these requests.  A guaranteed race condition.  Simulator's spans a different part of the request lifecycle than does the viewer's curl timeout.  More variability.

Do have a look at the comments I added in linden/indra/newview/lleventpoll.cpp, in the Cool VL Viewer sources, for the various modifications I implemented to deal with both SL and OpenSim idiosyncrasies... In particular:

	LLAppCoreHttp& app_core_http = gAppViewerp->getAppCoreHttp();
	// NOTE: be sure to use this policy, or to set the timeout to what it used
	// to be before changing it; using too large a viewer-side timeout would
	// cause to receive bogus timeout responses from the server (especially in
	// SL, where 502 replies may come in disguise of 499 or 500 HTTP errors)...
	// HB
	mHttpPolicy = app_core_http.getPolicy(LLAppCoreHttp::AP_LONG_POLL);
	if (!gIsInSecondLife)
	{
		// In OpenSim, wait for the server to timeout on us (will report a 502
		// error), while in SL, we now timeout viewer-side (in libcurl) before
		// the server would send us a bogus HTTP error (502 error report HTML
		// page disguised with a 499 or 500 error code in the header) on its
		// own timeout... HB
		mHttpOptions->setTransferTimeout(90);
		mHttpOptions->setRetries(0);
	}

Yes, it is indeed as bad as it looks... This said, my modified code performs just fine in both SL and OpenSim now, and the failed TP issues still seen are not related with event polls anyway (event polls are simply retried on timeouts).

Edited September 12, 2023 by Henri Beauchamp

[Monty Linden]

53 minutes ago, Henri Beauchamp said:

Yes, it is indeed as bad as it looks... This said, my modified code performs just fine in both SL and OpenSim now, and the failed TP issues still seen are not related with event polls anyway (event polls are simply retried on timeouts).

Well....  you don't really know that unless what you've received matches what was intended to be sent.  And it turns out, they don't necessarily match and retries are part of the problem.  I already know event-get is one mode of TP/RC failure.