Non Rendering avatars

[Jackson Redstar]

Is there a "trick" to force avatars to load? this is at a event, about 90 there, but cam zoomed and focused kinda tight so not that many on the cam view. was there at least 20 min before taking this screen shot. I catured both the texture console and statistics panel. Right click/texture refresh doesnt work force them to load, wonder if there is a way to force the viewer to render them?

[Coffee Pancake]

They aren't rendering because the viewer likely thinks they are rendered. whatever attachments they have, for some reason you've not been told .. perhaps a few packets got lost when you entered the region, but not sufficient to push the total over 0.0% 

PS .. while the texture console is useful, keeping it open has a notable impact on texture processing speed. Don't get in the habit of leaving it up as a kind of hyper verbose progress bar.

[Maitimo]

There's a couple of tricks that can help.

Teleporting away (to a different region) and back again. Your viewer will start again once you return to the original location.

Changing your active group tag (for some weird reason this also triggers your viewer to re-render the scene). This was a thing years ago but apparently it still works; I used it just yesterday.

Check your preferences to make sure your bandwidth settings are not too high or too low.  Either extreme will cause issues with rendering. The sweet spot will depend on your internet speed but you shouldn't ever need to go above 1500.

Lower your graphics settings and draw distance - this can help speed it up, and then you can raise them again once the scene is fully rendered.

 

[Jackson Redstar]

2 hours ago, Maitimo said:

The sweet spot will depend on your internet speed but you shouldn't ever need to go above 1500.

 

in 2023 - the 'sweetspot' for internet speed is 1.5 mbs - maybe that is a problem right there!

[Coffee Pancake]

26 minutes ago, Jackson Redstar said:

in 2023 - the 'sweetspot' for internet speed is 1.5 mbs - maybe that is a problem right there!

You misunderstand what that slider does.

Bandwidth does not cap or limit the speed stuff is downloaded, that goes as fast as your internet allows.

It caps the speed at which the server screams UDP data at you, UDP data is time sensitive and if you can't receive it all in time it gets junked. That's how UDP works.

Guess what is sent over UDP .. guess what there is a HUGE burst of when you enter a region .. 

 

Turn your bandwidth slider down, under 1mbs is really fine.

Edited January 29, 2023 by Coffee Pancake

[Jackson Redstar]

23 hours ago, Coffee Pancake said:

You misunderstand what that slider does.

Bandwidth does not cap or limit the speed stuff is downloaded, that goes as fast as your internet allows.

It caps the speed at which the server screams UDP data at you, UDP data is time sensitive and if you can't receive it all in time it gets junked. That's how UDP works.

Guess what is sent over UDP .. guess what there is a HUGE burst of when you enter a region .. 

 

Turn your bandwidth slider down, under 1mbs is really fine.

OK. Not knowing whay UDP is I looked it up. But I still wonder then in 2023 when internet speeds are often 100+ mbs download - we would need to limit download of these packets to 1.5 mbs. Suppose it makes sense but still seems rather outdated

[Coffee Pancake]

UDP is typically used for repeating information.

Like when you walk about in SL, there is constant stream of UDP packets telling everyone nearby where you are. if you miss a packet, it doesn't matter, it out of date already and here is a new packet. It's a very fast way to send updates.

TCP is the reliable protocol that everything else uses, but its slower as there is some back and forth chatter required. UDP is the server just throwing stuff at you, as most everything in SL is classed as an "object update" of some sort, UDP is used a lot.

TCP : hello .. hi yes .. i have some things .. thats nice, I will have 3 cakes please, did you ask for 3 cakes, yes please .... 

UDP : I needs cakes .. 🧁🎂 🍰  🎂 🍰 ☠️ 🧁🎂 🍰  🎂 🍰

[Wulfie Reanimator]

2 hours ago, Jackson Redstar said:

OK. Not knowing whay UDP is I looked it up. But I still wonder then in 2023 when internet speeds are often 100+ mbs download - we would need to limit download of these packets to 1.5 mbs. Suppose it makes sense but still seems rather outdated

My understanding is that most viewers don't have a proper queue for UDP packets, so if you allow all of them to arrive unrestricted, some will be ignored by the viewer because it was busy processing other packets.

LL's UDP "circuit" does use acknowledgements, meaning that certain missed packets will be sent again, but not all. I don't recall if object updates are made "reliable."

It's not exactly outdated, it's just lacking some features that could reduce the amount of packet loss. I recall @animats having implemented a packet queue in his experimental viewer, which solves an issue with missing objects mentioned in another thread.

Edited January 30, 2023 by Wulfie Reanimator

[Jackson Redstar]

48 minutes ago, Wulfie Reanimator said:

My understanding is that most viewers don't have a proper queue for UDP packets, so if you allow all of them to arrive unrestricted, some will be ignored by the viewer because it was busy processing other packets.

LL's UDP "circuit" does use acknowledgements, meaning that certain missed packets will be sent again, but not all. I don't recall if object updates are made "reliable."

It's not exactly outdated, it's just lacking some features that could reduce the amount of packet loss. I recall @animats having implemented a packet queue in his experimental viewer, which solves an issue with missing objects mentioned in another thread.

from what I read the server doesnt wait for your machine to handshake it just sends to packets and won't resend if missed. But I still have to question - these bandwidth specs were the same back in 2007 when if you were on a "fast" internent then it was probably up to 5mbs - and the typical computer back then was prob less than a 1/8th as fast as they are now.  So a decade and a half later there has been no improvement in delivering data via UDP?

 

[animats]

1 hour ago, Wulfie Reanimator said:

My understanding is that most viewers don't have a proper queue for UDP packets, so if you allow all of them to arrive unrestricted, some will be ignored by the viewer because it was busy processing other packets.

LL's UDP "circuit" does use acknowledgements, meaning that certain missed packets will be sent again, but not all. I don't recall if object updates are made "reliable."

It's not exactly outdated, it's just lacking some features that could reduce the amount of packet loss. I recall @animats having implemented a packet queue in his experimental viewer, which solves an issue with missing objects mentioned in another thread.

Right.

(Technical stuff follows.)

The original viewer was entirely single-thread. It dates from the era when few people had multiple CPU cores in home computers. Internally, the viewer has what are called "co-routines", which work something like Javascript callbacks or what's called "async" today. They're all stealing time from the main thread. So, if too much UDP data comes in too fast and is processed, the frame rate will suffer. This was a reasonable design twenty years ago, and many games and most web pages still work that way. But as you start to push for more complex scenes and higher performance, the limits of that approach show up.

ObjectUpdate and ObjectUpdateCompressed messages, which create objects, are reliable (they get re-transmitted if not acknowledged) but ImprovedTerseObjectUpdate, which says something moved, is not. The idea is that if you lose a move update, another one will be along soon, and you want the latest one.

Many of the messages are documented in the SL wiki. There's also the "message template file", the official documentation for the SL UDP message protocol.

Overall, the idea was to implement an eventually consistent system, where the viewer and simulator are not in lockstep sync but eventually agree on the state of the world. However, high speed eventually consistent systems were a rarity when SL was designed. The theory for doing this right wasn't worked out until about 2011. Until then, eventual consistency was mostly a banking concept - eventually the transactions settle and are reconciled. Banking is eventually consistent on a scale of days, not seconds.

The hard part about eventual consistency comes when you have messages A and B which can arrive in any order but which update the same object. If they show up in reverse order, that may have to be handled as a special case. (In  banking, this is where the withdrawal transaction shows up before the deposit transaction and the account looks overdrawn. Banks usually process deposits before withdrawals to try to minimize that problem, but it can still happen if a transaction is delayed in transit.)

In SL, this comes up in object creation. The ObjectUpdate which tells the viewer about a child prim can arrive before the one for the root prim. The child prim's location is sent as a offset vector and rotation relative to the parent, not in region coordinates, so it can't be drawn until the parent shows up. It's an orphan. So the viewer has to keep lists of orphans waiting for their parents to arrive. In practice, this happens all the time, but eventually all the parents do show up, and this is routinely handled. However, if you happened to be re-linking an object at the same time some viewer is finding out about it, I'm not sure that would consistently come out right at the viewer end.

This can also happen in the other direction. It turns out that the simulators are sensitive to the order of RegionHandshakeReply and AgentUpdate, which are viewer to sim messages. The "missing object" problem seems to be related to this.

Packet loss will result in out of order messages. If A is sent before B, and A gets lost, A will be re-sent, but will now arrive after B. SL's UDP message system does not guarantee in-order delivery. (Networking theory: In order delivery, reliable, no head of line blocking if a packet is lost - pick two. You cannot have all three. TCP picks the first two. SL UDP picks the last two. That means less lag on lossy network paths but more complexity, and bugs, associated with reconciling occasional out of order events.)

I have a vague suspicion that region crossing failures, especially of the type where the vehicle crosses and the avatar gets stuck, stem from an out of order message situation. But I can't confirm that.

Enough theory for one day. This gives a sense of why some intermittent funny stuff happens in SL.

[Theresa Tennyson]

21 hours ago, Jackson Redstar said:

from what I read the server doesnt wait for your machine to handshake it just sends to packets and won't resend if missed. But I still have to question - these bandwidth specs were the same back in 2007 when if you were on a "fast" internent then it was probably up to 5mbs - and the typical computer back then was prob less than a 1/8th as fast as they are now.  So a decade and a half later there has been no improvement in delivering data via UDP?

 

Radio operators haven't gotten faster at sending Morse code in over a hundred years. That's because there's a maximum speed you can send it and have it be understood by a human. Of course, there are many other ways of communicating now.

UDP = "Morse code;" TCP = "other ways of communicating." In Second Life, basically everything that can be improved by a faster connection is now sent by TCP.

Edited January 31, 2023 by Theresa Tennyson

[primerib1]

22 hours ago, Theresa Tennyson said:

 

UDP = "Morse code;" TCP = "other ways of communicating." In Second Life, basically everything that can be improved by a faster connection is now sent by TCP.

Not necessarily, no.

TCP requires buffer all along the way from sender to receiver. This can cause a phenomenon called "buffer bloat"

Some time-sensitive reliable protocols have been implemented over UDP, for example, Stream Control Transmission Protocol (SCTP)

(SCTP was originally designed to carry SS7 Telephony Signaling over IP ... and if you know SS7, you know that  not only it demands reliability, it is also very time critical. Due to SCTP's features, it's also used by some multimedia delivery protocols as well.)

((Also, SCTP is actually a Layer 4 Protocol, so it should be used in place of UDP or TCP, but because many network elements don't understand SCTP, there's danger that SCTP packets might be 'blackholed', so IETF created a workaround by wrapping SCTP inside UDP datagrams.))

Edit: Oof I misunderstood your statement. You were specifically talking about Second Life. In which case, you may likely be right.
My response where was for UDP usage in general ...

Also, Second Life probably can really benefit from using SCTP-over-UDP ...

Edited February 1, 2023 by primerib1

[animats]

I've started looking at what the server sends during the startup of a region, because I had a bug in my own viewer. Some few people might find this interesting.

Animats experimental viewer, during the first seconds of loading a new region, Vallone. Tracy profiler.

This is a timeline of what each thread is doing over time. You can see the rendering frames marked at the top. This viewer refreshes at a steady 60 FPS, and you can see the "Main thread" doing frames at that rate. Everything that changes the scene is in other threads. Bear in mind that this is not how the mainstream viewers work.

The main thread is only using about a third of the available time. All the action is at the beginning of the frame time, and then it waits for the display to catch up.

"Movement" is doing movement interpolation for moving objects. It's triggered to do its thing at the start of each rendering cycle, but the rendering doesn't wait for it. Note the little ticks below the "Movement" bar. The movement thread isn't very busy here. In a region with many moving objects, it could be far busier.

"Client" is receiving UDP messages. That's the interesting part here. UDP messages seem to come in groups of 5 or 6, spaced about 50 milliseconds apart. That's the sim server's pacing. The pacing keeps the single-thread viewers from overloading. That sending pattern and timing is consistent during the first 30 seconds, while the sim is still telling the viewer about everything in the region. It's on the conservative side, but the sim doesn't know much about how fast the viewer can accept updates. As the mainstream viewers get faster, that throttling might get some attention. You don't want a backlog, though, even if you have a queue. Although the Client thread here has a queue, if the queue builds up, interactive things like responding to key presses will get laggy. One option would be to separate messages into high priority ("Move this, now!") and low priority (new object updates).

Of the other threads, "Texture loader" is the controller for texture loading. It doesn't do the heavy lifting. Asset fetch #0..#4 actually make the HTTP requests and decode the data. Down at the bottom of the image, PriorityQueue is managing the re-ordering of texture fetches as the camera moves around. For this test, the camera was stationary, so it's not doing any work. QueueRegionMaterial makes material requests of the sim server. (Materials currently come from the sim server, unlike other assets, which come from the asset servers. This changes with the "materials viewer").

Not shown is the blast of UDP messages which contain the elevation data for a region. Those come in very rapidly at first connection, and you see that as a region's ground is drawn. No throttling there.

Although the protocol allows more than one SL message per UDP datagram, the sim servers do not seem to use that feature much. Maybe it was used back in the days of sending assets via UDP.

This doesn't explain why some things aren't loading. That's a separate problem. It just shows that the problem probably isn't UDP networking overload. That's throttled down to a rather low traffic level.

Now to go have some fun in world.

 

 

[Ardy Lay]

9 hours ago, animats said:

Not shown is the blast of UDP messages which contain the elevation data for a region. Those come in very rapidly at first connection, and you see that as a region's ground is drawn. No throttling there.

Does other UDP traffic from the simulator get lost during this blast?

[animats]

47 minutes ago, Ardy Lay said:

Does other UDP traffic from the simulator get lost during this blast?

Good question. I can't answer that for the C++ viewers. If that was happening, we ought to see pauses as a new region appears and the elevation data is filled in. Try flying over land and watch what's happening as new regions appear.

I went into it this deeply because the terrain-building process was taking several frame times and holding a lock, which causes an internal error to be logged. Extreme detail on this if anyone cares. Bear in mind that I'm using a completely different architecture than the C++ viewers, with many threads, and the timing constraints are very different. Berry Bunny's Crystal Frost project to build a viewer in C# using components from Unity will also have completely different timing issues. We're getting more diversity in viewers.

[Henri Beauchamp]

I never saw any UDP packet loss due to to high a UDP traffic (and I have my limit set to 16Mbps in my viewer) !

This is an urban legend that needs to cease.

UDP messages processing takes very little time when compared to the rest of the ”ancillary” tasks the viewers must deal with. One of the most heavy such task is by far textures prioritization, now that almost all the rest got pushed to threads.