Club viewer overload

[animats]

It's great that the sim side can now handle 90-100 avatars in a club. At last, we have crowds.

Avatar rendering, though, breaks down. I'm at a club with about 90 people. Firestorm is at 50-90 FPS in "Ultra". Memory usage 22GB system, 7.3GB in GPU.

Some avatars load without clothes, eyes, or hair. They never load properly, even after minutes. Network traffic drops to almost nothing. Somewhere, important updates got lost. Doesn't seem to be a BOM baking delay - tatoo layers show up, but eyes are missing. Some heads are on backwards.

Firestorm 6.6.8. We have a problem.

It's not Firestorm. This is the LL viewer 7.0.0.581125, Windows 64-bit, running under Wine on Linux. The same avatars are broken.

 

 

 

Edited August 6, 2023 by animats
LL viewer broken too.

[Wulfie Reanimator]

5 hours ago, animats said:

tatoo layers show up, but eyes are missing. Some heads are on backwards.

This probably isn't the point but -- those people are using mesh heads (which hasn't loaded, like most mesh in your pictures), almost definitely Evo X heads from the looks of it. They use BOM skins which don't match the LL head UV map, so what you're seeing is the LL body with a mismatching texture. 🙂 Just an expected side-effect of the wider issue of network traffic dying out.

Edited August 6, 2023 by Wulfie Reanimator

[Henri Beauchamp]

This is typical of a failed mesh skin download, and likely not the viewer's fault. In the Cool VL Viewer, you'd get something logged in this case, like:

2023-08-06 09:41:06Z WARNING: LLMeshRepoThread::run: Skin request failed for 576367a8-7916-0c74-1f74-80fd5704720a

Other usual suspects are with people who wear a new (suitable for their destination) outfit and immediately teleport to their destination without waiting for all attachments to actually attach/rez and/or for the server to fully bake their avatar: they might then see their avatar ”fully rezzed” on their screen, but others will not see some of the late rezzed attachments or bogus bakes (the latter may also bear a wrong UUID, causing failure by viewers to download the corresponding texture).

A long time ago, to help users with incomplete/in progress baking issues, I implemented in the Cool VL Viewer a small status icon (a red shirt) that appears while your avatar is baking. But I might add a warning dialog for when you TP away while not fully baked...

And yes, as Wulfie explained, the weird texture is due to a missing BoM mesh head that should be using it and make the legacy body head vanish.

Edited August 6, 2023 by Henri Beauchamp

[Zalificent Corvinus]

7 hours ago, animats said:

It's great that the sim side can now handle 90-100 avatars in a club. At last, we have crowds.

No, not really, the idea of an SL club with 100 avatars all mashing the "@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ Hoooooooooooooooo @@@@@@@@@@@@@@@@@@@@@@@@@@@@" button over and over is nauseating.

 

7 hours ago, animats said:

Avatar rendering, though, breaks down. I'm at a club with about 90 people. Firestorm is at 50-90 FPS in "Ultra". Memory usage 22GB system, 7.3GB in GPU.

Actually NO, Avatar rendering is NOT broken, as is your usual habit, you've grabbed the wrong end of the stick, again.

What's broken is server to client data transfer, all those pesky UDP packets, your client cannot render stuff unless the server tells it the UUID, so it can either find the thing in your cache, or suck down a copy off the CDN.

So what's the problem here?

"Dear Server, my idiot has TP'd into your domain at position <X,Y,Z> facing in direction D, with draw distance R, please supply uuid's of things they can see."

"Attention Client, stuff your idiot can see is as follows, Clubber #1, Maitreya Lara, uuid [data destroyed by cloudcrap], EvoX Head Lily uuid [data destroyed by cloudcrap] Truth Hair[ data destryoed by cloudcrap] Blueberry dress [data destroyed by cloudcrap,             Clubber #2: [[data destroyed],   Club[data destroyed].... [data destroyed]   [data destroyed]..."

 

Why is this happening in a over full region, because contrary to propaganda, SL regions do NOT handle that many avatars well, the new overloaded regions, in fact sseem to have a tendency to implode at about 110 avatars, crash and burn.

 

7 hours ago, animats said:

Some avatars load without clothes, eyes, or hair. They never load properly, even after minutes. Network traffic drops to almost nothing. Somewhere, important updates got lost. Doesn't seem to be a BOM baking delay - tatoo layers show up, but eyes are missing. Some heads are on backwards.

It's called "EvoX Head UV Mapping, you'd know that if you actually paid attention to what goes on in SL around you and what SecondLifers wear instead of assuming everyone is a system avatar in bad, low quality mesh clothing.

 

7 hours ago, animats said:

Firestorm 6.6.8. We have a problem.

It's not Firestorm. This is the LL viewer 7.0.0.581125, Windows 64-bit, running under Wine on Linux. The same avatars are broken.

Just to re-iterate, thiss iss NOT rendering failure on the part of ANY viewer, nor is it any nonsense about "teleporting before their new outfit baked" or any other SL Voodoo Superstition clap trap.

This is pure and simple, a case of Cloudcrap failing to correctly supply essential UDP packets, to clients, from the faulty Avatar-Overload super regions. It's an extension of the SAME problem that has buildings and objects not rendering when you log in or TP some place, sometimes for several mins, sometimes never, the whole "interest list transmission failure" thing.

 

 

EDIT:

Logically, this is an exponential problem.

10 avatars means the server sending of them details of 10 avatars, their own and the other 9 people.

10 Avatars, 100 sets of UDP avatar data packets

20 Avatars, 400 sets 

30 Avatars,  900 sets 

40 Avatars, 1600 sets

50 Avatars, 2500 sets

100 Avatars, 10,000 sets of UDP Avatar data packets.

 

Edited August 6, 2023 by Zalificent Corvinus

[animats]

I agree that there's a viewer-out-of-sync condition, but I'm not clear on why. This isn't a long-term network overload problem - that scene settles down into low network traffic while looking wrong.

Second Life is supposed to be what's called eventually consistent. That is, if server and viewer are out of sync, they're supposed to get back in sync eventually, or something has broken. Unfortunately, the theory of eventually consistent systems wasn't well worked out until the mid-2000s, and Second Life pre-dates that. (Look up "Merkel tree" for the theory.)

There are a few known holes in this. First, there are "reliable" and "unreliable" UDP messages. "Reliable" ones are re-sent if lost. But they are only re-sent a limited number of times before the sender gives up. Second, even if the actual network isn't losing any packets, both sender and receiver will deliberately drop UDP packets under overload, trusting in the retransmission system to send them again. Packets may be dropped on the receive side because updating is using too much of the frame cycle and the frame rate is dropping. This is why you can see a nonzero packet loss rate even on gigabit fiber. It's a desperation measure in single-thread viewers. So it's possible to lose even "reliable" messages.

Unclear if that's happening here. I suspect not, because it happens to the same avatars on repeated logins. This class of problem is very timing dependent.

Quote

It's an extension of the SAME problem that has buildings and objects not rendering when you log in or TP some place, sometimes for several mins, sometimes never, the whole "interest list transmission failure" thing.

The interest list being out of sync after login/TP is a different problem, unrelated to networking. When you first connect to a region, the viewer tells the server where it thinks the camera is. Then the server overrides that, moving the avatar based on last position, landing points, ban lines, land elevation, and solid objects. There's a brief period during which the server is sending object updates based on the original camera position. Then it switches to the actual position, and sends object updates from there. That sudden lurch seems to mess up interest list generation. I've seen this in both Firestorm and in my own Sharpview viewer, for the same objects.  In Sharpview, I put in a 2 second delay after login before requesting object updates, which seemed to work around that problem. I've mentioned this on the forums and discussed it at Server User Group. It's a recognized problem, but apparently hard to fix. Groans from the Lindens when interest list issues are mentioned.

Now that might be it. When you log in at "Last" position, the viewer has no idea where "Last" is. The server knows, and puts you there. The viewer's first guess for the location is arbitrary, probably the center of the region.

Interest list issues of this type tend to be repeatable. The same objects fail to show up on repeated logins. That's what's happening here with the missing heads.

In Sharpview, I put in a 2 second delay, after login, before sending the RegionHandshakeReply message, which starts object updates. That seemed to work around that problem. Someone working on one of the C++ viewers might want to try that. It's a hack, not a fix. If changing that timing affects avatar damage in crowded regions, that tells us what the problem is.

[Rachel1206]

I have tried to get the attention on this subject before in busy clubs. To me this is clearly a problem in the viewer code, as the following conditions seldom causes the above:

1. Arrive as first or among the first in a busy club/site.

All arriving will rezz correctly, at a busy site this of course can take longer.

2. Use low graphic setting as i.e. default and start with low draw distance like 16 meters.

Wait until avatars within draw distance has rezzed, increase in 8 or 16 meters intervals, wait for avatars within draw distance to rezz before increasing draw distance. When desired draw distance meet (be it 96, 128 or 256 meters), set graphics to your normal preference - I use high or ultra.

Avatars arriving after will rezz normally.

The problem is most pronounced if 40+ avatars in a club.

 

Tricks shared by hostess in clubs.

Change of group tag to none and then club tag seems to force an update, where avatars will in rezz after some times, which can be minutes.

Zoom out below your self, wait a little and return to normal view, none rezzed objects will rezz and often also will rezz missing avatars.

This only seems to work for ppl with low latency time - I normally have Ping Sim time of 188-224 ms and the tricks mentioned often do not work for me.

Edited August 7, 2023 by Rachel1206
Spelling

[Fionalein]

Can the answer to this riddle be as short as three letters? My guess yes, it possibly can!

U D P

I could do a joke on it but I'm not sure everyone would get it.

[Ardy Lay]

3 hours ago, Fionalein said:

Can the answer to this riddle be as short as three letters? My guess yes, it possibly can!

U D P

I could do a joke on it but I'm not sure everyone would get it.

Un-Deliverable Packet?  Yeah, I know the Internet uses "best effort" algorithms to deliver most traffic.  "Best-Effort" traffic is pretty much the first to be discarded when a queue's capacity is exceeded, and that happens constantly on the very "bursty" Internet.  I am very much aware of this as I have been forced to resort to leased Ethernet private lines to connect remote sites.  No amount of VPN and "predictive algorithms" and "forward error correction" applied by the person previously doing this work had been successful at getting the UDP data through intact.  When you are dealing with real-time, live, audio and video transport and don't have the luxury of delaying delivery until the lost packets can be resent, as is done with "reliable" UDP transports like SRT, you find other ways to succeed, or you fail.  End-to-end QoS is only available on "engineered paths" that currently cost more than "private lines".  Sometimes the simplest solution is the one that works.  You can take that both ways.

Zal's tactlessly delivered missive has some evidence that they have "been there, done that", at least theoretically.

I once got angry at being ignored on the same topic and rezzed a commercially available avatar head on the floor to demonstrate the problems caused when update traffic is sent to many agents.  One seemingly benign object was scripted with texture and geometry changes in such a way as to generate many "full updates" per second.  Multiply that by the number of agents present and BOOM, outbound network traffic volume increased to saturation at the simulator, and there were only 40 main agents and 12 child agents at the time.  I later repeated the deed using the same object and saturated a simulator with only 25 agents present, but it took slightly longer for people to realize something wasn't right.

There are many details to consider.  Most people have a tendency to preach their pet problem or solution as the truth, but I suspect that, with all the various capabilities of client devices and network paths being used, multiple solutions will have to be implemented as none will remove all the causes the way my one solution removed the one cause on my point-to-point application.

[animats]

12 hours ago, Rachel1206 said:

1. Arrive as first or among the first in a busy club/site.

All arriving will rezz correctly, at a busy site this of course can take longer.

2. Use low graphic setting as i.e. default and start with low draw distance like 16 meters.

Now that makes sense if it's an interest list startup problem. If you're there first, or you gradually increase your draw distance, that seems to be something the interest list system can handle. It's that sudden jump on arrival that breaks the interest list.

8 hours ago, Ardy Lay said:

"Best-Effort" traffic is pretty much the first to be discarded when a queue's capacity is exceeded, and that happens constantly on the very "bursty" Internet.

This doesn't look like a UDP overload or packet drop problem. It's too repeatable. The same avatars show up broken on successive logins.

I don't normally get any lost packets from SL. I have gigabit fiber from Sonic, which is far more bandwidth than SL needs. Even if UDP packets are lost, the important ones are supposed to be re-transmitted unless the server gets way behind.

Child prims not appearing on the interest list is a known bug. I've demonstrated that at Hyperion a few times. There's one big building which is a huge linkset. Parts of it frequently fail to appear. The same parts.

[Henri Beauchamp]

10 hours ago, animats said:

The same avatars show up broken on successive logins.

Either their mesh body cannot be downloaded (corrupted/missing file on a CDN server ?), in which case the viewer should log it as a warning, or they are simply not truly wearing the mesh (like I explained about people TPing away before their outfit has fully loaded and their avatar has fully baked).

[animats]

11 hours ago, Henri Beauchamp said:

Either their mesh body cannot be downloaded (corrupted/missing file on a CDN server ?), in which case the viewer should log it as a warning, or they are simply not truly wearing the mesh (like I explained about people TPing away before their outfit has fully loaded and their avatar has fully baked).

Not sure. Server User Group is cancelled this week, so I wasn't able to bring it up there.

Keep trying to figure out what's wrong. Right now it's too vague to file a JIRA.

I suspect that debugging this properly would require LL to send a hundred well-dressed bots to a region on the beta grid for load testing. Still, that's the price of success. At last, SL is getting crowds, and they almost work. This is a major win for SL.

[Odaks]

20 hours ago, animats said:

and they almost work

Reacting to the touch of sarcasm tempted me to add a laugh! I didn't though; I think it's great that LL are working to bring SL into the 2020's and recognise that that will be a major challenge. Your efforts to help are also greatly appreciated - good luck! Likedwise to the other contributers to the problems, of course.

Edited August 9, 2023 by Odaks
Afterthought

[OptimoMaximo]

On 8/6/2023 at 11:22 AM, Zalificent Corvinus said:

It's called "EvoX Head UV Mapping, you'd know that if you actually paid attention to what goes on in SL around you and what SecondLifers wear instead of assuming everyone is a system avatar in bad, low quality mesh clothing.

The bad and super low quality content is what goes around in SL.

[AmeliaJ08]

At a club with 92 people and yep, experiencing this. Some will just not load beyond their system avi no matter how long I leave it, it's like it has given up.

 

[Jackson Redstar]

I noticed this for a while and have been asking how to 'force' the viewer to render the half rendered people. the only solution I have found that sometimes works is derender the people - the go and remove them from the blacklist, and then switch groups to try to get the viewer to render them. sometimes it works, sometimes it don't

[Rachel1206]

On 8/14/2023 at 6:46 PM, Jackson Redstar said:

I noticed this for a while and have been asking how to 'force' the viewer to render the half rendered people. the only solution I have found that sometimes works is derender the people - the go and remove them from the blacklist, and then switch groups to try to get the viewer to render them. sometimes it works, sometimes it don't

I always wondered, why there is no optional "Refresh All Avatars" and "Refresh World" under the menu "World" - this would probably solve a lot of issues most users encounter in Second Life like deformed avatars and missing objects, missing parts of scene.

[Fionalein]

You can always TP to a busy shopping venue and back. Overload all* of your caches with new data. That way the club visitors will get dropped and you get a  new chance to render them right upon return.

 

*that's why clear cache won't work - it only clears one of them

[AmeliaJ08]

4 hours ago, Rachel1206 said:

I always wondered, why there is no optional "Refresh All Avatars" and "Refresh World" under the menu "World" - this would probably solve a lot of issues most users encounter in Second Life like deformed avatars and missing objects, missing parts of scene.

Yep. We know that SL often breaks when rendering a scene and the solution is to make it start over, really at this point there should be a way of forcing it with a click rather than the various tricks we use to make it do that.

 

Edited August 16, 2023 by AmeliaJ08

[Jackson Redstar]

If I remember correctly, in the old days there used to be a refresh scene/world on the viewer. Since most everything is server side now I wonder why there couldn't be a refresh/reload scene. But I am sure smarter people than me have thought of that and there are prob good reasons why it wouldn't work. Or even a right click on a avi and select a refetch/reload option to get the viewer to to re fetch the data for that avi. In most cases this is just an annoyance, but when filming the area with all these half baked avis, it becomes a real hassle

[Henri Beauchamp]

9 hours ago, Jackson Redstar said:

If I remember correctly, in the old days there used to be a refresh scene/world on the viewer. Since most everything is server side now I wonder why there couldn't be a refresh/reload scene. But I am sure smarter people than me have thought of that and there are prob good reasons why it wouldn't work. Or even a right click on a avi and select a refetch/reload option to get the viewer to to re fetch the data for that avi. In most cases this is just an annoyance, but when filming the area with all these half baked avis, it becomes a real hassle

This exists in the Cool VL Viewer.

You can right click on an avatar and select in the pie menu ”More>” and ”Refresh” to get the viewer to force-reload all textures and rigged meshes for that avatar. However it won't help when a mesh or texture fails to download at all (e.g. with 404 error due to the corresponding asset gone missing on the CDN server), or for avatars which UDP appearance message packets are incomplete (usually due to a missing mandatory wearable, or something in that vein: this is logged by the viewer when it happens).

There is also the ”Refresh visibility of objects” (ALT SHIFT R) feature, to make objects failing to render pop back up into existence; this feature is also auto-triggered on login and on sim change, since these often cause such ”missing objects” due to a race condition between the server interest list updates, the viewer FOV updates (the interest list contents sent by the server depends on the FOV sent by the viewer), and the render pipeline spatial groups rebuilds.

In the same vein, you can force-reload all textures for an object by selecting it (right click or edit) and using CTRL SHIFT U: this may help with blurry or corrupted textures.

Edited August 16, 2023 by Henri Beauchamp