The era of 150+ avatars in a region is here. We have a performance problem viewer side.

[animats]

Today the new Motown region passed 160 avatars. At 130, Firestorm was crashing on an 8MB machine. Avatars showing up nude and with heads on backwards. Firestorm ran fine at 80 avatars on a 32 MB machine with an NVidia 3070 GPU. With 80 avatars, Firestorm was using about 16 GB of RAM.

It looks like the "mega event region" thing mostly works server side, but the viewers are going to need some work. Merely pulling in the draw distance isn't enough for a crowd facing a stage. When the draw distance is short enough to de-render some of the crowd, you can't see the stage. Impostoring all but 3 avatars did not help. "Low" quality did not help on a smaller (8 GB) machine. There's nothing left to turn down.

I did a little testing with Sharpview, my experimental viewer, where avatars are just blocks for now. No problems with avatar overload or movement. So server and network are doing fine. Rendering all that clothing is the problem.

This is going to take some thought, and then considerable work, to get right.

 

[Qie Niangao]

2 hours ago, animats said:

Impostoring all but 3 avatars did not help.

That's a problem, right? I don't know viewer code, but it "feels" as if imposters save only rendering, not loading, so if full avatar complexity is kept in memory even for imposters, that seems like low-hanging fruit. (Or maybe the problem is we're devoted to having imposters instantly restored to full avatar glory if proximity changes?)

Also, whatever's gobbling up memory also must ride the network, another reason to trim. Imagine a mega event on mobile.

[Love Zhaoying]

Does the Official viewer have the same issues? 

[Coffee Pancake]

6 hours ago, Love Zhaoying said:

Does the Official viewer have the same issues? 

Yes.

 100%

The problem with avatars is the entire system is built around the assumption that people wont be wearing much of this new fangled mesh.

When this was immediately demonstrated to be wrong .. LL did .. nothing, and then kept doing nothing

There might be some tweaks, but fundamentally the system is unchanged.

Every attached rigged object mush be processed on the CPU. Which scales linearly. 

[Henri Beauchamp]

9 hours ago, animats said:

Today the new Motown region passed 160 avatars

Drat !  I missed an excellent occasion to stress-test my viewer... 😜

9 hours ago, animats said:

Firestorm was crashing on an 8MB machine.

I suppose you meant 8GB... But any viewer will have issues fitting any ”heavy scene” on a PC with only 8GB of RAM (especially since at least 1GB of that RAM will be used by the OS and services it runs). This said, on the condition you reduce your ”Texture memory” (and switch off the buggy ”Dynamic” setting, for FS) to 512MB, and reduce your draw distance to 64m or so, you might be able to attend large venues with a 8GB PC.

Also, under Windows, do make sure to allocate a large swap file (say, 16GB) of a fixed size else that Mickey Mouse OS will reallocate the swap file to fit the viewer's used memory and will return NULL pointers while doing so should the viewer request an allocation ==> Crash !

 

I would also suggest testing the Cool VL Viewer in the same tight-memory conditions: I worked a lot on the textures fetching/prioritizing/decoding algorithms, and it will use much less texture memory than other viewers while avoiding trashing and blur everywhere...

 

But let's face it. For today's SLing, 16GB of RAM is a strict minimum (with 32GB highly recommended).

[Jackson Redstar]

I know though from testing the official viewer handles high avatar loads WAY way better than FS, at least in terms of FPS, It is double if not more FPS than FS. It just well can't really use the official viewer very well for video recording, And the 8gig VRAM most good cards have now - that maxes out now with 30 or so avies in an event. I also know on Win 11 at least, you will want at least 32g of RAM, 16 is no longer enough

[animats]

9 hours ago, Qie Niangao said:

imposters save only rendering, not loading

Yes. Impostors are still rendered, just infrequently and to a flat billboard. Then the flat image is shown for a quarter second or so. Everything needed to display the full avatar has to be loaded.

SL mesh avatars don't really have useful levels of detail. All parts of the avatar have the same level of detail, so every tiny accessory is at full detail. The trouble is, mesh clothing relies on that. Many clothing items disappear entirely below high LOD. I discovered this when I was dressing animesh NPCs. Animesh get normal level of detail handing, and you get to see all the bad lower LODs. Plus, all the careful tweaks that got the mesh clothing layered properly fall apart at lower LODs.

That's why the overload exists.

This is a hard problem. If SL is going to have big crowds, somehow, we have to simplify non-close-up avatars more.

[Henri Beauchamp]

1 hour ago, animats said:

SL mesh avatars don't really have useful levels of detail.

They do have (there are 4 LODs for meshes) but, sadly, LL did not implement any serious incitation (massive upload cost increase for missing/bad LODs) for creators to use the mesh LODs properly, and the decimation algorithm in the viewer upload tool for automatically creating the ”missing” lower LODs was, and still is, sub-par. The result is that for most mesh objects seen in SL, their first 3 LODs have about the same rendering cost (with just the lowest LOD sacrificed to a few triangles to lower the upload cost, but you would pretty much never see it rendered anyway).

In the defense of LL, this is (mathematically) a very complex issue: generating missing lower LODs efficiently for all types of mesh objects at all viewing distances is an extremely difficult task, that even the most advanced contemporary algorithms cannot satisfactorily cope with.

This is why there should be a strong incitation for mesh creators to produce well optimized hand-made lower LODs for their mesh objects (so far, the human brain of talented and experienced creators is still far superior to any existing algorithm, or even any current AI)... The drawback would be that amateur/occasional mesh ”creators” like me would be (very) badly impacted (what ?.... L$10000 for that small mesh upload ?.. But... but ! 😲 ).

Anyway, it is too late... I mean, hundreds of thousands (or perhaps even millions) of mesh assets have already been uploaded to SL, and we must now cope with them.

1 hour ago, animats said:

If SL is going to have big crowds, somehow, we have to simplify non-close-up avatars more.

+1000 (no, not in LI units 🤪).

This is indeed the way to go... This is a viewer-side thing, and totally doable (I envision pre-rendering such ”far” or ”not too close” avatars in a thread, only a few times per seconds, and rendering the result in the main thread on next frames as a flat image, till the next lazy threaded update); this will however have to wait till a multi-threaded renderer is implemented (Vulkan, you know)...

Edited June 21, 2023 by Henri Beauchamp

[Kathrine Jansma]

2 hours ago, Henri Beauchamp said:

this will however have to wait till a multi-threaded renderer is implemented (Vulkan, you know)

Just in theory, in a system with two GPUs (e.g. CPU like AMD 7xxx with built in graphics + dedicated graphics card), could one use offscreen rendering on the secondary GPU for that too?

e.g. with https://registry.khronos.org/OpenGL/extensions/AMD/WGL_AMD_gpu_association.txt or https://registry.khronos.org/OpenGL/extensions/NV/WGL_NV_gpu_affinity.txt 

?

[animats]

4 hours ago, Henri Beauchamp said:

well optimized hand-made lower LODs for their mesh objects

Won't help for layered clothing. The "peek through" problem, where inner layers go through outer layers, is different for each LOD. That's a property of all the layers, not each individual one.

[LittleMe Jewell]

Draw distance is from the camera, isn't it?  So set draw distance really low, zoom in so that you are only viewing mostly stage and then derender all non-friends if there are still people in the view.

Hmm - never mind. That would derender all the folks on the stage also.  Though you could still do the really low draw distance and zoom you camera in.

Edited June 22, 2023 by LittleMe Jewell

[Jaylinbridges]

1 hour ago, LittleMe Jewell said:

Hmm - never mind. That would derender all the folks on the stage also.  Though you could still do the really low draw distance and zoom you camera in.

Just send all the Lindens and Moles a friend request. Problem solved   You can not go below 64m DD in the SL viewer.

I struggled last year at the SL19B opening speech with my old potato system.  Only 16 GB system memory and 2GB VRAM.  I also attended as a low lag kittycat, 10K complexity and no mesh anywhere.  I had everything turned down that I could in graphics.  They had the large capacity regions then too, but it was all client side lag that caused so many to crash after a few minutes there.  Patch lost audio inworld, and lindens were crashing on stage too.   I don't expect much improvement on June 22 at noon, slt, this year.

I ended up watching it on YouTube, after an attempted double click TP landed me on stage, instead of the front lawn, and I could not move away.  After a nasty IM from one of the Linden enforcers to "get off the stage", I hit the hardware reset button on the computer to get free, and logged back to the You Tube stream.

Edited June 22, 2023 by Jaylinbridges

[LittleMe Jewell]

2 hours ago, Jaylinbridges said:

You can not go below 64m DD in the SL viewer.

Ugh - I can go down to 32m in Firestorm.  And I have quite a few 3k and under outfits.

[Leora Greenwood]

8 hours ago, LittleMe Jewell said:

Ugh - I can go down to 32m in Firestorm.  And I have quite a few 3k and under outfits.

I was looking for some of my old low-complexity outfits yesterday but found none except tinies and small animals, or just dumping my current body, head and hair.  😢 😿

[Jaylinbridges]

I had no hope at Patch's opening speech.  I went from 60 fps to 3 fps as soon as the crowd of 200 avatars was in range.  My 2GB of VRAM maxed out, though I restrict textures to 512KB, and the system memory started to page.  Was all over in 5 mins - had to force a logout.  I was using the latest FS beta, with everything turned down.  Nothing helped.

I will try other viewers at the crowded events in the days ahead.  Keep the avatar count under 60 and I get along fine.  Is it worth it to spend $1500 to upgrade my system so I can attend some mega lag event?  

[Henri Beauchamp]

On 6/23/2023 at 5:31 AM, Jaylinbridges said:

I had no hope at Patch's opening speech.  I went from 60 fps to 3 fps as soon as the crowd of 200 avatars was in range.  My 2GB of VRAM maxed out, though I restrict textures to 512KB, and the system memory started to page.  Was all over in 5 mins - had to force a logout.  I was using the latest FS beta, with everything turned down.  Nothing helped.

I attended the event without much trouble. I arrived 1H30 before the speech and idled there for 1H15 with avatars piling up left and right with 200+ reached 20 minutes before the show. I then noticed that my VRAM was starting to get full (no issue rendering, but high discard bias, around 4.5, as a result): there was obviously a leakage of GL textures...

So I relogged to cleanup the VRAM and could enjoy the show at 50+ fps...

This said, this was with a high end PC: Ryzen 7900X, 64 GB RAM, RTX 3070 (8GB VRAM), under Linux.  The viewer reached 24GB or RAM usage and the VRAM was kept almost full (the viewer makes it so that the textures discard bias is adjusted to let around 800MB of VRAM free). The graphics settings were kept very high, maxed out excepted for shadows (off), ”Terrain and trees” water reflections only, mid WL sky quality setting, mesh LOD multiplier (a Cool VL Viewer boosting feature for badly designed meshes) reset to 1.0 (i.e. meshes LOD = objects LOD * 1.0 = 3.0), 256m draw distance (reduced at 128m just before the show, but all avatars were within that distance anyway), non-impostors number reduced to 12. The fps rate was good at 50 and over (with the rate limiter set a 60fps), with just a few dropouts every now and then while avatars rezzed around during the wait for the show, or when camming during the show.

I since debugged the GL textures leakage (4MB per avatar, due to ”no delete” local texture objects, i.e. texture bakes), so tomorrow's release of the Cool VL Viewer (v1.30.2.18) will behave waaaay better with 200+ avatars around. 😜

This said, the viewers textures management code is a pile of spaghetti code that would benefit greatly from a full rewrite from scratch !..

 

Edited June 24, 2023 by Henri Beauchamp

[animats]

Possible approach to avatar overload - textured jellydolls.

We all know the jellydolls that appear when an avatar is over the complexity limit.

Avatars in distance, normal.

 

Avatars in distance, jellydoll mode. See the difference? Look hard.

If jellydolls had a texture created from the real avatar via the impostor rendering system, you'd barely be able to tell at this distance. You'd get an overall impression of the character and could pick out individuals. But if an avatar is a tenth of the screen height, it doesn't take that much texture to make them look good. Texturing a jellydoll from an avatar won't look that great. But look at the picture above. If those jellydolls had roughly the texture of the real avatar, you couldn't tell the difference at this range.

It would need some prechecks. If the avatar has large attachments or is not close to humanoid, flat impostors would be more appropriate.

Jellydolls are animated, although at a very slow rate. Animation could omit most of the bento joints and run a bit faster.

A crowd of low-detail textured models could be rendered effectively.

Here's the competition, Improbable's MSquared system, with a crowd scene.

All that motion! All that action! All those blurry background avatars!

When avatars are moving around fast, the resolution needed decreases. Animating low rez avatars on each frame out to some distance is worth doing.

So this is a different approach to level of detail.

A rough guide to doing avatar LODs this way:

• HIgh - full SL detail and animation.
• Medium distance - drop to textured jelly doll with full movement and animation rate. (Go to flat impostor as at present if avatar is a bad match for a humanoid jelly doll.)
• Low - drop animation rate further depending on viewer resources.
• Lowest - impostor the jelly doll, with slow updates.

Discuss.

[Coffee Pancake]

All those avatars ... are identical. 

[JeromFranzic]

11 hours ago, animats said:

Possible approach to avatar overload - textured jellydolls.

We all know the jellydolls that appear when an avatar is over the complexity limit.

Avatars in distance, normal.

 

Avatars in distance, jellydoll mode. See the difference? Look hard.

If jellydolls had a texture created from the real avatar via the impostor rendering system, you'd barely be able to tell at this distance. You'd get an overall impression of the character and could pick out individuals. But if an avatar is a tenth of the screen height, it doesn't take that much texture to make them look good. Texturing a jellydoll from an avatar won't look that great. But look at the picture above. If those jellydolls had roughly the texture of the real avatar, you couldn't tell the difference at this range.

It would need some prechecks. If the avatar has large attachments or is not close to humanoid, flat impostors would be more appropriate.

Jellydolls are animated, although at a very slow rate. Animation could omit most of the bento joints and run a bit faster.

A crowd of low-detail textured models could be rendered effectively.

Here's the competition, Improbable's MSquared system, with a crowd scene.

All that motion! All that action! All those blurry background avatars!

When avatars are moving around fast, the resolution needed decreases. Animating low rez avatars on each frame out to some distance is worth doing.

So this is a different approach to level of detail.

A rough guide to doing avatar LODs this way:

• HIgh - full SL detail and animation.
• Medium distance - drop to textured jelly doll with full movement and animation rate. (Go to flat impostor as at present if avatar is a bad match for a humanoid jelly doll.)
• Low - drop animation rate further depending on viewer resources.
• Lowest - impostor the jelly doll, with slow updates.

Discuss.

I do something like this already, with my 1050 gpu (4gb vram)/i7-8750 cpu. System RAM (32 gb) is often nearly full at large events for me, but the jelly dolls/ALM off/no shadows/128m dd or less (plus m.2 ssd) usually makes it possible for me to sit through large events.

And yes, Cool VL Viewer in Linux is my choice for such digital tomfoolery! All the others likely will crash.

Jelly is good for us, I say.

[animats]

6 hours ago, Coffee Pancake said:

All those avatars ... are identical. 

If we had a half dozen or so standard avatar forms for distance use, that would cover most of the cases. For the rest, use the current flat impostors.

[animats]

If we just did average color of head, average color of upper, and average color of lower, put those on a jellydoll with about 5K triangles, and blurred a bit, that would probably handle huge numbers of avatars beyond 20-30m or so when under overload conditions. They could still all move.

[Marigold Devin]

On 6/21/2023 at 7:19 AM, animats said:

Today the new Motown region passed 160 avatars. At 130, Firestorm was crashing on an 8MB machine. Avatars showing up nude and with heads on backwards. Firestorm ran fine at 80 avatars on a 32 MB machine with an NVidia 3070 GPU. With 80 avatars, Firestorm was using about 16 GB of RAM.

It looks like the "mega event region" thing mostly works server side, but the viewers are going to need some work. Merely pulling in the draw distance isn't enough for a crowd facing a stage. When the draw distance is short enough to de-render some of the crowd, you can't see the stage. Impostoring all but 3 avatars did not help. "Low" quality did not help on a smaller (8 GB) machine. There's nothing left to turn down.

I did a little testing with Sharpview, my experimental viewer, where avatars are just blocks for now. No problems with avatar overload or movement. So server and network are doing fine. Rendering all that clothing is the problem.

This is going to take some thought, and then considerable work, to get right.

 

They'd be able to get 300 on if they chopped the lower body off us.

(Bad joke. Sorry not sorry. )

[animats]

9 hours ago, Coffee Pancake said:

All those avatars ... are identical. 

Yes. In examples where they aren't all identical, they're all variations on a few themes, like classic SL avatars. Here's a still from another Improbable video.

Note faceless avatar in yellow jacket.

What seems to make this work is really low resolution of anything but the hero avatars. ("Hero" is meant in the film context, where it just means something important enough to the audience to need full detail.) Even at their highest resoution, their avatars are nowhere near AAA title or SL level. More like early 2000s game level.

It's worth looking at this sort of thing, to see what you can get away with visually. Good background detail, which SL has, plus good hero avatar detail, which SL has, plus blurry low-detail background avatars, which SL does not have, is a way to scale up SL viewers for crowds.

[Coffee Pancake]

The idea of a standard avatar is great .. only that was SL 15 years ago.

Getting all the separate meshes backed into a single object would be enough to make jellys redundant

[animats]

Avatars at various distances. Best viewed with image at full screen size. These are all from Firestorm, in mouselook, from another avatar's point of view.

• At 10m, we need the full avatar, although maybe we could cut the texture resolutions.
• At 20m, we could use a simplified avatar. This is the hard one.
• At 30m, a textured jellydoll would work. If all you had was a uniform head color, top color, and bottom color, that would be good enough. Compare the Improbable images at that scale.

They'd all look better than what Improbable is doing. (We will have more saturated colors once PBR goes into the mainstream viewers. You've all seen my Sharpview images.)

So how do we do the 20-meter "simplified avatar", the hard case? Suggestions?

1 hour ago, Coffee Pancake said:

The idea of a standard avatar is great .. only that was SL 15 years ago.

Getting all the separate meshes baked into a single object would be enough to make jellys redundant

I agree, and I've discussed that before. Roblox does that. It's quite difficult, especially on existing content. That's probably the way forward for some future avatar system, but not something we can apply to existing content.

We need something simpler, like rendering the avatar's impostor image and somehow projecting that image onto a standard avatar form. Just use a generic human form, resized to match the avatar dimensions. Only animate the major bones, but animate them at full frame rate. If the projection is a really bad match (use an image matching program to check) drop to regular impostor mode. This is an optimization. The idea is to get most of the people in a crowd, not all of them. Most SL avatars are roughly humanoid at distance.

Anyway, that's an outline of an approach which might work.