Making Second Life look like an AAA game

[animats]

I've mentioned occasionally that I'm working on a new viewer. Here's some video from an early test version.

Second Life at full detail.

This is what Second Life and Open Simulator should look like. No more standing in front of a blurry object and waiting for it to load. Waiting, and waiting. And wondering if it's worth the wait. This changes the whole SL experience, for the better. Now Second Life looks like an AAA game.

Second Life content does not have too much detail. It just needs a more effective graphics system to display it.

What's going on here? This is an all-new viewer, with no Linden Lab code. It's written in Rust, and uses Vulkan for graphics. It has physically-based rendering. It's multi-threaded. One CPU is just refreshing the screen, at 50 to 60 FPS here. The other CPUs are making changes to the scene as the camera moves. All those high-detail textures are being loaded from cache just before the camera gets close enough to see them. If everything is in cache, this viewer can stay ahead of camera movement, even for very  high detail content like this. If the content has to come from the server, the objects that cover the most screen area are always loaded first. So what's in front of you is never blurry for more than a very brief period.

All this is very experimental. This is just the rendering part of the viewer. There's no user interface other than moving the camera. All this can do is look.

I'm working through the hard problems of building a high-detail metaverse here. The underlying technology is cutting-edge - the Rust programming language, Vulkan, WGPU for cross-platform graphics, and Rend3 to make WGPU usable. The lower levels libraries are not yet stable or complete. (For example, WGPU doesn't implement rigged mesh yet, so there are no avatars shown.) I'm doing this to see what's possible medium-term, not to produce a new SL viewer in the near term.

Linden Lab tried to do something like this once, as part of Project Interesting. But it was a tough retrofit for the old viewer code, and they were not successful.

[bigmoe Whitfield]

okay I'm in,  do you have anyplace I can watch the development side of this?  you got an old timer here curious.

[Galespring]

Would love to be a Guinea pig for testing, would like to see how quick it loads things around the places I frequent so could get people excited about this project more.

[animats]

This is just the rendering part at this point. Log in and cam. It's a long way from a full viewer.

I don't want to get end users too excited about this. Many viewer devs, both inside and outside Linden Lab, know what I'm up to. After a year of work, and some good results, I decided to put out a progress report.

This is partly in response to recent comments about excessively large textures. Those are not a real problem if you don't load all those pixels until you really need them. If you get close enough to a 2048x2048 texture to fill your screen, it will get loaded, but not otherwise.

Some of what I've done here could be done in the LL or Firestorm viewers. They have a texture loading priority system, left over from Project Interesting, but it was never finished.

All this applies to non-avatar objects. Avatars have their own rendering problems, and I have not looked hard at those yet.

Edited October 29, 2021 by animats
Explain

[Coffee Pancake]

The biggest difference between SL content and AAA game content is ... lighting.

We don't tend to light anything, because lighting is hard.

[Mollymews]

9 hours ago, animats said:

standing in front of a blurry object and waiting for it to load. Waiting, and waiting. And wondering if it's worth the wait

 

Linden Lab tried to do something like this once, as part of Project Interesting. But it was a tough retrofit for the old viewer code, and they were not successful.

i appreciate the effort and thought you put into these kinds of things. But I just say that back in the day before Project Interesting, the view had become diabolical. A world of constant fuzzy

we invented thought games to help us deal with it in our heads

the Lucky Fuzzy Game was quite popular. Go to a shop to play the Lucky Boards. Can't see the prize textures. Can't see the letters either. So click the board and get lucky. And if get lucky because is your letter that you can't see then win a prize, which you can't see either. But! is better to win a Fuzzy than not win anything at all. So \o/ thankyou! Click the next board which you can't see and try to get lucky again and win another Fuzzy!

Project Interesting broke the Lucky Fuzzy Game, which I "complained" about at the time. Blinking Lindens. Breaking my game! Whatever were they thinking! Going round breaking content as if that was a good thing!

while it may not have been continued to perfection, Project Interesting viewer released in 2014, a culmination of several years work, totally changed for the better what the world had become since the prior introduction of mesh

Edited October 29, 2021 by Mollymews
typs

[Profaitchikenz Haiku]

4 hours ago, Coffee Pancake said:

because lighting is hard.

Unless it's facelights ...

[SarahKB7 Koskinen]

SL isn't a video game though. Why would you want to make SL look like a video game?

A 3D digital virtual environment that runs on a computer should not automatically equate as "video game". Are flight simulators video games? No, of course not.

SL is a simulated world, not a video game. Why else do you think the regions are nicknamed "sims"?

Edited October 29, 2021 by SarahKB7 Koskinen

[Jackson Redstar]

in a Nutshell, it seems - Sansar. Each region of downloaded to local before you get there so no more lag waiting for textures no matter where you cam to

[Theresa Tennyson]

4 hours ago, SarahKB7 Koskinen said:

SL isn't a video game though. Why would you want to make SL look like a video game?

A 3D digital virtual environment that runs on a computer should not automatically equate as "video game". Are flight simulators video games? No, of course not.

SL is a simulated world, not a video game. Why else do you think the regions are nicknamed "sims"?

https://www.theverge.com/2021/10/19/22734169/microsoft-flight-simulator-game-of-the-year-edition-directx-12-support-date

[Kathlen Onyx]

4 hours ago, SarahKB7 Koskinen said:

SL isn't a video game though. Why would you want to make SL look like a video game?

A 3D digital virtual environment that runs on a computer should not automatically equate as "video game". Are flight simulators video games? No, of course not.

SL is a simulated world, not a video game. Why else do you think the regions are nicknamed "sims"?

SL is many things including a game. It's a 3D virtual environment, a chat platform, a building simulator, a multiplayer roleplaying game.  Last time I looked "the sims" was a video game. The opinion that SL is not a game is simply an opinion. In fact, most people treat it as one big Roleplaying game for the most part. 

[Nick0678]

4 hours ago, SarahKB7 Koskinen said:

SL isn't a video game though. Why would you want to make SL look like a video game?

A 3D digital virtual environment that runs on a computer should not automatically equate as "video game". Are flight simulators video games? No, of course not.

SL is a simulated world, not a video game. Why else do you think the regions are nicknamed "sims"?

Well..you can also put a NASA logo on it if you like... but that won't grant you a pilots license.

[Nick0678]

11 minutes ago, Sam1 Bellisserian said:

SL is many things including a game. It's a 3D virtual environment, a chat platform, a building simulator, a multiplayer roleplaying game.  Last time I looked "the sims" was a video game. The opinion that SL is not a game is simply an opinion. In fact, most people treat it as one big Roleplaying game for the most part. 

Ahh... yes I once tried to fly like avatars do in second life.. it didn't work.

Simulator they said...

[Lyssa Greymoon]

5 hours ago, SarahKB7 Koskinen said:

SL isn't a video game though. Why would you want to make SL look like a video game?

It shares the visual characteristics of 3D games. 

5 hours ago, SarahKB7 Koskinen said:

Are flight simulators video games? No, of course not.

They are if the planes have guns. 

[animats]

8 hours ago, Jackson Redstar said:

in a Nutshell, it seems - Sansar. Each region of downloaded to local before you get there so no more lag waiting for textures no matter where you cam to

That's not how this works. That's an illusion. The viewer is in a race with the user to load content before the user gets close enough to see through the illusion. High-resolution textures are constantly being loaded as the camera moves around, and distant textures are being dropped to lower resolution to keep the frame rate up. There's a lot going on behind the scenes to create the illusion that nothing is happening. Pay no attention to the man behind the curtain.

Here's another example, from an earlier, lower performance version of the system. This shows what it looks like when textures are not in local cache. This is a tour of a shopping event. Booth after booth of high-detail objects. You'll see some small or distant grey objects that haven't loaded yet, and some objects with fuzzy, low detail textures. As the camera gets close, the grey objects turn to textures, and fuzzy textures go to higher resolution. You never see a fuzzy texture in close-up for more than a second or two. The shopping experience improves. You can see the merchandise.

Next problem is getting rid of grey. What I'd like to do there is have a per-region list of objects, faces, and average colors. Download that and use it as the initial color for each object. Here's a test of what that would look like.

Mono-color mode. No textures, just colors. The object color is simply the texture reduced to 1x1 size, which is a single color.

The plan is to show something like this when you first enter a new area for which textures are not yet available. The amount of data needed to do this is small, and so this info can be cached for a very large number of regions. Texture loading will quickly follow, starting, of course, with the objects occupying the most screen space loaded first.

No more annoying, immersion-breaking grey. You can see where you are. You can see where you're going.

There is no fundamental obstacle to Second Life / Open Simulator looking like an AAA game title.

Edited October 29, 2021 by animats

[Katherine Heartsong]

20 minutes ago, animats said:

There is no fundamental obstacle to Second Life / Open Simulator looking like an AAA game title.

This. Superb work.

[Flea Yatsenko]

I think this is great, I've always been a fan of making the current viewer an "editor" for builders and having a lightweight, high performance viewer for people who don't want to build. The vast majority of users would be very happy with a lightweight viewer that just ran well, let them move a few things around, manage inventory, change appearance, and of course be social. I am a builder but I would use the lightweight viewer most of the time. I'm sure most builders would agree with me.

[Coffee Pancake]

1 hour ago, Flea Yatsenko said:

I think this is great, I've always been a fan of making the current viewer an "editor" for builders and having a lightweight, high performance viewer for people who don't want to build. The vast majority of users would be very happy with a lightweight viewer that just ran well, let them move a few things around, manage inventory, change appearance, and of course be social. I am a builder but I would use the lightweight viewer most of the time. I'm sure most builders would agree with me.

Removing or simplifying the edit controls wouldn't make the viewer appreciably lighter or more performant.

It would just remove some UI .. that you now have to switch viewers the moment you need to move stuff in your home or tint something.

 

LL did have a simple UI viewer for a while which bombed SL back to the bare minimum to explore and chat with people, it was a disaster, no one found it useful and it had no impact on retention.

https://nwn.blogs.com/nwn/2011/03/basic-second-life-viewer.html

A result of this project is the viewer still ships useless touch-like controls for movement as the default.

[Flea Yatsenko]

6 hours ago, Coffee Pancake said:

Removing or simplifying the edit controls wouldn't make the viewer appreciably lighter or more performant.

It would just remove some UI .. that you now have to switch viewers the moment you need to move stuff in your home or tint something.

 

LL did have a simple UI viewer for a while which bombed SL back to the bare minimum to explore and chat with people, it was a disaster, no one found it useful and it had no impact on retention.

https://nwn.blogs.com/nwn/2011/03/basic-second-life-viewer.html

A result of this project is the viewer still ships useless touch-like controls for movement as the default.

I know, I think a long term goal would be to have viewers share optimized bulk assets instead of just downloading individual assets and trying to make it work like a real game. SL's greatest strength is how the entire thing is basically a world editor for a game, but that means a lot of optimizations you normally get can't happen. But I think it's extremely impressive to get the kind of performance animats is getting considering he's working with dynamically loaded unoptimized individual assets. You could do a lot of optimization if you were allowed to make assumptions that assets in SL wouldn't change and they could be bundled/compiled into something more efficient to handle. But currently every viewer has to assume that at any time anything on a sim can be edited and changed.

[animats]

14 minutes ago, Flea Yatsenko said:

You could do a lot of optimization if you were allowed to make assumptions that assets in SL wouldn't change and they could be bundled/compiled into something more efficient to handle.

That's what makes Unreal Engine work. Vast amounts of prep and optimization during level building. If you can never see area A from area B, you never need to draw A when in B. Or even load it. That kind of thing.

[Kathrine Jansma]

While it might seem obvious, that prepping and optimizing upfront has huge benefits, does it hold when trying?

With a modern I/O pipeline and multi-cores, you should be able to real-time stream the necessary data into the rendering pipeline (at least for higher spec systems). Like having io_uring or similar APIs feed a ton of parallel request to the SSD thats usually idleing due to the inefficient serialized request patterns? That obviously doesn't help with occlusion maps etc. but how about JITtting that part? Like have one viewer compute the part and offer it to the server for caching and other viewers opting in to just download and use it.

Do you have any numbers how many MB/s or objects/s one needs from the I/O and texture decoding subsystem for a smooth experience?

[animats]

19 hours ago, Flea Yatsenko said:

I think a long term goal would be to have viewers share optimized bulk assets instead of just downloading individual assets

I've thought about viewer cooperation to boost performance, but it has griefing problems. Never trust the client. There are also terms of service problems.

An impostor server, to create the illusion of very large draw distance, is an option.

This would store pictures of entire regions. The TOS allows you to take pictures of SL and redistribute them. Each region would have pictures from at least 8 horizontal directions, plus straight down. Beyond draw distance, you'd be shown pictures of adjacent regions, positioned like billboards at the edge of the region. If you're flying, you'd see mostly the straight down images, like flying over a map. Distant images would be blurred a bit, as if there was a bit of haze. This is roughly what Google Earth and GTA-type games do to show you a big world in 3D.

With that, users would have more of a sense of place. If there are mountains to the north, you'd be able to see them, and have a sense of direction. When boating, you'd be able to see distant shorelines. When flying, you'd be able to see distant terrain and airport runways, as if you were flying over the map.

This takes a few tricks to deal with SL content. The picture-taker would only cover maybe the first 128m above ground, so as not to show sky junk for miles around. The downward pictures would be taken from a height just above the highest object with a ground connection. It would be like "terrain view" in Google Maps, vs. "map view".

The pictures would be taken by a bot that photographs the world once a week or so. Mainland only, plus some of the larger multi-sim regions. For a single region, it's unnecessary, of course.

This is just a concept for now. I might take some pictures of New Babbage's 14 sims as a demo. A real version would require LL support, or support from other large Open Simulator grids.

 I used to have a little "impostor garden" in Vallone, where visitors could compare 3D objects and impostors. From 20m away, the impostors looked pretty good. For this, I'm talking about impostors of objects at least 100m away.

Here's GTA V:

Use of distant impostors in GTA V. Second Life should look this good.

This is something LL could do. It' s not that difficult. You get a huge improvement in visual quality for a low investment.

It would show off that SL is a really big world, unlike those dinky Facebook worlds.

[Mollymews]

5 hours ago, animats said:

An impostor server, to create the illusion of very large draw distance, is an option.

some feedback on this idea

back when the viewer was first open sourced and OpenSim servers were being thought about and experimented with, one of things that some of us played with was a seamless contiguous world

walk in any direction in a straight line and will eventually end up back where you started. Even when regions were notionally positioned alone (surrounded by void space) on the grid. If public access to the region was allowed then it would be included in the users journey path. If not then skip to the next public access region. Same with regions on the diagonals within the user draw distance 

what the experiments found was this was made easier when there was a local server instance connected to the remote world server,  the local server acting as the user's client.  The viewer connecting to its own local server instance.  This being seamless. Log in to the viewer, which starts the local server, which acts as the user client program as far as the remote world server is concerned.  The rule being that it was a one-to-one relationship: each viewer instance having its own instance of a local server dedicated to it. The local server being pass-through

this one-to-one relationship satisfying the remote world server's need requirement for robust user credentials identification/validation. Can't have more than one account passing their user credentials thru the same local server

the benefit was that the local server could have a very large draw distance, able to download assets from far far away and cache them locally, and the client viewer draw distance able to be set to less depending on user hardware

i never myself went much further with this beyond the concept experiment stage.  14 years ago I never had the programming chops or time to do it myself, and I don't have the time now either to learn the chops needed to bring it to a commercial-grade level.  I still like the idea of it tho

 

ps add

for anyone reading and wanting to have a go at this (and how it was gone about back in the day) then use a mod of the opensource bot client software as the local server. we don't need a installation of a full on OpenSim like server instance just for caching far away assets. Having lots of far away assets cached by an independent process, can make what animats is mentioning about far distance billboard impostering a whole lot simpler to implement without overburdening the viewer itself. That independent process can also as a background task work thru the cache creating optimised copies  of the assets, for presentation to the viewer

 

pps add

i don't mean for my posts to ever mean that I am raining on your efforts animats.  I appreciate what you do. You are tenacious and attentive to what you do. You don't quit easy.  I admire this in people. Same with TPV developers. They don't know how to quit easy either. And without that attentiveness and tenacity then the world would be a whole lot less than what it is. I have the attentiveness of a butterfly. When something takes me longer than an hour to do, then I pretty much leave it and flit to the next shiny

Edited November 1, 2021 by Mollymews

[InternetHITS]

 

It would be awesome if we can use SL as a test environment for machine learning.

So we can test it virtual before going RL.

Imagine how many customers you would get.

 

 

 

 

 

[NiranV Dean]

1 hour ago, InternetHITS said:

 

It would be awesome if we can use SL as a test environment for machine learning.

So we can test it virtual before going RL.

Imagine how many customers you would get.

 

 

 

 

 

What would said machine learn other than that humans have completely unrealistic desires, proportions and beauty standards.