Decompressing files with graphics cards

[Wulfie Reanimator]

https://devblogs.microsoft.com/directx/directstorage-1-1-coming-soon/

Most of the important jargon is in there, but in short, Microsoft is releasing a new feature for developers that allows them to load files via the GPU, up to 40% faster in practice in some undisclosed game(s).

It's obviously not gonna be a magic fix for everything, but having the option to off-load tasks from the CPU could be helpful especially in Second Life's always-streaming environment. (Especially with viewers that still don't really do much with the GPU, relatively speaking.)

Edited April 4, 2023 by Wulfie Reanimator

[animats]

JPEG 2000 decode is a huge headache.

You can definitely do JPEG 2000 decompression in the GPU. There are several implementations available:

• https://developer.nvidia.com/blog/accelerating-jpeg-2000-decoding-for-digital-pathology-and-satellite-images-using-the-nvjpeg2000-library (NVidia GPUs only)
• http://cuj2k.sourceforge.net/ (from 2007)
• http://apps.man.poznan.pl/trac/jpeg2k/browser (from 2013, Affero GPL license and incompatible with LGPL.)

Plus there are some commercial ones.

This is more useful when you're not using the GPU for graphics, because it's going to take up GPU time that graphics needs. It's going to depend on the GPU. If you try this on a GPU with slow compute shaders, it's probably not going to be great and may be a net loss.

Outside of SL, JPEG 2000 images are used mostly for huge images divided into tiles, such as aerial photographs and medical images. Tiles can be decompressed in parallel and at different resolutions, so you can zoom in. That doesn't help for SL, where images are single-tile. So JPEG 2000 decoders tend to have the wrong feature set for SL.

I've been looking at JPEG 2000 decoding for my own viewer, and spent some time getting OpenJPEG to stop getting segfaults on SL content. It can now handle all but two images at Bug Island, LL's collection of objects that have given viewers trouble.  Two images there were encoded years ago with some off-brand JPEG 2000 encoder while JPEG 2000 was undergoing a revision, and there are rare images in world from that era.

OpenJPEG now passes valgrind, the pointer validation tool. But OpenJPEG is old and brittle C code with a long history of security bugs. There's an effort underway to convert it to Rust to help with that.

The LL viewers and Firestorm currently use Kakadu, which is not cheap. The free decoders all have problems. Less than they used to, though.

[Kathrine Jansma]

7 hours ago, animats said:

his is more useful when you're not using the GPU for graphics, because it's going to take up GPU time that graphics needs. It's going to depend on the GPU. If you try this on a GPU with slow compute shaders, it's probably not going to be great and may be a net loss.

It could be nice for Systems with an APU & GPU, e.g. Intel or AMD CPU with a graphics core plus some external CPU. A bit like Intels video transcoder.

[Kathrine Jansma]

I doubt DirectStorage would be such a big thing for SL style viewers. Games have assets on NVMe storage, so you can optimize to get the 3-15 GB/s streamed from those devices instead of the shoddy 100-500MB/s you often get for a single threaded access. And if you need to decompress a 3 GB/s stream of assets in real time, some GPU support is super helpful of course. But unless you have all the scene cached in an SL setup, your network pipe is typically faaaar away from anything like 3 GB/s, maybe you get 10 GBit/s in rare cases with fast fibre setups.

If you have extra CPU cores around, you can easily do the decoding and I/O in a thread pool with an efficient async I/O API like io_uring or IOCP and  get some decent throughput. But getting the textures into the GPU for usage is crappy right now with OpenGL and needing to bind all the textures etc.

Sure, if you set minimum OpenGL version to something newish (and not working on OS X), rewrite the rendering loop to use bindless textures and other modern tricks, you could benefit from a fast pipe that pumps data into the GPU memory for decoding/processing. But its more likely you would switch to Vulkan APIs first, so maybe LL has a look at the pipeline from Network / Cache to the GPU at that time and profile it a bit.

[Henri Beauchamp]

On 4/4/2023 at 5:34 AM, animats said:

I've been looking at JPEG 2000 decoding for my own viewer, and spent some time getting OpenJPEG to stop getting segfaults on SL content. It can now handle all but two images at Bug Island, LL's collection of objects that have given viewers trouble.  Two images there were encoded years ago with some off-brand JPEG 2000 encoder while JPEG 2000 was undergoing a revision, and there are rare images in world from that era.

OpenJPEG now passes valgrind, the pointer validation tool. But OpenJPEG is old and brittle C code with a long history of security bugs. There's an effort underway to convert it to Rust to help with that.

You might want to have a look at the (heavily) patched OpenJPEG v1.4 version I am using in the Cool VL Viewer (libopenjpeg/ directory in the viewer sources): it also passes all the tests of Bug Island with the exception of those two textures (but it does not crash at least).

It has got all the known security and crash bugs fixed, many personal micro-optimizations, SSE optimizations backported by me from OpenJPEG 2, a couple AVX opts by Kathrine Jansma, and it is well behaved while rezzing textures (e.g. the alpha channel is decoded early, avoiding pseudo-opaque textures while rezzing, like what happens with OpenJPEG v2).

I recently tried (again) OpenJPEG 2 (v2.5 this time, with a patch for partial texture streams), but it fails many times where v1.4 succeeds, and rezzes textures in a ugly way (especially tree/foliage textures, quite a few of which don't even decode fully/properly). So I decided (once more) to keep the patched v1.4 for my viewer...

[animats]

1 hour ago, Henri Beauchamp said:

I recently tried (again) OpenJPEG 2 (v2.5 this time, with a patch for partial texture streams), but it fails many times where v1.4 succeeds, and rezzes textures in a ugly way (especially tree/foliage textures, quite a few of which don't even decode fully/properly). So I decided (once more) to keep the patched v1.4 for my viewer...

Try OpenJPEG 2.5 from Github, rather than the release version. We got a few fixes in that were needed to stop blowing up on some SL content.

Can you get your fixes pulled back into the main version? The OpenJPEG devs have been cooperative about fixes others have generated. OpenJPEG has some maintenance funding.

(Overly technical stuff follows.)

The developer of the Rust crate "jpeg2k", which is a safe wrapper for OpenJPEG, tried a few things. He compiled OpenJPEG for a WASM target, which sandboxes it and prevents it from taking down the whole program when it crashes. But there was a 2.6x slowdown. He ran OpenJPEG through c2rust, which generates unsafe Rust code from C code. It segfaulted on the same files that caused OpenJPEG to segfault. That's when we had to start using gdb and valgrind, and glaring at OpenJPEG C code. OpenJPEG files are a stream of packets, and as you process more packets the picture improves. You can cut off the file at any point and get a lower-rez picture. This is a strange way to do things, but that's what we have to work with.

Most of the complexity is in the wavelet math, which is well defined. Most of the trouble comes from the very general, overly complex, and badly documented packetization and tiling. There are all sorts of strange options. Some format change which introduced special "medical identification data" (the main use of JPEG 2000 is CAT and MRI scans) seems to be responsible for those textures at Bug Island that won't decode, due to an encoder/decoder incompatibility that existed some years back.  I've seen a few such textures in world.

All this is Not Fun.

[Henri Beauchamp]

6 hours ago, animats said:

Can you get your fixes pulled back into the main version?

That would be unpractical... It has been years I modified OpenJPEG v1.4 and, as usual, being the ana1 (*) programmer I am, I also modified the code formatting to something cleaner, meaning even with a simple 'diff', you'd get a large file, with irrelevant, formatting changes... It should be considered a fork now.

Also, some changes that went info v2 would have to be actually reverted to get back the well-behaved/non-ugly rezzing experience of v1.4 (e.g. foliage rezzing is totally yuck with v2).

6 hours ago, animats said:

Try OpenJPEG 2.5 from Github, rather than the release version. We got a few fixes in that were needed to stop blowing up on some SL content.

I might try, time permitting, but I got more prominent work to do right now (EE/PBR dual-renderer, for example)...

 

(*) With 1=l of course... Sometimes, this forum auto-chastising and auto-censoring ”feature” bothers me... We'd need to have it use an AI to avoid censoring jokes applying to self... 😝

Edited April 7, 2023 by Henri Beauchamp

[animats]

8 hours ago, Henri Beauchamp said:

(e.g. foliage rezzing is totally yuck with v2).

Huh. How does that happen? I thought that for a given truncated file, the results were predetermined.

Are you decompressing with multiple threads working on the same image?

Edited April 7, 2023 by animats

[Henri Beauchamp]

6 hours ago, animats said:

Huh. How does that happen? I thought that for a given truncated file, the results were predetermined.

Good question... Yet, it happens.

6 hours ago, animats said:

Are you decompressing with multiple threads working on the same image?

I tried multi-threading with OpenJPEG v2.5, but it crashes when also using the multi-threaded image decoder (based on a LLThreadPool) in my viewer (maybe a pthread issue: not sure how the latter deals with sub-threads started from threads). So no, it was just one thread per decoded image, like I have now with v1.4.

Edited April 7, 2023 by Henri Beauchamp