feat: WASM png support

[william-silversmith]

Hi Jeremy,

I added PNG support for Neuroglancer and tested it with grayscale images using a newly compiled libpng.wasm module. I modeled this PR after #318

The important differences are:

• I used two third party libraries spng and miniz which have permissive licenses (BSD-2 and MIT respectively) in order to get PNGs to decode to a raw buffer.
• The codec is in C rather than C++.

I have a working PR for CloudVolume that can write PNGs.

I ran PNG on CREMI aplus, which was a bigger volume than my previous small tests and again saw a ~25% improvement in storage size, so I think this is real.

I'm a bit unhappy with how I had to use a macro in order to safely free ctx at each if statement. If you know a better C idiom, I'd be happy to update it.

Thanks!

[jbms]

Thanks.

I'm assuming spng and miniz are unmodified. If that is the case, can you download them (and check their sha256 checksum) as part of the build script rather than including their source code directly?

[jbms]

Note: Best way to download them is probably to do it as part of the Dockerfile.

[william-silversmith]

I can give that a shot. I assumed you might want me to modify spng to remove the encoder part though to shrink the WASM size.

My colleague Ran mentioned that maybe I should experiment with webp too which claims on its website to generate even smaller lossless images https://developers.google.com/speed/webp/. Maybe I should check that out before we merge?

[jbms]

As far as removing the encoder, I think it will make no difference, the webassembly optimizer should remove code that it can statically determine to be unused.

There are other codecs to explore --- webp, avif, jpeg2k, jpeg-xl. However, I can imagine that for different situations, different codecs will be preferred.

It isn't ideal to add a bunch of codecs to the neuroglancer precomputed format, since it means every implementation needs to support them, but I imagine if we add them to neuroglancer they will also be useful for other formats like zarr. The imagecodecs Python package already semi-officially adds support for them to zarr under the names imagecodecs_*.

[william-silversmith]

I guess that makes sense, though it's possible WebP might mostly dominate PNG according to this study (on data that is pretty different): https://developers.google.com/speed/webp/docs/webp_lossless_alpha_study

Do you anticipate neuroglancer (and Precomputed) eventually supporting all the codecs you listed? Based on the denoising paper I've been expecting people to want to at least support JPEG-XL or AVIF.

[jbms]

It seems plausible that all would be supported by Neuroglancer, though I would just assume wait until there is a specific use case for each one.

It sounds like the webp compression study you linked (https://developers.google.com/speed/webp/docs/webp_lossless_alpha_study) may have been inaccurate as far as webp decoding speed: https://groups.google.com/a/webmproject.org/g/webp-discuss/c/FPOfZs2cCS4/m/2F0vs7qGiKIJ
It looks like PNG decoding may actually be about twice as fast.

Also it would be nice to get code splitting working in Neuroglancer before adding too many codecs, so that users don't have to download all codecs even if they aren't using them. Unfortunately that is currently blocked by the fact that esbuild only supports code splitting for the esm bundle format (evanw/esbuild#16) and Firefox does not support esm for workers (https://bugzilla.mozilla.org/show_bug.cgi?id=1247687).

Though we could actually split out the wasm files even if we can't split out the javascript --- originally I had the wasm bundled into the javascript, and then split the javascript, but now that splitting the javascript can't be done, just splitting the wasm could be a reasonable alternative.

[william-silversmith]

Interesting, it looks like that groups discussion is from 2013 and the study says it was last updated in 2017 (but doesn't say what was updated). They may have improved the codec implementation in the intervening years, so I guess we'll have to test it ourselves (we have different data than they tested anyway).

I was thinking similarly regarding splitting out the WASM. So far the burden isn't too high (similar to a about two big chunks).

148K  ./src/neuroglancer/mesh/draco/neuroglancer_draco.wasm (50 KB gzipped)
43K   ./src/neuroglancer/sliceview/compresso/compresso.wasm (11 KB gzipped)
43K   ./src/neuroglancer/sliceview/png/libpng.wasm (19 KB gzipped)

jpegjs is 39.3 KB unminified for the decoder.

In any case, I'll finish up this PR later today.

[william-silversmith]

I think this PR is ready for review. I moved the libraries to the docker.

[william-silversmith]

Okay, I moved the header logic into a JS function.

[jbms]

Sorry, I think I wasn't clear. Currently the javascript asyncComputation interface that you have does not allow the caller to determine or validate the data type, the dimensions, or the number of channels. To fix that, in addition to the encoded data, the caller of the async request should also pass in width, height, num_components, and dataType so that those can be validated when decoding the png (see decode_jpeg_request.ts for an example). The actual header decoding does not have to happen in javascript.

[william-silversmith]

Okay! I validated the chunk size and added support for convertToGrayscale. However, I'm having trouble figuring out how to access dataType and it seems the jpeg decoder omits this check as well. Would you have a tip for how to do that? I looked in the debugger and its not attached to VolumeChunk or the immediate context around await this.chunkDecoder(chunk, cancellationToken, response);

[jbms]

The jpeg library only supports 8-bit so there is no need to check data type.

The data type can be obtained from this.source.spec.dataType

[william-silversmith]

Thanks! That worked. I think that takes care of the comments so far.

[jbms]

I know you put in some export to implement this header parsing in javascript, but is there any advantage over just relying on spng to do that?

[william-silversmith]

The main advantage I can think of is that the error message is easier to write in an informative way. I can revert it back if you'd like.

[william-silversmith]

One other refinement I could make is to set covertToGrayscale = true if the expected number of channels is 1. This would allow some laxness in decoding PNGs that were sloppily set to RGB mode but are still supposed to be grayscale. Not sure how you'd feel about that. It might make implementations a bit easier, though they would be inefficient.

[jbms]

I think it is better not to convert to grayscale for the precomputed format --- that will just encourage laxness and force every implementation to handle that case.

[william-silversmith]

Okay! I fixed up those items.

[jbms]

Thanks!

[william-silversmith]

Thank you Jeremy! I'll release PNG support in CloudVolume today.