Support for Float16 in JS & WASM environments

[dontcallmedom]

@jasonmayes mentions the lack of support for float16 in JS and WASM environments as problematic for quantized models.

I see there was some discussion in JS land to add support for float16, but with no clear update in the past 2 years. Would this use case help give more momentum to the proposal?

There is a short discussion in the WebAssembly CG in the context of SIMD. Can anyone more involved in WebAssembly comment on this (@huningxin maybe?)

[jasonmayes]

Quite possibly. Machine Learning is growing in popularity especially the last 3 years and I believe it will touch every industry in the future. Being able to support this will enable to reduce the size of ML models in the future and the amount of memory they take up when being executed - which in some cases can be very large. As the JS community get more confident in creating ML models, no matter what framework they choose to use, having this would help in such situations where quantisation is desired.

Adding @pyu10055 @dsmilkov @nsthorat @annxingyuan @tafsiri @lina128 for any thoughts related to this topic too (TF.js team) and for visibility.

[cynthia]

It feels like reducing the size of the model (e.g. for transport) via FP16 is not as difficult of a problem than having proper type support. (While wasteful, one can cast a half precision model to full and run it as full.)

There is also the question of the newer types, which definitely step into non-standard territory.

[wchao1115]

float16 is pretty important in the context of ML performance on NVIDIA tensor core. So having the native data type support in JS would be very handy. Typically quantization to float16 can be done more readily than int8 quantization. Int8 would be important too in the future as its support in the native GPU platform is becoming more common.

[mingqiusun]

related discussion on WASI-NN
WebAssembly/wasi-nn#4

[kgryte]

While float16 may be useful, arguably equally, if not more useful, would be native bfloat16 support. Ideally both would be natively supported. The former for a more classic half-precision type and the latter for ML use.

[cynthia]

bfloat16 is not a standard format, which makes it rather difficult to adopt into a W3C spec.

[somethingelseentirely]

@cynthia Well, given the current industry adoption and hardware support behind it, a standard is bound to happen at some point.
Since this doesn't have the complexity of WebSQL where standardising SQLite was prohibitively expensive, a W3C standard might as well become the defining document for bfloat16. Why leave all the low-level fun to I3E 😁.

[cynthia]

It's fine that bfloat16 is not a standard yet. The field is constantly innovating, and not having a formal standard probably helps them move faster. But it does make it hard to adopt in a W3C standard.

Besides, the benefits seem to mostly for training accelerators; and the current scope (correct me if I am wrong) doesn't include support for backward passes.

a W3C standard might as well become the defining document for bfloat16

This would likely result in an unpleasant surprise for those who have to look it up later. We have enough cases of "it's in that spec because of legacy reasons" and continuing that antipattern is the last thing we should be promoting.

[dontcallmedom]

see webmachinelearning/webnn#373 for an update on this topic