Support arm (non tier 1) platforms

[jacobrosenthal]

The wasi story is really exciting in the idea of showing we can run the same code on multi platforms and architectures so I went to do that, and in compiling for arm I found that only tier 1 basically x86 platforms are supported currently.

/home/pi/wasmtime/wasmtime-jit/src/link.rs:(.text._ZN12wasmtime_jit4link8relocate17h8a250ae005184b62E+0x680): undefined reference to `__rust_probestack'
collect2: error: ld returned 1 exit status

Found this in probestack so it seems like an upstream problem

//! Finally it's worth noting that at the time of this writing LLVM only has
//! support for stack probes on x86 and x86_64. There's no support for stack
//! probes on any other architecture like ARM or PowerPC64. LLVM I'm sure would
//! be more than welcome to accept such a change!

And found these
rust-lang/rust#43241
rust-lang/rust#49355

[sunfishcode]

Wasmtime doesn't support ARM yet. We want to support ARM, and we know Wasm and WASI are portable to ARM platforms, but we haven't had the resources to do the actual port yet.

[ebeasant-arm]

LLVM/Clang 8/9 on aarch64 doesn't support stack probes using the __rust_probestack() implementation, but it does supply __chkstk(), which it appears that wasmtime uses as a fallback (see link.rs, line 338 and 347). At the very least a #cfg guard change here will allow wasmtime to compile on native aarch64.

[pepyakin]

Ok, this indeed works on aarch64. My tests however show that neither __rust_probestack nor __chkstk exist on ARMv7.

[jlb6740]

When trying to compile in a container with ubuntu on aarch64 I still received an error for an undefined reference to __chkstk:

wasmtime_jit.6so9sdbw-cgu.15:(.text._ZN12wasmtime_jit4link11link_module17hdaec8638121cc26dE+0x240): undefined reference to `__chkstk'

I thought maybe I needed compiler_rt but after compiling and installing, the linking error remains. @pepyakin Did you need to do any other steps to get this compile?

[pepyakin]

Hey, good question! I didn't do any additional steps. FWIW, I compiled on an aarch64 host. TBH, I am not sure what OS was that but most likely some actual version of Ubuntu.

[rahul-thakoor]

I'm also getting this error when compiling on aarch64 and using latest ubuntu:

wasmtime_jit.9xnaws2x-cgu.14:(.text._ZN12wasmtime_jit4link11link_module17h3c526a745ac9bdafE+0x16c): undefined reference to `__chkstk'
wasmtime_jit.9xnaws2x-cgu.14:(.text._ZN12wasmtime_jit4link11link_module17h3c526a745ac9bdafE+0x170): undefined reference to `__chkstk'
collect2: error: ld returned 1 exit status

[pepyakin]

I just got a fresh new machine with similar/same configuration and I can confirm the presence of the linking error. Seems like I messed up with testing #800 .

I checked and turns out that compiler-builtins (which rustc uses instead of compiler-rt) doesn't supply __chkstk nor ___chkstk. It also seems that I somehow managed to mess up testing of #800 :'-(

[brooksmtownsend]

A project that I contribute to has a dependency on the rust crate wasmtime = "0.15.0", and running cargo build on a Raspberry Pi 3B using Ubuntu 20.04 LTS armv7l results in the following error:

   Compiling wasmtime-runtime v0.15.0
error: unsupported platform
   --> /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/wasmtime-runtime-0.15.0/src/traphandlers.rs:169:21
    |
169 |                     compile_error!("unsupported platform");
    |                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

error[E0308]: mismatched types
   --> /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/wasmtime-runtime-0.15.0/src/traphandlers.rs:156:52
    |
156 |         unsafe fn get_pc(cx: *mut libc::c_void) -> *const u8 {
    |                   ------                           ^^^^^^^^^ expected *-ptr, found `()`
    |                   |
    |                   implicitly returns `()` as its body has no tail or `return` expression
...
169 |                     compile_error!("unsupported platform");
    |                                                           - help: consider removing this semicolon
    |
    = note: expected raw pointer `*const u8`
                 found unit type `()`

error: aborting due to 2 previous errors

For more information about this error, try `rustc --explain E0308`.
error: could not compile `wasmtime-runtime`.

To learn more, run the command again with --verbose.

I understand that wasmtime does not support arm, just thought I'd contribute and say that others are still watching 😄

[jlb6740]

Hi @brooksmtownsend have you seen this #1494? Does this platform support Arm64?

[cfallin]

The RPi3 does have a 64-bit CPU, but I think most distros (at least the default Raspbian, and it looks like Ubuntu install in the logs above too) run a 32-bit kernel and userland. Unfortunately we only support ARM64 (which would require a 64-bit userland) right now. @brooksmtownsend, I've used wasmtime successfully on a Gentoo 64-bit install on a RPi4, FWIW, but I know switching distro/install is quite a hassle.

ARM32 support may come at some point in the future, but I'm not sure what our architecture support roadmap looks like so I can't make any promises, sorry! If anyone is interested in working on this, I'd be happy to provide guidance as best as I can.

[brooksmtownsend]

@cfallin @jlb6740 great callout and this was all done on a fresh install of, unfortunately, the 32-bit server OS. I'll try out switching to the 64-bit version of Ubuntu (I'm not dedicated to Ubuntu, but figured I'll stick with changing one thing at a time) and see how that goes. Thanks!

[ebeasant-arm]

If you can get it installed, Ubuntu 64-bit is well worth the effort: With the recent upgrade of the Pi Imager, its not much harder than putting Rasbian on it from scratch :-) There's been a huge amount of work on Cranelift/wasmtime very recently to support aarch64, which is now maturing nicely, and being used in anger. Keep up the good work!

[brooksmtownsend]

Awesome! Looking forward to getting it, I actually didn't mean to install the 32-bit version in the first place

[tschneidereit]

@brooksmtownsend we'd love to hear how things are going, so it'd be great if you could report back here. Do note that #1494 just landed, so ARM64 support is quite new, which might mean running into early-adopter issues. Which mostly means that hearing how things are going would be even more appreciated :D

[brooksmtownsend]

@tschneidereit i tried building with the aarch64 version of Ubuntu 20.04, unfortunately I ran into the same issue. This is with wasmtime-runtime v0.15.0, is that too early of a version?

[tschneidereit]

@brooksmtownsend thank you for reporting this! I think v0.15.0 might indeed be too old: I think the new backend hadn't landed by then, and we certainly didn't do aarch64 builds in CI. We do do that by now, see the dev and v0.16.0 releases.

However, the cross-compilation process isn't documented (at least that I could find). I just filed #1720 to rectify that. Here's the patch that added aarch64 builds to CI, in case you're interested.

[bjorn3]

The new backend framework has a working AArch64 backend.

[pannous]

Has this not be merged into python wasmtime?

Successfully installed wasmtime-0.33.0

from wasmtime import *

"unsupported architecture for wasmtime: arm64"

[bjorn3]

An AArch64 wheel is uploaded for Linux: https://github.com/bytecodealliance/wasmtime-py/blob/c4956dee5a1b66355df6b42117d604c33b48006b/.github/workflows/main.yml#L58 Are you on Windows or macOS? It seems that no AArch64 wheel for those is build. macOS should work fine AFAIK. I am not sure if the Windows AArch64 calling convention is supported yet.

[pannous]

macOS Monterey 12.1 M1, Python 3.8.9

[bjorn3]

Could you open an issue at https://github.com/bytecodealliance/wasmtime-py/issues to distribute a wheel for macOS AArch64?