Strange behavior on aarch64 linux #970

Lokathor · 2022-07-23T20:45:19Z

Now here's a log of when I went to try out a cross test just now on my Raspberry Pi using the 64-bit OS (aarch64 linux)

lokathor@skarmory:~/programming/arm7tdmi_aeabi $ cross test --target arm-unknown-linux-gnueabi
error: DEPRECATED: future versions of rustup will require --force-non-host to install a non-host toolchain as the default.
warning: toolchain 'nightly-x86_64-unknown-linux-gnu' may not be able to run on this system.
warning: If you meant to build software to target that platform, perhaps try `rustup target add x86_64-unknown-linux-gnu` instead?
info: syncing channel updates for 'nightly-x86_64-unknown-linux-gnu'
714.2 KiB / 714.2 KiB (100 %) 367.4 KiB/s in  1s ETA:  0s
info: latest update on 2022-07-23, rust version 1.64.0-nightly (848090dcd 2022-07-22)
info: downloading component 'cargo'
  6.6 MiB /   6.6 MiB (100 %)   2.2 MiB/s in  3s ETA:  0s
info: downloading component 'rust-std'
 30.4 MiB /  30.4 MiB (100 %)   2.2 MiB/s in 16s ETA:  0s
info: downloading component 'rustc'
 55.6 MiB /  55.6 MiB (100 %)   1.2 MiB/s in 33s ETA:  0s 
info: installing component 'cargo'
info: installing component 'rust-std'
 30.4 MiB /  30.4 MiB (100 %)   6.3 MiB/s in  4s ETA:  0s
info: installing component 'rustc'
 55.6 MiB /  55.6 MiB (100 %)   7.1 MiB/s in  7s ETA:  0s

  nightly-x86_64-unknown-linux-gnu installed - (error reading rustc version)

info: checking for self-updates
info: downloading component 'rust-std' for 'arm-unknown-linux-gnueabi'
info: installing component 'rust-std' for 'arm-unknown-linux-gnueabi'
 26.5 MiB /  26.5 MiB (100 %)   6.5 MiB/s in  3s ETA:  0s
Error: 
   0: no container engine found

The last error on the end is totally my fault, I just forgot to install the container engine.

So I install podman and then this happens instead

lokathor@skarmory:~/programming/arm7tdmi_aeabi $ cross test --target arm-unknown-linux-gnueabi
{"msg":"exec container process `/bin/sh`: Exec format error","level":"error","time":"2022-07-23T20:43:18.000272390Z"}

And I'm not sure what's going on, but I'm pretty sure that the fact that cross tried to install an x86_64 thing on my aarch64 device when I was asking for ARM means that I'm probably breaking some sort of assumption that cross is making.

The text was updated successfully, but these errors were encountered:

Emilgardis · 2022-07-23T22:54:00Z

Hi!

cross mounts a toolchain that can be run in the docker image, rn, it's only x86_64, hence why it's installing the toolchain for x86_64.

see #751 for progress on the effort to be able to use aarch64 toolchains automatically, with #817 it is possible already though (see the example).

This error is interesting though, seems like the podman machine can't run x86_64, however, we should automatically install the interpreters. (I'm not 100% on how podman works though, so I may be wrong with that)

Try doing the following

podman run --rm ghcr.io/cross-rs/x86_64-unknown-linux-gnu:latest uname -a

Lokathor · 2022-07-25T05:55:55Z

lokathor@skarmory:~/programming/arm7tdmi_aeabi $ podman run --rm ghcr.io/cross-rs/x86_64-unknown-linux-gnu:latest uname -a
Trying to pull ghcr.io/cross-rs/x86_64-unknown-linux-gnu:latest...
Getting image source signatures
Copying blob 58690f9b18fc skipped: already exists  
Copying blob fb15d46c38dc skipped: already exists  
Copying blob b51569e7c507 skipped: already exists  
Copying blob da8ef40b9eca skipped: already exists  
Copying blob ba4cfa0d02e9 done  
Copying blob 7e4f4e01d8cc done  
Copying blob 385151f4cf4a done  
Copying blob dd3a4857dad0 done  
Copying blob 9b3d6abb553f done  
Copying blob 7457ea83291e done  
Copying blob 94489cc40d0c done  
Copying blob f4c64b50a179 done  
Copying blob 634c7871468d done  
Copying blob f47e7b8372aa done  
Copying blob 70626baa3f44 done  
Copying blob 8c9e9eeed2e0 done  
Copying blob de83618d0833 done  
Copying config d909ec0e17 done  
Writing manifest to image destination
Storing signatures
{"msg":"exec container process `/bin/uname`: Exec format error","level":"error","time":"2022-07-25T05:47:37.000298421Z"}
lokathor@skarmory:~/programming/arm7tdmi_aeabi $

So, I guess podman is what's confused somehow?

Emilgardis · 2022-07-25T08:45:44Z

Yes, it's unable to emulate x86_64, I'm not sure how to fix it.

Lokathor · 2022-07-28T04:58:06Z

Sorry about the issue edits i thought i could get things working with Docker instead, but it appears there's an error down that path as well:

lokathor@skarmory:~/programming/arm7tdmi_aeabi $ cross test --target arm-unknown-linux-gnueabi
Unable to find image 'ghcr.io/cross-rs/arm-unknown-linux-gnueabi:0.2.4' locally
0.2.4: Pulling from cross-rs/arm-unknown-linux-gnueabi
58690f9b18fc: Pull complete 
b51569e7c507: Pull complete 
da8ef40b9eca: Pull complete 
fb15d46c38dc: Pull complete 
b8c877a978f4: Pull complete 
fbb7925cfbf9: Pull complete 
98fc6bed5cb2: Pull complete 
c670efbb9bd8: Pull complete 
b3894c48ee6c: Pull complete 
d0d9b717f205: Pull complete 
e0e52724f9a3: Pull complete 
6123aad58248: Pull complete 
1d4b87bdab60: Pull complete 
Digest: sha256:6173a1c1990f9eea091626c9a916bc721bb7919d943c9b94c979d9d10c7c7c21
Status: Downloaded newer image for ghcr.io/cross-rs/arm-unknown-linux-gnueabi:0.2.4
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
standard_init_linux.go:219: exec user process caused: exec format error

Alexhuszagh · 2022-07-28T05:16:17Z

@Lokathor Can you provide the invocation with -vv, or cross test --target arm-unknown-linux-gnueabi -vv? I would need the invocation. Also, we're working on providing native images (see #975).

Lokathor · 2022-07-28T05:19:02Z

lokathor@skarmory:~/programming/arm7tdmi_aeabi $ cross test --target arm-unknown-linux-gnueabi -vv
+ cargo metadata --format-version 1 --filter-platform arm-unknown-linux-gnueabi
+ rustc --print sysroot
+ rustup toolchain list
+ rustup target list --toolchain nightly-x86_64-unknown-linux-gnu
+ rustup component list --toolchain nightly-x86_64-unknown-linux-gnu
+ /usr/bin/docker
+ /usr/bin/docker run --userns host -e 'PKG_CONFIG_ALLOW_CROSS=1' -e 'XARGO_HOME=/xargo' -e 'CARGO_HOME=/cargo' -e 'CARGO_TARGET_DIR=/target' -e 'CROSS_RUNNER=' -e TERM -e 'USER=lokathor' --rm --user 1000:1000 -v /home/lokathor/.xargo:/xargo:Z -v /home/lokathor/.cargo:/cargo:Z -v /cargo/bin -v /home/lokathor/programming/arm7tdmi_aeabi:/project:Z -v /home/lokathor/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu:/rust:Z,ro -v /home/lokathor/programming/arm7tdmi_aeabi/target:/target:Z -w /project -i -t ghcr.io/cross-rs/arm-unknown-linux-gnueabi:0.2.4 sh -c 'PATH=$PATH:/rust/bin cargo test --target arm-unknown-linux-gnueabi -vv'
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
standard_init_linux.go:219: exec user process caused: exec format error
+ rustup component list --toolchain nightly-x86_64-unknown-linux-gnu

Alexhuszagh · 2022-07-28T05:21:50Z

lokathor@skarmory:~/programming/arm7tdmi_aeabi $ cross test --target arm-unknown-linux-gnueabi -vv

cargo metadata --format-version 1 --filter-platform arm-unknown-linux-gnueabi

rustc --print sysroot

rustup toolchain list

rustup target list --toolchain nightly-x86_64-unknown-linux-gnu

rustup component list --toolchain nightly-x86_64-unknown-linux-gnu

/usr/bin/docker

/usr/bin/docker run --userns host -e 'PKG_CONFIG_ALLOW_CROSS=1' -e 'XARGO_HOME=/xargo' -e 'CARGO_HOME=/cargo' -e 'CARGO_TARGET_DIR=/target' -e 'CROSS_RUNNER=' -e TERM -e 'USER=lokathor' --rm --user 1000:1000 -v /home/lokathor/.xargo:/xargo:Z -v /home/lokathor/.cargo:/cargo:Z -v /cargo/bin -v /home/lokathor/programming/arm7tdmi_aeabi:/project:Z -v /home/lokathor/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu:/rust:Z,ro -v /home/lokathor/programming/arm7tdmi_aeabi/target:/target:Z -w /project -i -t ghcr.io/cross-rs/arm-unknown-linux-gnueabi:0.2.4 sh -c 'PATH=$PATH:/rust/bin cargo test --target arm-unknown-linux-gnueabi -vv'
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
standard_init_linux.go:219: exec user process caused: exec format error

rustup component list --toolchain nightly-x86_64-unknown-linux-gnu

Ok it's using the right toolchain but it seems like it's unable to emulate x86_64 for some reason. Can you just confirm it by trying:

docker run --platform linux/amd64 hello-world
docker run --platform linux/amd64 ubuntu:20.04 dpkg --print-architecture

The latter we'd expect to fail for sure (since dpkg is a compiled binary) if it cannot emulate x86_64.

Lokathor · 2022-07-28T05:23:38Z

lokathor@skarmory:~/programming/arm7tdmi_aeabi $ docker run --platform linux/amd64 hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
2db29710123e: Pull complete 
Digest: sha256:53f1bbee2f52c39e41682ee1d388285290c5c8a76cc92b42687eecf38e0af3f0
Status: Downloaded newer image for hello-world:latest
standard_init_linux.go:219: exec user process caused: exec format error

lokathor@skarmory:~/programming/arm7tdmi_aeabi $ docker run --platform linux/amd64 ubuntu:20.04 dpkg --print-architecture
Unable to find image 'ubuntu:20.04' locally
20.04: Pulling from library/ubuntu
d7bfe07ed847: Pull complete 
Digest: sha256:fd92c36d3cb9b1d027c4d2a72c6bf0125da82425fc2ca37c414d4f010180dc19
Status: Downloaded newer image for ubuntu:20.04
standard_init_linux.go:219: exec user process caused: exec format error

Alexhuszagh · 2022-07-28T05:25:54Z

Yep, then it's a Qemu emulation error. I wonder if Podman has support for this. This likely should go in our Wiki for known bugs. If you would like to use cross on an aarch64 Linux host, you may be able to use Podman (no guarantees, however). The installation instructions are here, and you can use the same 2 above commands to test if it works:

podman run --platform linux/amd64 hello-world
podman run --platform linux/amd64 ubuntu:20.04 dpkg --print-architecture

If it works, you can provide CROSS_CONTAINER_ENGINE=podman to cross and it will work out-of-the-box. In fact, Podman is generally better since it runs rootless by default on Linux.

Alexhuszagh · 2022-07-28T05:29:11Z

In the meantime, however, I've confirmed that specific target works out-of-the-box with a native aarch64 Linux Docker image. I can export it to a tarball and send it to you if desired (or try to host it via Dockerhub or something). The image was built via cargo build-docker-image --platform linux/arm64/v8=aarch64-unknown-linux-gnu arm-unknown-linux-gnueabi

Lokathor · 2022-07-28T05:29:21Z

Appears that the same podman problem I had at the start of the issue still applies to those too:

lokathor@skarmory:~/programming/arm7tdmi_aeabi $ podman run --platform linux/amd64 hello-world
Resolved "hello-world" as an alias (/etc/containers/registries.conf.d/shortnames.conf)
Trying to pull docker.io/library/hello-world:latest...
Getting image source signatures
Copying blob 2db29710123e done  
Copying config feb5d9fea6 done  
Writing manifest to image destination
Storing signatures
{"msg":"exec container process `/hello`: Exec format error","level":"error","time":"2022-07-28T05:27:34.000651929Z"}

lokathor@skarmory:~/programming/arm7tdmi_aeabi $ podman run --platform linux/amd64 ubuntu:20.04 dpkg --print-architecture
Resolved "ubuntu" as an alias (/etc/containers/registries.conf.d/shortnames.conf)
Trying to pull docker.io/library/ubuntu:20.04...
Getting image source signatures
Copying blob d7bfe07ed847 done  
Copying config 20fffa419e done  
Writing manifest to image destination
Storing signatures
{"msg":"exec container process `/usr/bin/dpkg`: Exec format error","level":"error","time":"2022-07-28T05:28:29.000668079Z"}

Alexhuszagh · 2022-07-28T05:30:47Z

Want me to send you a tarball of an image for arm-unknown-linux-gnueabi? I have a native aarch Linux image for that target on my local machine. Not sure the best way to send it to you. I can upload it to Dockerhub and provide instructions here (currently, it's a little rough-around-the-edges and requires Cross.toml). Not a great solution, but it is a solution.

I've confirmed it passes all of cross's tests emulated via Qemu on an x86_64 host.

Lokathor · 2022-07-28T05:31:32Z

I can wait for the full setup stuff to be done at your normal pace and in whatever hosting situation you need to figure out.

I've got cross working on a box I can ssh into, which is enough for my current needs of testing just one crate full of assembly, so there's not a huge rush.

Alexhuszagh · 2022-07-28T05:33:02Z

I already have the tarball, so it's just docker save, send the file, and then docker load on your end. Then Cross.toml would be something like:

[target.arm-unknown-linux-gnueabi]
name = "ghcr.io/cross-rs/arm-unknown-linux-gnueabi:local"
toolchain = ["linux/arm64=aarch64-unknown-linux-gnu"]

I was working on this problem as we were speaking so it's no big deal (and actually had built this image 2 hours ago just for this reason).

Lokathor · 2022-07-28T05:34:27Z

uh, sure. i'm not sure how you can send it easily though. aren't they like hundreds of megabytes?

Alexhuszagh · 2022-07-28T05:35:00Z

uh, sure. i'm not sure how you can send it easily though. aren't they like hundreds of megabytes?

Yeah, I can probably upload it to Dockerhub (similar to a Github registry).

Lokathor · 2022-07-28T05:39:09Z

Alright. I can probably try that tomorrow once it's uploaded.

Alexhuszagh · 2022-07-28T05:47:48Z

Alright. I can probably try that tomorrow once it's uploaded.

Image is docker.io/ahuszagh/aarch64-cross:arm-unknown-linux-gnueabi.

Set this to be your Cross.toml:

[target.arm-unknown-linux-gnueabi]
name = "docker.io/ahuszagh/aarch64-cross:arm-unknown-linux-gnueabi"
toolchain = ["linux/arm64=aarch64-unknown-linux-gnu"]

The image details are here just in case you want to verify it.

Update: added docker.io for the registry so it works implicitly with podman too.

Alexhuszagh · 2022-07-28T05:54:21Z

Wait this might have been something we assumed, because Docker comes with binfmt support by default on x86_64 (I believe, it worked out-of-the-box for me), but it might not on aarch64.

Try the following and see if it works:

sudo apt-get install qemu binfmt-support qemu-user-static
docker run --privileged --rm tonistiigi/binfmt --install all

This installs Qemu and binfmt (recognizes the binary format and chooses the right Qemu emulator to use) so the containers can run automatically. Just know that emulated Docker images are quite slow (my builds for these cross-compiled emulators take hours, if not longer, for builds that normally take 15 min). Using the aarch64 Linux image may be much faster.

Emilgardis · 2022-07-28T06:24:27Z

ah ofc, it's definitely that, I hit the same issue on my M1 and just used tonistiigi/binfmt without really thinking about it... my bad! Hopefully it's that simple

Lokathor · 2022-07-29T05:09:22Z

Seems to be working!

jemalloc is really loud though, here's what happened.

lokathor@skarmory:~ $ sudo apt-get install qemu binfmt-support qemu-user-static
(...downloading and unpacking...)
Created symlink /etc/systemd/system/multi-user.target.wants/binfmt-support.service → /lib/systemd/system/binfmt-support.service.
Processing triggers for man-db (2.9.4-2) ...
Scanning processes...                                                                                                                                                                                                                         
Scanning processor microcode...                                                                                                                                                                                                               
Scanning linux images...                                                                                                                                                                                                                      

Running kernel seems to be up-to-date.

Failed to check for processor microcode upgrades.

No services need to be restarted.

No containers need to be restarted.

No user sessions are running outdated binaries.

lokathor@skarmory:~ $ docker run --privileged --rm tonistiigi/binfmt --install all
Unable to find image 'tonistiigi/binfmt:latest' locally
latest: Pulling from tonistiigi/binfmt
43a8a261ebd9: Pull complete 
fbbe11fe460b: Pull complete 
Digest: sha256:5bf63a53ad6222538112b5ced0f1afb8509132773ea6dd3991a197464962854e
Status: Downloaded newer image for tonistiigi/binfmt:latest
{
  "supported": [
    "linux/arm64",
    "linux/amd64",
    "linux/riscv64",
    "linux/ppc64le",
    "linux/s390x",
    "linux/386",
    "linux/mips64le",
    "linux/mips64",
    "linux/arm/v7",
    "linux/arm/v6"
  ],
  "emulators": [
    "python3.9",
    "qemu-alpha",
    "qemu-armeb",
    "qemu-cris",
    "qemu-hppa",
    "qemu-i386",
    "qemu-m68k",
    "qemu-microblaze",
    "qemu-mips",
    "qemu-mips64",
    "qemu-mips64el",
    "qemu-mipsel",
    "qemu-mipsn32",
    "qemu-mipsn32el",
    "qemu-ppc",
    "qemu-ppc64",
    "qemu-ppc64le",
    "qemu-riscv32",
    "qemu-riscv64",
    "qemu-s390x",
    "qemu-sh4",
    "qemu-sh4eb",
    "qemu-sparc",
    "qemu-sparc32plus",
    "qemu-sparc64",
    "qemu-x86_64",
    "qemu-xtensa",
    "qemu-xtensaeb"
  ]
}

lokathor@skarmory:~ $ cd programming/
lokathor@skarmory:~/programming $ cd arm7tdmi_aeabi/
lokathor@skarmory:~/programming/arm7tdmi_aeabi $ git pull
(...git stuff...)

lokathor@skarmory:~/programming/arm7tdmi_aeabi $ cross test --target arm-unknown-linux-gnueabi
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
   Compiling libc v0.2.126
   Compiling cfg-if v1.0.0
   Compiling bytemuck v1.10.0
   Compiling arm7tdmi_aeabi v0.1.3 (/project)
<jemalloc>: MADV_DONTNEED does not work (memset will be used instead)
<jemalloc>: (This is the expected behaviour if you are running under QEMU)
<jemalloc>: MADV_DONTNEED does not work (memset will be used instead)
<jemalloc>: (This is the expected behaviour if you are running under QEMU)
<jemalloc>: MADV_DONTNEED does not work (memset will be used instead)
<jemalloc>: (This is the expected behaviour if you are running under QEMU)
<jemalloc>: MADV_DONTNEED does not work (memset will be used instead)
<jemalloc>: (This is the expected behaviour if you are running under QEMU)
<jemalloc>: MADV_DONTNEED does not work (memset will be used instead)
<jemalloc>: (This is the expected behaviour if you are running under QEMU)
   Compiling getrandom v0.2.7
<jemalloc>: MADV_DONTNEED does not work (memset will be used instead)
<jemalloc>: (This is the expected behaviour if you are running under QEMU)
<jemalloc>: MADV_DONTNEED does not work (memset will be used instead)
<jemalloc>: MADV_DONTNEED does not work (memset will be used instead)
<jemalloc>: (This is the expected behaviour if you are running under QEMU)
<jemalloc>: (This is the expected behaviour if you are running under QEMU)
    Finished test [unoptimized + debuginfo] target(s) in 1m 28s
     Running unittests src/lib.rs (/target/arm-unknown-linux-gnueabi/debug/deps/arm7tdmi_aeabi-49e690dd3b375a6b)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.12s

     Running tests/assembly_code_tests.rs (/target/arm-unknown-linux-gnueabi/debug/deps/assembly_code_tests-706df36395c0f42d)

running 10 tests
test test_aeabi_idiv ... ok
test test_aeabi_idivmod ... ok
test test_aeabi_uidiv ... ok
test test_aeabi_uidivmod ... ok
test test_aeabi_uread4 ... ok
test test_aeabi_uread8 ... ok
test test_aeabi_uwrite4 ... ok
test test_aeabi_uwrite8 ... ok
test test_libc_memmove ... ok
test test_libc_memset ... ok

test result: ok. 10 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 7.39s

So, I guess maybe notes about this need to go in the wiki or something?

(PS: sorry about the delay in getting back to this issue, but the device this is on is in set up in a distant room with no AC, so with the heat wave on I'm only going out there a little bit between like 11pm and midnight.)

Alexhuszagh · 2022-07-29T05:30:10Z

Seems to be working!

Wonderful, glad to hear it.

So, I guess maybe notes about this need to go in the wiki or something?

For sure, good idea.

(PS: sorry about the delay in getting back to this issue, but the device this is on is in set up in a distant room with no AC, so with the heat wave on I'm only going out there a little bit between like 11pm and midnight.)

No worries, I hope everything's ok. You're also helping us out (and the documentation), so there's absolutely no rush. Thanks for all the help.

Lokathor · 2022-07-29T05:32:31Z

Oh no worries the main house is safe and cool, but the raspberry pi is just over in a shed in the back yard is all.

Alexhuszagh · 2022-07-29T17:39:28Z

This has been documented on the wiki under FAQ/Exec Format Error. Let me know if anything else needs to be added.

Emilgardis added information-required more information is required to reproduce A-aarch64-host Area: ARMv8 hosts labels Jul 23, 2022

Lokathor changed the title ~~Strange behavior on aarch64 linux~~ Strange Podman behavior on aarch64 linux Jul 28, 2022

Lokathor changed the title ~~Strange Podman behavior on aarch64 linux~~ Strange behavior on aarch64 linux Jul 28, 2022

Alexhuszagh added upstream A-container-engine Area: container engines labels Jul 28, 2022

Emilgardis removed the information-required more information is required to reproduce label Jul 29, 2022

Alexhuszagh closed this as completed Jul 29, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Strange behavior on aarch64 linux #970

Strange behavior on aarch64 linux #970

Lokathor commented Jul 23, 2022

Emilgardis commented Jul 23, 2022

Lokathor commented Jul 25, 2022

Emilgardis commented Jul 25, 2022

Lokathor commented Jul 28, 2022

Alexhuszagh commented Jul 28, 2022 •

edited

Loading

Lokathor commented Jul 28, 2022

Alexhuszagh commented Jul 28, 2022 •

edited

Loading

Lokathor commented Jul 28, 2022 •

edited

Loading

Alexhuszagh commented Jul 28, 2022 •

edited

Loading

Alexhuszagh commented Jul 28, 2022

Lokathor commented Jul 28, 2022

Alexhuszagh commented Jul 28, 2022 •

edited

Loading

Lokathor commented Jul 28, 2022

Alexhuszagh commented Jul 28, 2022 •

edited

Loading

Lokathor commented Jul 28, 2022

Alexhuszagh commented Jul 28, 2022

Lokathor commented Jul 28, 2022

Alexhuszagh commented Jul 28, 2022 •

edited

Loading

Alexhuszagh commented Jul 28, 2022 •

edited

Loading

Emilgardis commented Jul 28, 2022

Lokathor commented Jul 29, 2022 •

edited

Loading

Alexhuszagh commented Jul 29, 2022

Lokathor commented Jul 29, 2022 •

edited

Loading

Alexhuszagh commented Jul 29, 2022

Strange behavior on aarch64 linux #970

Strange behavior on aarch64 linux #970

Comments

Lokathor commented Jul 23, 2022

Emilgardis commented Jul 23, 2022

Lokathor commented Jul 25, 2022

Emilgardis commented Jul 25, 2022

Lokathor commented Jul 28, 2022

Alexhuszagh commented Jul 28, 2022 • edited Loading

Lokathor commented Jul 28, 2022

Alexhuszagh commented Jul 28, 2022 • edited Loading

Lokathor commented Jul 28, 2022 • edited Loading

Alexhuszagh commented Jul 28, 2022 • edited Loading

Alexhuszagh commented Jul 28, 2022

Lokathor commented Jul 28, 2022

Alexhuszagh commented Jul 28, 2022 • edited Loading

Lokathor commented Jul 28, 2022

Alexhuszagh commented Jul 28, 2022 • edited Loading

Lokathor commented Jul 28, 2022

Alexhuszagh commented Jul 28, 2022

Lokathor commented Jul 28, 2022

Alexhuszagh commented Jul 28, 2022 • edited Loading

Alexhuszagh commented Jul 28, 2022 • edited Loading

Emilgardis commented Jul 28, 2022

Lokathor commented Jul 29, 2022 • edited Loading

Alexhuszagh commented Jul 29, 2022

Lokathor commented Jul 29, 2022 • edited Loading

Alexhuszagh commented Jul 29, 2022

Alexhuszagh commented Jul 28, 2022 •

edited

Loading

Alexhuszagh commented Jul 28, 2022 •

edited

Loading

Lokathor commented Jul 28, 2022 •

edited

Loading

Alexhuszagh commented Jul 28, 2022 •

edited

Loading

Alexhuszagh commented Jul 28, 2022 •

edited

Loading

Alexhuszagh commented Jul 28, 2022 •

edited

Loading

Alexhuszagh commented Jul 28, 2022 •

edited

Loading

Alexhuszagh commented Jul 28, 2022 •

edited

Loading

Lokathor commented Jul 29, 2022 •

edited

Loading

Lokathor commented Jul 29, 2022 •

edited

Loading