Issue with latest Docker hub arm64 image

[manalejandro]

Checklist

• This is a bug report, not a question. Ask questions on discuss.ipfs.io.
• I have searched on the issue tracker for my bug.
• I am running the latest kubo version or have an issue updating.

Installation method

ipfs-desktop

Version

No response

Config

https://hub.docker.com/layers/ipfs/kubo/latest/images/sha256-0b53c782c90a6486ced5411f29085b4aa784723fc75a21ac7223aab0206e1964

Description

Could be one architecture setting?

ipfs | exec /entrypoint.sh: exec format error

[lidel]

@manalejandro what is the platform you are using for running this?
Are you able to test an image published before May 10 and see if the problem is still there?

[manalejandro]

@manalejandro what is the platform you are using for running this? Are you able to test an image published before May 10 and see if the problem is still there?

I have it deployed on one Pi4, something strange happened, i download this image from the hub and it doesn't exist.

REPOSITORY                             TAG             IMAGE ID                                                                  CREATED        SIZE
ipfs/kubo                              master-latest   sha256:2f93a1c01b4cb8aab338fde8cbcd04e216b48a2d1dcc33d1065e9d5d05df736b   9 hours ago    76.6MB

[manalejandro]

Downloading the correct image the same error occurs when executing the IPFS command, it smells to me that it is the architecture in the CI.

$ docker pull ipfs/kubo@sha256:d20c43407cd520f01bbe5b692f18d3287a8264a875ac76d542d23330a14b913e
$ docker tag ipfs/kubo@sha256:d20c43407cd520f01bbe5b692f18d3287a8264a875ac76d542d23330a14b913e ipfs/kubo:master-latest

ipfs  | exec /sbin/tini: exec format error
ipfs exited with code 0

[lidel]

@manalejandro Are you able to test an image for v0.19.2 and see if the problem is still there?

[manalejandro]

@manalejandro Are you able to test an image for v0.19.2 and see if the problem is still there?

that image works well.

ipfs  | ipfs version 0.19.2
ipfs  | Found IPFS fs-repo at /data/ipfs
ipfs  | Initializing daemon...

[manalejandro]

If I build the docker image with the latest version it works perfectly, could be some parameter that affects the architecture when building it for image deployment.

ipfs  | ipfs version 0.21.0-dev
ipfs  | Found IPFS fs-repo at /data/ipfs
ipfs  | Initializing daemon...
ipfs  | Kubo version: 0.21.0-dev-61f8c73
ipfs  | Repo version: 13
ipfs  | System version: arm64/linux
ipfs  | Golang version: go1.19.1

[lidel]

@manalejandro interesting, how are you building the image? docker build -t kubo-test . ?
Did you docker pull golang:1.19-buster and docker pull busybox:1.31.1-glibc beforehand, just to be sure you use same tags as CI?

[manalejandro]

@manalejandro interesting, how are you building the image? docker build -t kubo-test . ? Did you docker pull golang:1.19-buster and docker pull busybox:1.31.1-glibc beforehand, just to be sure you use same tags as CI?

Hi, I cross compile the last commit with docker from a x64 machine for the Pi4 arm64 image:

$ docker run --rm --privileged multiarch/qemu-user-static --reset -p yes

$ git clone https://github.com/ipfs/kubo && cd kubo

$ docker buildx build --platform arm64 -t ipfs .

The last commit I tried was 61f8c73 but now it doesn't work with the latest code from github.

[manalejandro]

@manalejandro interesting, how are you building the image? docker build -t kubo-test . ? Did you docker pull golang:1.19-buster and docker pull busybox:1.31.1-glibc beforehand, just to be sure you use same tags as CI?

Hi, I cross compile the last commit with docker from a x64 machine for the Pi4 arm64 image:

$ docker run --rm --privileged multiarch/qemu-user-static --reset -p yes

$ git clone https://github.com/ipfs/kubo && cd kubo

$ docker buildx build --platform arm64 -t ipfs .

The last commit I tried was 61f8c73 but now it doesn't work with the latest code from github.

Sorry I'm very rude and didn't look at the Dockerfile arguments, now the last commit works.

$ docker buildx build --build-arg "BUILDPLATFORM=linux/arm64" --build-arg "TARGETPLATFORM=linux/arm64" --build-arg "TARGETOS=linux" --build-arg "TARGETARCH=arm64" --platform arm64 -t 192.168.1.124:5000/ipfs .

Kubo version: 0.22.0-dev-f797e9e

[lidel]

@manalejandro ack. Just want to confirm: you are building your own image because the upstream one does not work, right?

[manalejandro]

@manalejandro ack. Just want to confirm: you are building your own image because the upstream one does not work, right?

Yes, that's the point of my issue.

[pablomendezroyo]

same issue

[Marketen]

With docker versions v0.20.0 & v0.21.0 I have the same issue. When building an image on an arm64 device with these versions, a docker inspect <image-tag> returns "Architecture": "arm64", the expected architecture.

However, when executing docker run --rm --entrypoint="" <image-tag> uname -m to get current architecture, the docker container built with these images retuns x86_64.

This does not happen on versions v0.19.0 and lower. where the same command docker run --rm --entrypoint="" <image-tag> uname -m returns the expected architecture on the container build with arm64: aarch64

[kallisti5]

Same issue here. 0.19.1, 0.19.2 work ok. 0.20.0 and 0.21.0 both error out like you packed your arm64 build in a x86 container.

{"msg":"exec container process `/sbin/tini`: Exec format error","level":"error","time":"2023-07-13T14:38:22.392159Z"}

See jesec/flood#451 (comment) for a similar issue.

More data:

pi:~# uname -a
Linux pi 5.15.93-0-rpi4 #1-Alpine SMP PREEMPT Thu Feb 9 23:56:17 UTC 2023 aarch64 Linux

pi:~# podman run -it --entrypoint /bin/sh docker.io/ipfs/kubo:v0.20.0
{"msg":"exec container process `/bin/sh`: Exec format error","level":"error","time":"2023-07-13T14:46:12.600165Z"}

pi:~# podman run -it --entrypoint ipfs docker.io/ipfs/kubo:v0.20.0
USAGE
  ipfs  - Global p2p merkle-dag filesystem.

  ipfs [--config=<config> | -c] [--debug | -D] [--help] [-h] [--api=<api>] [--offline] [--cid-base=<base>] [--upgrade-cidv0-in-output] [--encoding=<encoding> | --enc] [--timeout=<timeout>] <command> ...

SUBCOMMANDS
  BASIC COMMANDS
    init          Initialize local IPFS configuration
    add <path>    Add a file to IPFS
    cat <ref>     Show IPFS object data
    get <ref>     Download IPFS objects
    ls <ref>      List links from an object
    refs <ref>    List hashes of links from an object

The above confirms that the chroot is all x86_64 or something... and the ipfs binary itself is arm64. If you're doing some multi-stage built, your final step is using an x86_64/amd64 container to install the arm64 kubo into.

[kallisti5]

suspect: 6f08cde

Maybe your caching image layers means it's caching x86 layers and not figuring out the arch is different? 3 minutes seems way to quick to pull multiple base images of differing architectures.

Also of note:
6f08cde#diff-dd2c0eb6ea5cfc6c4bd4eac30934e2d5746747af48fef6da689e85b752f39557R54

Maybe the target image is getting the default arm64 because BUILDPLATFORM is unset?

[samsapti]

Also of note:
6f08cde#diff-dd2c0eb6ea5cfc6c4bd4eac30934e2d5746747af48fef6da689e85b752f39557R54

Maybe the target image is getting the default arm64 because BUILDPLATFORM is unset?

@kallisti5 you're on to something, but it's not the caching that's causing the problem. It's this:

--platform=${BUILDPLATFORM:-linux/amd64}

It's not because BUILDPLATFORM is unset, it's set. It's because it's the wrong variable that's used. BUILDPLATFORM is the platform that the image is built ON, while TARGETPLATFORM is the platform the image is built FOR. So BUILDPLATFORM is always set to linux/amd64, and the variable that should be used in the final stage is actually TARGETPLATFORM, like so:

FROM --platform=${TARGETPLATFORM:-linux/amd64} busybox:1.31.1-glibc

[kallisti5]

AAH. Yup. I missed that lol. You're compiling on BUILDPLATFORM for TARGETPLATFORM.. so the installed container needs to be TARGETPLATFORM instead of BUILDPLATFORM

[deavmi]

With docker versions v0.20.0 & v0.21.0 I have the same issue. When building an image on an arm64 device with these versions, a docker inspect <image-tag> returns "Architecture": "arm64", the expected architecture.

However, when executing docker run --rm --entrypoint="" <image-tag> uname -m to get current architecture, the docker container built with these images retuns x86_64.

This does not happen on versions v0.19.0 and lower. where the same command docker run --rm --entrypoint="" <image-tag> uname -m returns the expected architecture on the container build with arm64: aarch64

God help me, I am testing this now because I have the same issue on latest

[lidel]

Upstream image was fixed by #10021 today.
Once we release Kubo 0.22, the upstream latest image will be fixed too.

Your options here:

• (A) build image yourself until Kubo 0.22 is released
• (B) use the master- dev images until kubo 0.22 is released
  • starting with ipfs/kubo:master-2023-07-27-c5868a8, it works fine on my rPI (arm64):

    

[deavmi]

Upstream image was fixed by #10021 today. Once we release Kubo 0.22, the upstream latest image will be fixed too.

For now, you can use the master-latest, starting with ipfs/kubo:master-2023-07-27-c5868a8, it works fine on my rPI (arm64):

Thank you, this seems to work as well!

(A little off tangent question, where's the ipfs config file created within this image?)

[lidel]

@deavmi IPFS_HOME is at /data/ipfs but you may want to customize your instance via scripts in /container-init.d instead – see https://docs.ipfs.tech/install/run-ipfs-inside-docker/#customizing-your-node
(In the future, ask support questions like this at https://discuss.ipfs.tech/c/help/13 🙏 )