Pack 0.30.0 fails to fetch builder with Docker engine 24.0.5 on remote host over SSH

[phroggyy]

Summary

When using pack 0.30.0 and running pack build against a remote docker host (set with DOCKER_HOST) running Docker engine 24.0.5, pack errors with a call to http://dummy.

This looks like it is possibly related to docker/cli#4437 and by extension moby/moby#45942.

---

Reproduction

Steps

1. Setup a remote host with Docker
2. Install pack-cli 0.30.0 (apt install pack-cli)
3. build an image with an explicit builder: DOCKER_HOST=ssh://builder@1.2.3.4 pack build --builder=heroku/builder-classic:22

Current behavior

An error is returned

ERROR: failed to build: failed to fetch builder image 'index.docker.io/heroku/builder-classic:22': error during connect: Post "http://dummy%2F/v1.38/images/create?fromImage=heroku%2Fbuilder-classic&tag=22": http: invalid Host header

Expected behavior

I expected pack to run without errors

---

Environment

pack info

➜ pack report
Pack:
  Version:  0.30.0+git-c38f7da.build-4952
  OS/Arch:  darwin/arm64

Default Lifecycle Version:  0.17.0

Supported Platform APIs:  0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 0.10, 0.11, 0.12

docker info

From running with DOCKER_HOST=ssh://builder@my.ip.address

Client:
 Version:    24.0.5
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.11.2-desktop.1
    Path:     /Users/leosjoberg/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.20.2-desktop.1
    Path:     /Users/leosjoberg/.docker/cli-plugins/docker-compose
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.0
    Path:     /Users/leosjoberg/.docker/cli-plugins/docker-dev
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.20
    Path:     /Users/leosjoberg/.docker/cli-plugins/docker-extension
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v0.1.0-beta.6
    Path:     /Users/leosjoberg/.docker/cli-plugins/docker-init
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /Users/leosjoberg/.docker/cli-plugins/docker-sbom
  scan: Docker Scan (Docker Inc.)
    Version:  v0.26.0
    Path:     /Users/leosjoberg/.docker/cli-plugins/docker-scan
  scout: Command line tool for Docker Scout (Docker Inc.)
    Version:  0.20.0
    Path:     /Users/leosjoberg/.docker/cli-plugins/docker-scout

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 3
 Server Version: 24.0.5
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 8165feabfdfe38c65b599c4993d227328c231fca
 runc version: v1.1.8-0-g82f18fe
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 5.19.0-1029-aws
 Operating System: Ubuntu 22.04.3 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 15.27GiB
 Name: ip-10-25-181-35
 ID: 640b3a6c-afb3-4170-9d94-71a37c80109c
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

[jjbustamante]

Hi @phroggyy

Thanks for reporting this one, I will try to reproduce it, we have some other related tickets and I want to check if this is the same error. Once I reproduce it I will move it to ready state and try to add it to a milestone.

[jjbustamante]

Hi @matejvasek

Just wanted to ping you about this error to check if have some idea, just because I think the error is being generated by the configuration at this line added by this PR

[matejvasek]

@jjbustamante you are probably right, back then I assumed host should contain protocol, but it should not.

[matejvasek]

it probably should be just dummy.

[matejvasek]

@phroggyy Would you verify that changing http://dummy/ => dummy at

pack/cmd/docker_init.go

Line 55 in c38f7da

dockerClient.WithHost("http://dummy/"),

fixes the issue?

[jjbustamante]

@matejvasek I tried that but got the following error:

> DOCKER_HOST=ssh://jbustamante@34.29.52.176 pack build -p ruby-bundler --builder cnbs/sample-builder:bionic sample-remote-app
ERROR: failed to build: failed to fetch builder image 'index.docker.io/cnbs/sample-builder:bionic': error during connect: Post "http://dummy%2F/v1.38/images/create?fromImage=cnbs%2Fsample-builder&tag=bionic": ssh: rejected: connect failed (open failed)


> DOCKER_HOST=ssh://jbustamante@34.29.52.176 /Users/jbustamante/go/src/github.com/buildpacks/pack/out/pack build -p ruby-bundler --builder cnbs/sample-builder:bionic sample-remote-app
ERROR: unable to parse docker host `dummy`

[matejvasek]

It could be possibly tcp://dummy, or maybe we don't even need to set that at all?

[rochacon]

@matejvasek it seems like we don't need to set it, the client will override it for some protocols: https://github.com/moby/moby/blob/master/client/request.go#L108-L111

Or we could use the github.com/docker/docker/client.DummyHost constant?

[jjbustamante]

@matejvasek

Actually I made a mistake with my original test, after validating that my remote docker host was actually working correctly, I tried again.

I reproduced the error correctly:

> DOCKER_HOST=ssh://jbustamante@34.29.52.176 /Users/jbustamante/go/src/github.com/buildpacks/pack/out/pack build -p ruby-bundler --builder cnbs/sample-builder:bionic sample-remote-app
ERROR: failed to build: failed to fetch builder image 'index.docker.io/cnbs/sample-builder:bionic': error during connect: Post "http://dummy%2F/v1.38/images/create?fromImage=cnbs%2Fsample-builder&tag=bionic": http: invalid Host header

I just removed the last / inhttp://dummy/ => dummy at

pack/cmd/docker_init.go

Line 55 in c38f7da

dockerClient.WithHost("http://dummy/"),

and it worked

> DOCKER_HOST=ssh://jbustamante@34.29.52.176 /Users/jbustamante/go/src/github.com/buildpacks/pack/out/pack build -p ruby-bundler --builder cnbs/sample-builder:bionic sample-remote-app
bionic: Pulling from cnbs/sample-builder
Digest: sha256:1fbc7290c4af32a7803c084aa2e7f0b5a24a22bed6e10701791848d54fd27605
Status: Image is up to date for cnbs/sample-builder:bionic
bionic: Pulling from cnbs/sample-stack-run
Digest: sha256:22f6e78d4f4d541279069a191c27614cb3f935a5a1259e662ca5f22a85d1541a
....

[jjbustamante]

Hi @phroggyy

We just released a RC pack version, it will be great if you can give it a try and confirm the issue was solved