"fill out specgen: conflict at mount destination /.../: duplicate mount destination"

[cjw296]

Issue Description

When trying to create a container with a mounted config directory using docker-py, I gety the following error:

docker.errors.APIError: 500 Server Error for http+docker://localhost/v1.40/containers/create: 
Internal Server Error ("fill out specgen: conflict at mount destination /etc/clickhouse-server/config.d/: duplicate mount destination")

Steps to reproduce the issue

I've put together a reproducer here, the crux of the code that fails is:

client = docker.from_env()
client.containers.run(
        image_tag,
        ports={'9000/tcp': 0},
        detach=True,
        auto_remove=True,
        volumes={CLICKHOUSE_CONFIG: {'bind': '/etc/clickhouse-server/config.d/', 'mode': 'ro'}},
)

If you want to use the full reproducer, clone https://github.com/cjw296/podman-issues and follow the README.

Describe the results you received

DOCKER_HOST=unix://$XDG_RUNTIME_DIR/podman/podman.sock python3 dockerpy-reproducer.py all
postgres running and exposed on localhost:45677
Traceback (most recent call last):
  File "/opt/circleci/.pyenv/versions/3.11.1/lib/python3.11/site-packages/docker/api/client.py", line 268, in _raise_for_status
    response.raise_for_status()
  File "/opt/circleci/.pyenv/versions/3.11.1/lib/python3.11/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http+docker://localhost/v1.40/containers/create

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/circleci/project/dockerpy-reproducer.py", line 72, in <module>
    cli()
  File "/opt/circleci/.pyenv/versions/3.11.1/lib/python3.11/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/circleci/.pyenv/versions/3.11.1/lib/python3.11/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/opt/circleci/.pyenv/versions/3.11.1/lib/python3.11/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/circleci/.pyenv/versions/3.11.1/lib/python3.11/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/circleci/.pyenv/versions/3.11.1/lib/python3.11/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/circleci/.pyenv/versions/3.11.1/lib/python3.11/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/circleci/project/dockerpy-reproducer.py", line 68, in all
    ctx.invoke(clickhouse_container)
  File "/opt/circleci/.pyenv/versions/3.11.1/lib/python3.11/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/circleci/project/dockerpy-reproducer.py", line 50, in clickhouse_container
    container = client.containers.run(
                ^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/circleci/.pyenv/versions/3.11.1/lib/python3.11/site-packages/docker/models/containers.py", line 847, in run
    container = self.create(image=image, command=command,
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/circleci/.pyenv/versions/3.11.1/lib/python3.11/site-packages/docker/models/containers.py", line 906, in create
    resp = self.client.api.create_container(**create_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/circleci/.pyenv/versions/3.11.1/lib/python3.11/site-packages/docker/api/container.py", line 431, in create_container
    return self.create_container_from_config(config, name, platform)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/circleci/.pyenv/versions/3.11.1/lib/python3.11/site-packages/docker/api/container.py", line 448, in create_container_from_config
    return self._result(res, True)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/circleci/.pyenv/versions/3.11.1/lib/python3.11/site-packages/docker/api/client.py", line 274, in _result
    self._raise_for_status(response)
  File "/opt/circleci/.pyenv/versions/3.11.1/lib/python3.11/site-packages/docker/api/client.py", line 270, in _raise_for_status
    raise create_api_error_from_http_exception(e) from e
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/circleci/.pyenv/versions/3.11.1/lib/python3.11/site-packages/docker/errors.py", line 39, in create_api_error_from_http_exception
    raise cls(e, response=response, explanation=explanation) from e
docker.errors.APIError: 500 Server Error for http+docker://localhost/v1.40/containers/create: Internal Server Error ("fill out specgen: conflict at mount destination /etc/clickhouse-server/config.d/: duplicate mount destination")

Describe the results you expected

python3 dockerpy-reproducer.py all
postgres running and exposed on localhost:32768
clickhouse running and exposed on localhost:32769

podman info output

host:
  arch: amd64
  buildahVersion: 1.23.1
  cgroupControllers:
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: 'conmon: /usr/bin/conmon'
    path: /usr/bin/conmon
    version: 'conmon version 2.0.25, commit: unknown'
  cpus: 4
  distribution:
    codename: jammy
    distribution: ubuntu
    version: "22.04"
  eventLogger: journald
  hostname: ip-10-0-3-78
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1002
      size: 1
    - container_id: 1
      host_id: 165536
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1001
      size: 1
    - container_id: 1
      host_id: 165536
      size: 65536
  kernel: 5.15.0-1028-aws
  linkmode: dynamic
  logDriver: journald
  memFree: 15188135936
  memTotal: 16468393984
  ociRuntime:
    name: crun
    package: 'crun: /usr/bin/crun'
    path: /usr/bin/crun
    version: |-
      crun version 0.17
      commit: 0e9229ae34caaebcb86f1fde18de3acaf18c6d9a
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/user/1001/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: 'slirp4netns: /usr/bin/slirp4netns'
    version: |-
      slirp4netns version 1.0.1
      commit: 6a7b16babc95b6a3056b33fb45b74a6f62262dd4
      libslirp: 4.6.1
  swapFree: 0
  swapTotal: 0
  uptime: 1m 58.32s
plugins:
  log:
  - k8s-file
  - none
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries: {}
store:
  configFile: /home/circleci/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/circleci/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 0
  runRoot: /run/user/1001/containers
  volumePath: /home/circleci/.local/share/containers/storage/volumes
version:
  APIVersion: 3.4.4
  Built: 0
  BuiltTime: Thu Jan  1 00:00:00 1970
  GitCommit: ""
  GoVersion: go1.17.3
  OsArch: linux/amd64
  Version: 3.4.4

Podman in a container

No

Privileged Or Rootless

None

Upstream Latest Release

Yes

Additional environment details

I've reproduced on a relatively new podman v4 as well, and haven't seen this reported before...

Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting

[Luap99]

Please try this on the latest release 4.5.
Duplicate of #11822

[cjw296]

Yeah, same issue on:

$ podman --version
podman version 4.4.4

[cjw296]

@Luap99 - is there any way to work around this (hacks accepted...) for older versions of podman?
I tried getting the latest version using the instructions but it didn't go well.

[Luap99]

The old issue should be fixed in 4.4. I tried you reproducer against main and it still fails, likely a different cause, will reopen

[Luap99]

podman debug logs show this

DEBU[0005] User mount /home/pholzing/test/podman-issues/config:/etc/clickhouse-server/config.d/ options [ro] 
DEBU[0005] User mount :/etc/clickhouse-server/config.d/ options [] 

[Luap99]

If you remove the final / from you directory it should work, I see the problem in the code.

so volumes={CLICKHOUSE_CONFIG: {'bind': '/etc/clickhouse-server/config.d', 'mode': 'ro'}},