Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

version 2.2.0.0: Error while creating mount source path '/host_mnt/c/code': mkdir /host_mnt/c/code: file exists #5516

Closed
2 tasks done
jlouros opened this issue Jan 21, 2020 · 50 comments

Comments

@jlouros
Copy link

jlouros commented Jan 21, 2020

  • I have tried with the latest version of my channel (Stable or Edge)
  • I have uploaded Diagnostics
  • Diagnostics ID: 5742C908-05AC-4B28-87FE-A1FF54758C53/20200121184954

Expected behavior

should be able to attach volume point to a folder on host

Actual behavior

C:\Program Files\Docker\Docker\resources\bin\docker.exe: Error response from daemon: error while creating mount source path '/host_mnt/c/code': mkdir /host_mnt/c/code: file exists.

Information

  • Is it reproducible? Yes, every time I try to attach a volume pointing to C:\code
  • Is the problem new? Yes, after updating Docker Desktop to version 2.2.0.0
  • Did the problem appear with an update? Yes, with version 2.2.0.0. Worked fine on the previous version
  • Windows Version: Windows 10 Pro version 1909 (OS Build 18363.592)
  • Docker Desktop Version: 2.2.0.0 (42247)
  • Are you running inside a virtualized Windows e.g. on a cloud server or on a mac VM: No

image

Steps to reproduce the behavior

  1. run docker run --rm -v c:/code:/data alpine ls /data
@djs55
Copy link

djs55 commented Jan 21, 2020

@jlouros thanks for your report.

Could you check the canonical case of your directory C:\code -- is it C:\Code or C:\code (or C:\CoDe ...). I've managed to reproduce a similar error by

PS > mkdir \code
PS > docker run --rm -v c:/Code:/data alpine ls /data
C:\Program Files\Docker\Docker\resources\bin\docker.exe: Error response from daemon: error while creating mount source path '/host_mnt/c/Code': mkdir /host_mnt/c/Code: file exists.

PS > docker run --rm -v c:/code:/data alpine ls /data
PS >

but I don't know if this is this is the same error you saw or not.

Unfortunately I can't find the diagnostics upload :( Could you try re-uploading diagnostics? Or failing that the 2 interesting log files would be

  • AppData\Local\Docker\log.txt
  • AppData\Roaming\Docker\log\vm\lifecycle-server.log
    Could you upload those somewhere and send me a link?

@jlouros
Copy link
Author

jlouros commented Jan 21, 2020

hi @djs55

the problem was the canonical case. If I change it to docker run --rm -v c:/Code:/data alpine ls /data (capital C) it works fine.

I was a bit concerned because docker-compose also started failing for me.
Powershell (I guess all Windows-based shells) accept both c:\code and c:\Code.
Since I use relative paths on docker-compose.yml, the startup would fail because the current that was c:\code\my-proj (instead of using c:\Code\my-proj)

Thanks for your help

@jlouros jlouros closed this as completed Jan 21, 2020
@jlouros
Copy link
Author

jlouros commented Jan 21, 2020

after checking the logs I saw the error message:

[19:58:53.178][GoBackendProcess  ][Info   ] time="2020-01-21T19:58:53Z" msg="rejecting access to file using non-canonical name code (canonical name is Code)"

I guess it should be level Error instead of Info

@djs55
Copy link

djs55 commented Jan 21, 2020

@jlouros thanks for the fast confirmation.

I recently made the fileserver more strict about case to avoid problems where the file contents it cached multiple times in the VM under different names, because only the canonical filename would be invalidated when the file on the host has changed. The extra-strictness on docker run -v was not intended. The error message is also terrible. Maybe I can improve the error message at least.

@simonferquel
Copy link

I suspect we could automatically rewrite mount sources to the canonical path as well. We'll talk about it tomorrow :)

@stephen-turner
Copy link
Contributor

Reopening: even though there's a workaround, we still need to fix it.

@tmuka
Copy link

tmuka commented Jan 22, 2020

I'm experiencing this issue too, using both Git Bash, and WSL.
my Diagnostic upload id is 2F761B85-0512-4FD3-A883-643AE53C0115/20200122154524 if that helps. The same docker-compose projects worked fine before this Docker Desktop 2.2 update. And fwiw, i'm not seeing the "non-canonical name" error message in my logs.

@SirRippovMaple
Copy link

I'm also having this issue, with a slight variation. I have the following volume mount in my docker-compose.yml:

    prometheus:
        image: prom/prometheus
        ports:
            - "9090:9090"
        volumes:
            - ./config/prometheus.yml:/etc/prometheus/prometheus.yml
            - prometheus_data:/prometheus

Though I have the correct casing in my docker-compose.yml, when docker-compose converts ./config/prometheus.yml to an absolute path, it will use lowercase for the remainder of the path that it fills in. So if the docker-compose file is in C:\Foo, it will use /host_mnt/c/foo/config/prometheus.yml. Then I get this error:

ERROR: for master_prometheus_1  Cannot start service prometheus: error while creating mount source path '/host_mnt/c/foo/config/prometheus.yml': mkdir /host_mnt/c/foo: file exists

I'll get this in my log file:

[19:41:56.517][GoBackendProcess  ][Info   ] time="2020-01-22T19:41:56-05:00" msg="rejecting access to file using non-canonical name foo (canonical name is Foo)"

@aaronmorgan
Copy link

@trumpi exactly the behaviour I'm seeing, caused by an uppercase character in my absolute path.
I've worked around the problem by checking out to a new path that includes no uppercase characters.

@simonferquel
Copy link

We have a fix incoming for this one, thanks for the (numerous) repro cases 👍

@mat007
Copy link
Member

mat007 commented Jan 23, 2020

Sorry our internal repo PR auto-closed the ticket, we’ll keep it open until it’s been confirmed fixed.

@manucohenyashar
Copy link

IDEs and Tools like SAM CLI internally use regular casing when mounting code and it's not possible to change that, which means that until this issue is solved it all the tools that rely on docker mount will fail.

@stephen-turner
Copy link
Contributor

We have a new private build that (we believe) fixes this bug. https://download-stage.docker.com/win/stable/42579/Docker%20Desktop%20Installer.exe. Please try it and let us know how you get on.

@SirRippovMaple
Copy link

@stephen-turner - thank you, it works!

@Clorith
Copy link

Clorith commented Feb 2, 2020

I'm afraid the private build proposed by @stephen-turner did not work for me, I am still seeing the same error.

Versions:

  • Windows 10 Pro: version 10.0.18363
  • Docker Desktop Community edition: version 2.2.0.1 (42579)
  • Docker Compose: version 1.25.3

Diagnostics ID D054EF4A-AB53-4724-BCC3-607227D90AC2/20200202103217

Seeing the error as this:

ERROR: for wordpress  Cannot start service wordpress: error while creating mount source path '/host_mnt/c/projects/health-check/.docker/config/php-fpm/php.ini': mkdir /host_mnt/c/projects/health-check/.docker/config/php-fpm: file exists

When trying to just do a docker-compose up from my project directory in C:/projects/health-check (I've also double-checked the casing, and also tried using the full path name instead of the relative one), using the repository at https://github.com/WordPress/health-check/tree/develop

This was functioning as expected prior to the 2.2.0 update

@24githubs
Copy link

I'm afraid the private build proposed by @stephen-turner did not work for me, I am still seeing the same error.

Versions:

  • Windows 10 Pro: version 10.0.18363
  • Docker Desktop Community edition: version 2.2.0.1 (42579)
  • Docker Compose: version 1.25.3

Diagnostics ID D054EF4A-AB53-4724-BCC3-607227D90AC2/20200202103217

Seeing the error as this:

ERROR: for wordpress  Cannot start service wordpress: error while creating mount source path '/host_mnt/c/projects/health-check/.docker/config/php-fpm/php.ini': mkdir /host_mnt/c/projects/health-check/.docker/config/php-fpm: file exists

When trying to just do a docker-compose up from my project directory in C:/projects/health-check (I've also double-checked the casing, and also tried using the full path name instead of the relative one), using the repository at https://github.com/WordPress/health-check/tree/develop

This was functioning as expected prior to the 2.2.0 update

+1 same here..
ERROR: for mydb Cannot start service mydb: error while creating mount source path '/host_mnt/c/link/mydb': mkdir /host_mnt/c: file exists

@manucohenyashar
Copy link

manucohenyashar commented Feb 3, 2020 via email

@simonferquel
Copy link

For users experiencing an issue with the private build, can you check if it works when using the same exact casing as the file path on windows?

@tmuka
Copy link

tmuka commented Feb 3, 2020

We have a new private build that (we believe) fixes this bug. https://download-stage.docker.com/win/stable/42579/Docker%20Desktop%20Installer.exe. Please try it and let us know how you get on.

My project came up the first time after installing the private build, but then after stopping it and trying to bring it up a 2nd time I get the same file exists errors as before.

[tmuka@WRKSTN-92L31S2:/d/www/example.org]$ docker-compose down -v
Stopping exampleorg_phpmyadmin_1 ... done
Stopping exampleorg_mysql_1      ... done
Removing exampleorg_phpmyadmin_1 ... done
Removing exampleorg_wordpress_1  ... done
Removing exampleorg_mysql_1      ... done
Removing network exampleorg_default
Removing volume exampleorg_mysql
[tmuka@WRKSTN-92L31S2:/d/www/example.org]$ docker-compose up --force-recreate
Creating network "exampleorg_default" with the default driver
Creating volume "exampleorg_mysql" with default driver
Creating exampleorg_wordpress_1 ... error
Creating exampleorg_mysql_1     ...

Creating exampleorg_mysql_1     ... done
Creating exampleorg_phpmyadmin_1 ... done

ERROR: for wordpress  Cannot start service wordpress: error while creating mount source path '/host_mnt/d/www/example.org': mkdir /host_mnt/d/www/example.org: file exists
ERROR: Encountered errors while bringing up the project.
[tmuka@WRKSTN-92L31S2:/d/www/example.org]$

my volume mount config in docker-compose.yml is relative like this, without any case specified.

    volumes:
      - "./:/var/www/html/wp-content"
    volumes:
      - "mysql:/var/lib/mysql"

edit: my diagnostics id 2F761B85-0512-4FD3-A883-643AE53C0115/20200203145833

@simonferquel
Copy link

@tmuka @Clorith thanks for your diagnostics / repro cases. We identified a case where a panic makes the whole grpcFuse go nuts. We are writing a fix.

@Sieboldianus
Copy link

Sieboldianus commented Feb 6, 2020

I can't, I'm on Windows Education with MS Spy turned off.. 🙄 (WSL2 is only available on Insiders build)

This one appears to refer to the same issue:

So the issue was introduced in Docker Desktop 2.2.0.0. Perhaps downgrading solves this?

@stephen-turner
Copy link
Contributor

We now have a new build for you to try. To recap, we fixed one cause of this error in 2.2.0.1. The new build, 2.2.0.2, fixes the other known cause of the same error. You can get it from https://download-stage.docker.com/win/stable/42644/Docker%20Desktop%20Installer.exe. Everyone who's still seeing this error, please do try it the new build and let us know whether it fixes it for you. Thank you.

@Clorith
Copy link

Clorith commented Feb 6, 2020

Wonderful, works on my end now, thank you all for your hard work!

@PeterBooker
Copy link

That resolved the issue for me too. Much appreciated.

@YouveGotMeowxy
Copy link

YouveGotMeowxy commented Feb 6, 2020

@stephen-turner how long until this gets pushed out to the Edge version? I don't really like swapping channels a bunch of times and I just switched from Stable to Edge. :)

@pixelmixer
Copy link

pixelmixer commented Feb 7, 2020

This issue still persists for me on the 2.2.02 build. I did notice that if I remove mounts that do not have junctions then it works fine. Like everyone else on this thread, this worked fine pre 2.2.0.0.

This does not work in the latest build from this morning. "\Settings\Backups" is a normal directory while "\Share" is a folder junction pointing to an external drive.

volumes:
      - 'C:\Users\pixelmixer\Settings\Backups:/backups'
      - 'C:\Users\pixelmixer\Share:/shares'

This allows the container to start properly if I exclude the directory junction.

volumes:
      - 'C:\Users\pixelmixer\Settings\Backups:/backups'

As anticipated, mounting only the directory with the junction still fails.

volumes:
      - 'C:\Users\pixelmixer\Share:/shares'

@STaRDoGG
Copy link

STaRDoGG commented Feb 7, 2020

Just wanted to note that in my issue above (#5516 (comment)) the path contains no junctions on my system.

@Sieboldianus
Copy link

We now have a new build for you to try. To recap, we fixed one cause of this error in 2.2.0.1. The new build, 2.2.0.2, fixes the other known cause of the same error. You can get it from https://download-stage.docker.com/win/stable/42644/Docker%20Desktop%20Installer.exe. Everyone who's still seeing this error, please do try it the new build and let us know whether it fixes it for you. Thank you.

I can confirm. Works on my end, too! Many thanks!

@STaRDoGG
Copy link

STaRDoGG commented Feb 7, 2020

Side Note: Not the same issue as this thread, but I noticed while reading it that @djs55 mentions changes to the file system strictness:

I recently made the fileserver more strict about case to avoid problems where the file contents it cached multiple times in the VM under different names, because only the canonical filename would be invalidated when the file on the host has changed.

So it made me wonder if it's also somehow causing this issue as well, that I hadn't had before, using the same docker-compose code (previously I was able to run phpmyadmin using the same code): docker-library/php#944

I could be wrong, but thought it might be worth mentioning anyway.

@djs55
Copy link

djs55 commented Feb 7, 2020

@STaRDoGG are you referring to this issue: phpmyadmin/docker#267 ? If so I think this comment is particularly interesting: phpmyadmin/docker#267 (comment) -- without any volume mounts you still saw the error

docker run --name phpmyadmin -d -p 9024:80 --net=my-db -e TZ=America/Chicago -e "PMA_HOST=mysql" -e "MYSQL_ROOT_PASSWORD=redacted" -e "PMA_ABSOLUTE_URI=https://my.domain.rocks:33333/phpmyadmin/" phpmyadmin/phpmyadmin:latest
Unable to find image 'phpmyadmin/phpmyadmin:latest' locally
...
5fe7c1ba6fbd: Download complete
docker: failed to register layer: error creating overlay mount to /var/lib/docker/overlay2/32a716d9ae4c60363ebb11152502fba95ee7589bf28474ff9dbc93519f33b6a9/merged: too many levels of symbolic links.

From your repro I would conclude that

  • the bug isn't in the new volume sharing support since you reproduced it without any -v
  • the "failed to register layer" suggests a problem inside the Linux VM with the overlayfs (which is used to create the filesystems used by the containers by composing together downloaded layers).

Could you confirm that I'm looking at the right comments? (These threads can get a little confusing). If you haven't captured a diagnostics report after a repro, could you do this and quote the id here? The logs in the diagnostics report might be useful. My guess is that this would be a bug in the Docker engine itself -- the part that's shared between Linux, Mac and Windows.

@stephen-turner
Copy link
Contributor

So to summarise, it looks like the issue on this ticket is fixed. Thank you for all your testing.

@pixelmixer has one remaining issue with NTFS junctions. That issue is #5582.

@STaRDoGG
Copy link

STaRDoGG commented Feb 7, 2020

@djs55

Could you confirm that I'm looking at the right comments? (These threads can get a little confusing). If you haven't captured a diagnostics report after a repro, could you do this and quote the id here? The logs in the diagnostics report might be useful. My guess is that this would be a bug in the Docker engine itself -- the part that's shared between Linux, Mac and Windows.

Yessir, those are the right comments you were looking at. It turns out that for some some reason certain containers give that error w/out any volume mounts at all.

i.e.

docker run php:apache and docker run wordpress:apache

I'm not sure why, but this only started happening recently, like I mentioned earlier, I was running phpmyadmin with that same run command (including the volume mounts) not too long ago, but made some changes to other services in a docker compose file (no changes to the phpmyadmin container), removed the phpmyadmin container, then tried to up the stack again, and started getting that error.

I will try to capture a diagnostics report, and post it later on tonight (it's almost 4am here and have to try and ink out a couple hours of sleep before work, lol).

@stephen-turner
Copy link
Contributor

Thanks @STaRDoGG. When you do, could you create a new ticket for it please? Thanks.

@stephen-turner
Copy link
Contributor

We have just released 2.2.0.3, which fixes this issue, so closing this ticket.

@YouveGotMeowxy It will be on Edge within 24 hours too.

@ermiaswalelgne
Copy link

Finally, I can also confirm that the new build ( https://download-stage.docker.com/win/stable/42579/Docker%20Desktop%20Installer.exe ) from @stephen-turner fixes this bug that I have been struggling with. Thanks a lot. I have to mention that first I need to clean all previous docker installations including docker toolkit. Then I had to share my C drive from docker settings > Resources > File sharing

@STaRDoGG
Copy link

Unfortunately for me, I still have the same issue with an overlay error (which I don't really understand).

I just installed:

image

Using this Compose (I left out the MySQL service, to focus on the problem container):

version: '2'

services:

  phpmyadmin:
    container_name: phpmyadmin
    image: 'phpmyadmin/phpmyadmin:latest'
    restart: unless-stopped
    environment:
      - TZ='America/Chicago'
      - "PMA_HOST=mysql"
      - "MYSQL_ROOT_PASSWORD=RlL1#2Z%^W"
      - "PMA_ABSOLUTE_URI=https://my.domain.rocks:3112/phpmyadmin/"
    volumes:
      - "/f/Sites/Web/.config/PHPMyAdmin/Files:/Files:rw,shared"    
      - "/f/Sites/Web/.config/PHPMyAdmin/config.user.inc.php:/etc/phpmyadmin/config.user.inc.php"
      - "/f/Sites/Web/.config/PHPMyAdmin/apache2.conf:/etc/apache2/apache2.conf"
      - "/f/Sites/Web/.config/PHPMyAdmin/Themes/:/var/www/html/themes/"
    depends_on:
      - mysql
    networks:
      - odb

networks:
  odb:
    external:
      name: my-db

Gives me this error (which is the same is the previous Docker Edge version):

{"message":"Status: failed to register layer: error creating overlay mount to /var/lib/docker/overlay2/a54d40ac5df89eb2c2c9737db85772fe5782ea47c693ceba2eb1889ec2d7f25e/merged: too many levels of symbolic links, Code: 1","details":"Status: failed to register layer: error creating overlay mount to /var/lib/docker/overlay2/a54d40ac5df89eb2c2c9737db85772fe5782ea47c693ceba2eb1889ec2d7f25e/merged: too many levels of symbolic links, Code: 1"}

For what it might also be worth, since I don't know if the 2 are related in any way, but I'm just throwing it out there, I also get this error when trying to create an Adminer container, and mount a local CSS file to the container:

{"message":"Error response from daemon: error while creating mount source path '/c/Servers/Docker/Containers/Adminer/pepa-linha-dark/adminer.css': mkdir /c/Servers/Docker/Containers/Adminer/pepa-linha-dark/adminer.css: file exists","details":"Error response from daemon: error while creating mount source path '/c/Servers/Docker/Containers/Adminer/pepa-linha-dark/adminer.css': mkdir /c/Servers/Docker/Containers/Adminer/pepa-linha-dark/adminer.css: file exists"}

I'm running on Win 10 x64 (Version 10.0.18363.657])

@stephen-turner
Copy link
Contributor

@STaRDoGG That doesn't appear to be the same bug. Could you create a new ticket please, and include diagnostics.

@STaRDoGG
Copy link

@STaRDoGG That doesn't appear to be the same bug. Could you create a new ticket please, and include diagnostics.

Hi @stephen-turner, apologies for posting it here; I figured I would since it was addressed above, plus toward the end of the post it shows that I still get the bug in this thread with the Adminer container. I'll post below, the compose I use for that container as it may still help.

I created the new issue as requested, including a diagnostics code. Thanks for looking into it, it's been driving me insane over roughly the last month that I can't figure out what the problem is, lol

@STaRDoGG
Copy link

Here's my Compose file that still gives the bug in the topic:

version: '2'

services:

  adminer:
    # https://hub.docker.com/_/adminer
    container_name: adminer
    image: adminer:latest
    restart: unless-stopped
    environment:
      - ADMINER_DEFAULT_SERVER=mysql
      - 'ADMINER_PLUGINS=dump-xml enum-types tinymce dump-zip file-upload slugify dump-alter edit-calendar foreign-system sql-log version-noverify dump-bz2 edit-foreign frames struct-comments wymeditor dump-date edit-textarea json-column table-indexes-structure dump-json email-table table-structure dump-php enum-option tables-filter'
    volumes:
      - '/c/Servers/Docker/Containers/Adminer/pepa-linha-dark/adminer.css:/var/www/html/adminer.css' # Error
    networks:
      - odb

networks:
  odb:
    external:
      name: my-db

Error:

{"message":"Error response from daemon: error while creating mount source path '/c/Servers/Docker/Containers/Adminer/pepa-linha-dark/adminer.css': mkdir /c/Servers/Docker/Containers/Adminer/pepa-linha-dark/adminer.css: file exists","details":"Error response from daemon: error while creating mount source path '/c/Servers/Docker/Containers/Adminer/pepa-linha-dark/adminer.css': mkdir /c/Servers/Docker/Containers/Adminer/pepa-linha-dark/adminer.css: file exists"}

Docker version:

image

@mat007
Copy link
Member

mat007 commented Feb 14, 2020

@STaRDoGG just a thought: have you tried with C:/Servers instead of /c/Servers in the compose file?
/c is the mingw prefix, right?

@STaRDoGG
Copy link

@mat007 I hadn't tried that variation yet, but it does give me a different, strange error now, when I use this instead (all else is the same as above):

Code:

    volumes:
      - 'C:/Servers/Docker/Containers/Adminer/pepa-linha-dark/adminer.css:/var/www/html/adminer.css'

Error:

Error response from daemon: invalid mode: /var/www/html/adminer.css

Previously, I've always used the format I showed originally and it always worked (and still seems to for most of my other containers). But, it's interesting that the new error calls it a "mode"?

@mat007
Copy link
Member

mat007 commented Feb 14, 2020

But, it's interesting that the new error calls it a "mode"?

It’s probably because of the : right after C, forget my suggestion.
I must admit I don’t how to address full paths in Compose on Windows, if /c/Servers was working before then I guess it’s the way to go 🤷‍♂

@STaRDoGG
Copy link

@mat007 np, ty for the attempt. =)

@docker-robott
Copy link
Collaborator

Closed issues are locked after 30 days of inactivity.
This helps our team focus on active issues.

If you have found a problem that seems similar to this, please open a new issue.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows.
/lifecycle locked

@docker docker locked and limited conversation to collaborators Jul 13, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests