Add support for volume scopes #22077

cpuguy83 · 2016-04-15T18:00:48Z

Add support for volume scopes

This is similar to network scopes where a volume can either be local or global.
A global volume is one that exists across the entire cluster where as a local volume exists on a single engine.

Includes fixes to the plugin RPC generator to support passing structs to the plugin which was required for this PR.

cpuguy83 · 2016-04-15T18:14:34Z

Global networks are stored in the KV store, but for volumes we already rely on the volume plugins to tell us everything about the volumes so there is seemingly no requirement to do that here.

thaJeztah · 2016-04-15T20:28:29Z

docs/extend/plugins_volume.md

+{}
+```
+
+Get the list of volumes registered with the plugin.


Possibly copy-pasta; Think this should be something about "Capabilities"?

cpuguy83 · 2016-04-21T16:04:01Z

Note for reviewers, most of the change here is fixing known limitations in the plugin RPC parser/generator + lots of new tests.

cpuguy83 · 2016-04-29T13:30:34Z

docker/engine-api#216

vdemeester · 2016-04-29T16:12:59Z

@cpuguy83 Am I able to set the scope on a volume for a plugin that is not "multi-host" ? (like on a volume that is using local driver)

cpuguy83 · 2016-04-29T16:27:01Z

@vdemeester It will be local
All scopes will default to local unless the volume driver says it is global

vdemeester · 2016-04-29T16:29:17Z

@cpuguy83 ah ok good ! Thanks :)

vdemeester · 2016-05-04T13:09:49Z

@cpuguy83 docker/engine-api#216 merged, you can add a vendoring commit 😛

cpuguy83 · 2016-05-04T13:12:31Z

@vdemeester #22422

vdemeester · 2016-05-04T13:13:58Z

@cpuguy83 ah right 😝

cpuguy83 · 2016-05-04T14:14:53Z

vendor updated this one is ready.

thaJeztah · 2016-05-05T13:20:49Z

pkg/plugins/pluginrpc-gen/parser.go

+				}
+				// import paths have quotes already added in, so need to remove them for name comparison
+				name = strings.TrimPrefix(name, "\"")
+				name = strings.TrimSuffix(name, "\"")


nit: could be just

name = strings.Trim(name, `"`)

thaJeztah · 2016-05-05T13:24:54Z

Supported scopes are global and local. Any other value in Scope will be ignored and assumed to be local.

Would it be better to error out if an unknown scope is provided? I'm never really comfortable with silently ignoring invalid values. It could easily lead to backward compatibility issues if it's ignored now, but becomes a valid value in future.

cpuguy83 · 2016-05-05T14:38:46Z

@thaJeztah Instead what we can do is just not register the driver if it's providing an invalid scope.

thaJeztah · 2016-06-05T16:04:30Z

can you fix the copy/paste typo here; #22077 (comment) ?

cpuguy83 · 2016-06-05T16:08:05Z

@thaJeztah haha, (finally) fixed.

thaJeztah · 2016-06-05T16:09:40Z

LGTM

thaJeztah · 2016-06-05T19:09:00Z

@cpuguy83 looks like the tests needs to be updated to not use '

Now handles `package.Type` and `*package.Type` Fixes parsing issues with slice and map types. Signed-off-by: Brian Goff <cpuguy83@gmail.com>

This is similar to network scopes where a volume can either be `local` or `global`. A `global` volume is one that exists across the entire cluster where as a `local` volume exists on a single engine. Signed-off-by: Brian Goff <cpuguy83@gmail.com>

cpuguy83 · 2016-06-05T19:37:47Z

COBRA!

Fixed.

kolyshkin · 2016-06-08T18:38:08Z

@cpuguy83 @thaJeztah Is anyone working on adding this to https://github.com/docker/go-plugins-helpers? If no one is doing it, maybe I should give a try.

cpuguy83 · 2016-06-08T18:40:30Z

Help would be much appreciated.
Swarm could also use a treatment.

thaJeztah · 2016-06-08T18:40:50Z

not that I'm aware of (@cpuguy83?) so contributions welcome!

tiagoalves83 · 2016-07-09T21:17:05Z

At the time, is it possible to test global volumes?
I am trying to achieve NFS volumes to work in swarm mode without success.
issue #24484

cpuguy83 · 2016-07-10T14:33:31Z

@tiagoalves83 scopes are for external tooling, like github.com/docker/swarm, swarm-mode will not be making use of this.

Also note that the local volume driver can mount an nfs share docker volume create --opt type=nfs --opt device=:/export/path --opt o=addr=1.2.3.4, for example.

govint · 2016-07-13T05:06:48Z

What if a plugin is serving up a set of datastores and some of those are local datastores? How should the plugin report scope - global (there are global datastores) or local? What if the plugin has only local datastores accessible to it (no NFS for example).

The user has no control where the volumes are placed anyway. Any plan to enumerate datastores for a plugin and allow users to specify placement.

@cpuguy83

cpuguy83 · 2016-07-15T14:09:16Z

@govint Always send local scope? It's probably not the best idea to have a plugin serving two roles.

What we'll probably end up doing for swarm mode is something like having the orchestrator pick a node then ask the plugin "does this node have access to the requested data?"

tiagoalves83 · 2016-07-21T06:15:04Z

@cpuguy83 nfs local volumes are not working for swarm workers, just the manager.

My Enviroment:
MacOSX running docker-machine 0.7.0, nfs-server (192.168.99.1), docker cli 1.12.0-rc4
I have 2 nodes: swmaster-1 (192.168.99.100) and swnode-1 (192.168.99.101) boot2docker 1.12.0-rc4.
From within swarm nodes (docker-machine ssh ...) I can successfully mount nfs:

docker-machine ssh swmaster-1
sudo mkdir /test-nfs
sudo mount -t nfs 192.168.99.1:/Volumes/HDD/tmp /test-nfs
sudo ls /test-nfs #OK listing all files
exit

docker-machine ssh swnode-1
sudo mkdir /test-nfs
sudo mount -t nfs 192.168.99.1:/Volumes/HDD/tmp /test-nfs
sudo ls /test-nfs #OK listing all files
exit

Now, using docker (swarm-mode)

eval $(docker-machine env swmaster-1)
docker volume create --driver local --opt type=nfs --opt o=addr=192.168.99.1,rw --opt device=:/Volumes/HDD/tmp --name nfs-vol-tmp
docker service create --endpoint-mode dnsrr --mount type=volume,source=nfs-vol-tmp,target=/mount --name test-nfs-1 alpine /bin/sh -c "while true; do echo 'OK'; sleep 3; done"
docker service create --endpoint-mode dnsrr --mount type=volume,source=nfs-vol-tmp,target=/mount --name test-nfs-2 alpine /bin/sh -c "while true; do echo 'OK'; sleep 3; done"
docker service create --endpoint-mode dnsrr --mount type=volume,source=nfs-vol-tmp,target=/mount --name test-nfs-3 alpine /bin/sh -c "while true; do echo 'OK'; sleep 3; done"
docker service create --endpoint-mode dnsrr --mount type=volume,source=nfs-vol-tmp,target=/mount --name test-nfs-4 alpine /bin/sh -c "while true; do echo 'OK'; sleep 3; done"
docker service create --endpoint-mode dnsrr --mount type=volume,source=nfs-vol-tmp,target=/mount --name test-nfs-5 alpine /bin/sh -c "while true; do echo 'OK'; sleep 3; done"
docker service create --endpoint-mode dnsrr --mount type=volume,source=nfs-vol-tmp,target=/mount --name test-nfs-6 alpine /bin/sh -c "while true; do echo 'OK'; sleep 3; done"
docker service create --endpoint-mode dnsrr --mount type=volume,source=nfs-vol-tmp,target=/mount --name test-nfs-7 alpine /bin/sh -c "while true; do echo 'OK'; sleep 3; done"

Than, inside swmaster-1, volume is being mounted and it is working

docker@swmaster-1:~$ docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS               NAMES
f7ff90bb6324        alpine:latest       "/bin/sh -c 'while tr"   3 minutes ago       Up 3 minutes                            test-nfs-7.1.9e2q6954xjjbqdwpz14k5dji5
be174c6dcf31        alpine:latest       "/bin/sh -c 'while tr"   3 minutes ago       Up 3 minutes                            test-nfs-6.1.928csw1stg2gpx8d7r3dieiqr
67b01f475bfb        alpine:latest       "/bin/sh -c 'while tr"   4 minutes ago       Up 4 minutes                            test-nfs-3.1.6dv7j6mxmo4y6pkxim6yn2705
0046534afcdf        alpine:latest       "/bin/sh -c 'while tr"   4 minutes ago       Up 4 minutes                            test-nfs-2.1.8loforolv5b34rlztfxwhwjvu

docker@swmaster-1:~$ docker exec f7ff90bb6324  /bin/sh -c "ls /mount" # OK listing files
docker@swmaster-1:~$ docker exec be174c6dcf31  /bin/sh -c "ls /mount" # OK listing files
docker@swmaster-1:~$ docker exec 67b01f475bfb  /bin/sh -c "ls /mount" # OK listing files
docker@swmaster-1:~$ docker exec 0046534afcdf  /bin/sh -c "ls /mount" # OK listing files

Now, for containers created at the swnode-1, volume is not working

docker@swnode-1:~$ docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS               NAMES
3d38d66326e2        alpine:latest       "/bin/sh -c 'while tr"   16 seconds ago      Up 15 seconds                           test-nfs-5.1.ap0nhivlvifunar8z9n9333hx
a17e411d9690        alpine:latest       "/bin/sh -c 'while tr"   45 seconds ago      Up 43 seconds                           test-nfs-4.1.1lud5nyumwcfxn6yrurctkj1i
ee13c9b9ed05        alpine:latest       "/bin/sh -c 'while tr"   5 minutes ago       Up 5 minutes                            test-nfs-1.1.cvc75ku4zyqg5mfz2dz94v718

docker@swnode-1:~$ docker exec 3d38d66326e2  /bin/sh -c "ls /mount" # NOT OK empty folder
docker@swnode-1:~$ docker exec a17e411d9690  /bin/sh -c "ls /mount" # NOT OK empty folder
docker@swnode-1:~$ docker exec ee13c9b9ed05  /bin/sh -c "ls /mount" # NOT OK empty folder

docker volume ls and docker volume inspect inside "swnode-1" show the same info that "swmaster-1" for "nfs-vol-tmp" volume, except for the Label field that is null in swnode-1 and "{}" for swmaster-1

docker@swmaster-1:~$ docker volume inspect nfs-vol-tmp
[
    {
        "Name": "nfs-vol-tmp",
        "Driver": "local",
        "Mountpoint": "/mnt/sda1/var/lib/docker/volumes/nfs-vol-tmp/_data",
        "Labels": {},
        "Scope": "local"
    }
]

docker@swnode-1:~$ docker volume inspect nfs-vol-tmp
[
    {
        "Name": "nfs-vol-tmp",
        "Driver": "local",
        "Mountpoint": "/mnt/sda1/var/lib/docker/volumes/nfs-vol-tmp/_data",
        "Labels": null,
        "Scope": "local"
    }
]

Something that I noticed is that when I remove the volume from the swarm docker volume rm nfs-vol-tmp the volume still being listed docker volume ls to the worker node "swnode-1". Is that normal ?

tiagoalves83 · 2016-07-29T03:11:45Z

Same issue in 1.12.0 version #25202

cpuguy83 · 2016-07-29T03:27:25Z

@tiagoalves83 nothing to do with this pr.
If you are having an issue please open an issue with all the details so we can take a look.
Thanks.

Since Docker 1.12, volumes can report their scope as either "global" or "local". Global scope means volumes are available on all nodes in a Docker Swarm cluster (so Swarm should act accordingly, for example it should not try to remove a global volume on each node). By default (or when started with -scope auto), scope is autodetected, meaning if driver's home is on Virtuozzo Storage, global scope is set and reported, otherwise it's assumed as local. This can be overriden from the command line ("-scope local" or "-scope global"). For more details, see moby/moby#22077 Signed-off-by: Kir Kolyshkin <kir@openvz.org>

GordonTheTurtle added the status/0-triage label Apr 15, 2016

cpuguy83 changed the title ~~Remote volplugin caps~~ Add support for volume scopes Apr 15, 2016

cpuguy83 added status/1-design-review and removed status/0-triage labels Apr 15, 2016

cpuguy83 force-pushed the remote_volplugin_caps branch from 4f8374b to 9316d8f Compare April 15, 2016 18:09

cpuguy83 force-pushed the remote_volplugin_caps branch from 9316d8f to 50b3949 Compare April 15, 2016 18:29

thaJeztah reviewed Apr 15, 2016
View reviewed changes

cpuguy83 mentioned this pull request Apr 21, 2016

Add label filter support for volume #21567

Closed

cpuguy83 mentioned this pull request Apr 27, 2016

Container start fails, but volume remains mounted #21921

Closed

thaJeztah added status/2-code-review and removed status/1-design-review labels Apr 28, 2016

cpuguy83 mentioned this pull request Apr 29, 2016

Add Scope field to Volume docker/engine-api#216

Merged

cpuguy83 force-pushed the remote_volplugin_caps branch from 50b3949 to a985412 Compare May 4, 2016 14:13

cpuguy83 force-pushed the remote_volplugin_caps branch from a985412 to e73bd72 Compare May 4, 2016 14:17

thaJeztah added this to the 1.12.0 milestone May 4, 2016

thaJeztah reviewed May 5, 2016
View reviewed changes

cpuguy83 force-pushed the remote_volplugin_caps branch from e73bd72 to c7a6b10 Compare May 5, 2016 15:03

thaJeztah added status/3-docs-review and removed status/2-code-review labels Jun 5, 2016

thaJeztah added impact/api impact/changelog labels Jun 5, 2016

cpuguy83 force-pushed the remote_volplugin_caps branch from c2f06bc to d68468e Compare June 5, 2016 16:07

cpuguy83 force-pushed the remote_volplugin_caps branch from d68468e to 385a9d8 Compare June 5, 2016 16:08

cpuguy83 added 2 commits June 5, 2016 15:37

Enhance pluginrpc-gen parser

79ff6ea

Now handles `package.Type` and `*package.Type` Fixes parsing issues with slice and map types. Signed-off-by: Brian Goff <cpuguy83@gmail.com>

Add support for volume scopes

2f40b1b

This is similar to network scopes where a volume can either be `local` or `global`. A `global` volume is one that exists across the entire cluster where as a `local` volume exists on a single engine. Signed-off-by: Brian Goff <cpuguy83@gmail.com>

cpuguy83 force-pushed the remote_volplugin_caps branch from 385a9d8 to 2f40b1b Compare June 5, 2016 19:37

thaJeztah added status/4-merge and removed status/3-docs-review labels Jun 5, 2016

vdemeester merged commit 28dde09 into moby:master Jun 6, 2016

cpuguy83 deleted the remote_volplugin_caps branch June 8, 2016 18:40

porcupie mentioned this pull request Aug 9, 2016

Can not compile plugin with docker 1.10+ - missing Capabilities Volume API method yp-engineering/rbd-docker-plugin#41

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for volume scopes #22077

Add support for volume scopes #22077

cpuguy83 commented Apr 15, 2016

cpuguy83 commented Apr 15, 2016

thaJeztah Apr 15, 2016

cpuguy83 commented Apr 21, 2016

cpuguy83 commented Apr 29, 2016

vdemeester commented Apr 29, 2016

cpuguy83 commented Apr 29, 2016

vdemeester commented Apr 29, 2016

vdemeester commented May 4, 2016

cpuguy83 commented May 4, 2016

vdemeester commented May 4, 2016

cpuguy83 commented May 4, 2016

thaJeztah May 5, 2016

thaJeztah commented May 5, 2016

cpuguy83 commented May 5, 2016

thaJeztah commented Jun 5, 2016

cpuguy83 commented Jun 5, 2016

thaJeztah commented Jun 5, 2016

thaJeztah commented Jun 5, 2016

cpuguy83 commented Jun 5, 2016

kolyshkin commented Jun 8, 2016

cpuguy83 commented Jun 8, 2016

thaJeztah commented Jun 8, 2016

tiagoalves83 commented Jul 9, 2016

cpuguy83 commented Jul 10, 2016

govint commented Jul 13, 2016 •

edited

Loading

cpuguy83 commented Jul 15, 2016

tiagoalves83 commented Jul 21, 2016

tiagoalves83 commented Jul 29, 2016

cpuguy83 commented Jul 29, 2016

Add support for volume scopes #22077

Add support for volume scopes #22077

Conversation

cpuguy83 commented Apr 15, 2016

cpuguy83 commented Apr 15, 2016

thaJeztah Apr 15, 2016

Choose a reason for hiding this comment

cpuguy83 commented Apr 21, 2016

cpuguy83 commented Apr 29, 2016

vdemeester commented Apr 29, 2016

cpuguy83 commented Apr 29, 2016

vdemeester commented Apr 29, 2016

vdemeester commented May 4, 2016

cpuguy83 commented May 4, 2016

vdemeester commented May 4, 2016

cpuguy83 commented May 4, 2016

thaJeztah May 5, 2016

Choose a reason for hiding this comment

thaJeztah commented May 5, 2016

cpuguy83 commented May 5, 2016

thaJeztah commented Jun 5, 2016

cpuguy83 commented Jun 5, 2016

thaJeztah commented Jun 5, 2016

thaJeztah commented Jun 5, 2016

cpuguy83 commented Jun 5, 2016

kolyshkin commented Jun 8, 2016

cpuguy83 commented Jun 8, 2016

thaJeztah commented Jun 8, 2016

tiagoalves83 commented Jul 9, 2016

cpuguy83 commented Jul 10, 2016

govint commented Jul 13, 2016 • edited Loading

cpuguy83 commented Jul 15, 2016

tiagoalves83 commented Jul 21, 2016

tiagoalves83 commented Jul 29, 2016

cpuguy83 commented Jul 29, 2016

govint commented Jul 13, 2016 •

edited

Loading