Flex volumes which implement getvolumename API are getting unmounted during run time.

[chakri-nelluri]

Kubernetes version: 1.6
Component/Area: Storage
Affected modules: Flex volume

Kubernetes version (use kubectl version):
Client Version: version.Info{Major:"1", Minor:"6+", GitVersion:"v1.6.0-alpha.1.482+b2ea780731417d-dirty", GitCommit:"b2ea780731417da74d312bfce8ac560ae7248831", GitTreeState:"dirty", BuildDate:"2017-02-07T21:51:55Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"6+", GitVersion:"v1.6.1-1+00a638423087f6-dirty", GitCommit:"00a638423087f6e81819667f662f194a2a2d8da9", GitTreeState:"dirty", BuildDate:"2017-04-15T16:16:36Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"}

What happened:
Flex volumes which implement getvolumename API are getting unmounted during run time.

What you expected to happen:
Volumes should not be unmounted.

How to reproduce it (as minimally and precisely as possible):
Use flex volume driver which implements getvolumename.

Anything else we need to know:
Workaround:

• Do not implement getvolumename API. The API default uses PV Name as the volume name.
• Detach call uses volume name, so the plugin detach has to work with PV Name.

Temporary fix for 1.6:

• Pass PV Name to flex volume driver using extra options.

[chakri-nelluri]

Investigation:

• Reconciler is trying to reconstruct volume spec. Today we do not store any metadata, so flex volume options are missing/cannot be reconstructed in volume spec.
• Getvolumename returns wrong volume name using this reconstructed volume spec.
• Reconciler unmounts the existing volume(which has different name) as it does not match desired state.

[saad-ali]

@chakri-nelluri and I discussed this. Proposed fix is to add a metadata file so that volume reconstruction can use that and recover complete volume spec instead of just the information in the mount path.

CC @jingxu97 @kubernetes/sig-storage-bugs

[saad-ali]

@chakri-nelluri, @shilpamayanna:

Freeze date for 1.7 was June 1, so we missed the boat for the big fix (kubernetes/community#650).

In the meantime is there anything temporary that needs to go in for 1.7 to mitigate this in the master/1.7 branches?

[saad-ali]

CC @verult who's looking in to Flex volume issues.

[saad-ali]

With #47400 this is complete for 1.7. Moving to 1.8 for "real fix"

[klausenbusk]

The current behavior (1.7.0) seems a bit broken/weird.
With a volumes spec like so:

volumes:
- name: "datadir"
  flexVolume:
    driver: "sp/dobs"
    fsType: "ext4"
    options:
      volumeName: mysql-development

K8s would call:

(kube-controller-manager) attach {"kubernetes.io/fsType":"ext4","kubernetes.io/pvOrVolumeName":"datadir","kubernetes.io/readwrite":"rw","volumeName":"mysql-development"} 165.227.x.x
(kubelet) waitforattach /dev/disk/by-id/scsi-0DO_Volume_mysql-development {"kubernetes.io/fsType":"","kubernetes.io/pvOrVolumeName":"datadir","kubernetes.io/readwrite":"rw","volumeName":"mysql-development"}

But when detaching it call:

detach datadir 165.227.x.x

but the doc says:

<driver executable> detach <mount device> <node name>
<driver executable> waitforattach <mount device> <json options>

So I would expect it to call:

detach /dev/disk/by-id/scsi-0DO_Volume_mysql-development 165.227.x.x

So either the doc is wrong or something is broken.

The current behavior means that I can't use a meaningful volumeName in the podSpec, as the volumeName need to match my custom option volumeName which is passed to my flexVolume plugin.

So:

volumes:
- name: "mysql-development"
  flexVolume:
    driver: "sp/dobs"
    fsType: "ext4"
    options:
      volumeName: mysql-development

[klausenbusk]

This broken behavior is caused by: #47400 :/

[klausenbusk]

Issue opened: #51407

We need a proper fix for this, the current behavior isn't ideal.

[chakri-nelluri]

Thanks @klausenbusk This is a doc issue. I will update it.

If you have any other use case, please feel free to slack me @chakri-nelluri or reply here. I want to know more details.

[k8s-github-robot]

[MILESTONENOTIFIER] Milestone Removed

@chakri-nelluri

Important:
This issue was missing labels required for the v1.8 milestone for more than 7 days:

kind: Must specify exactly one of [kind/bug, kind/cleanup, kind/feature].
priority: Must specify exactly one of [priority/critical-urgent, priority/important-longterm, priority/important-soon].

Removing it from the milestone.

Additional instructions available here The commands available for adding these labels are documented here

[mingfang]

detach() should be called with at least the same parameters as the attach() call.

[klausenbusk]

Is their any plan on improving this? I don't like the current solution, as every volume name need to be unique. A consequence thereof, is that people can't use PV with the same name in different namespaces. (Edit: Not sure about that)

A hack around it, is naming the PV like name-<UUID>, but that is hackish.. :/

Edit: I totally misunderstood this, sorry for the noise :/

[chakri-nelluri]

This is already fixed as part of 47400