Live migration

[sdpeters]

Reflect Ceph's support for live migration of RBD images between clusters in the NVMe-oF gateway.

Since a gateway or gateway group in any Ceph cluster presents an NVMe-oF subsystem to hosts, migration between Ceph clusters implies migration between subsystems. Migration of NVMe namespaces between subsystems is theoretically possible, using an NVMe feature called "dispersed namespaces". Without that feature the NVMe spec forbids the appearance of any namespace in more than one subsystem (as you might do while the namespace was moving from one Ceph cluster to another, and temporarily accessible by gateways in both Ceph clusters).

One obstacle there is that the Linux NVMe-oF maintainers have some issues with the dispersed namespace feature which seem to make it unlikely that Linux will support dispersed namespaces. If there's another NVMe-oF host the gateway will be used with that does support dispersed namespaces, we look closer at dispersed namespace support in the gateway.

One alternative is to allow gateway groups to span clusters, but that might be awkward operationally.

Perhaps we could enable subsystems to be passed from one gateway group to another, enabling all the namespaces they contain to migrate. This isn't as flexible as we'd like, but can comply with the standard.

[This should probably be marked as a release 2 feature]