Metadata SPI

[chungers]

The Metadata SPI is a service provider interface that makes it possible for plugins that implement this interface to export read-only data that can be accessed using a path. For example, it is possible for compliant plugins to expose useful parameters like:

GET /region/us-west-2/vpc/vpc1234/cidr returns 172.168.0.0/16
GET /region/us-west-2/vpc/vpc1234/az/a/subnet/sn-i234/cidr returns 172.168.100.0/24
GET /flavor-swarm/join/tokens/manager returns the local swarm flavor's manager join token
GET /flavor-swarm/join/ip returns the local swarm flavor's IP of the manager to join
GET /instance-aws/metadata/iam returns the IAM role of the host that is running an AWS instance plugin
etc., ...

This makes it possible to create plugins that can use a variety of methods such as via API calls or via metadata (e.g. on AWS, via HTTP GET at link local address 169.254.169.254:80), to export useful configuration parameters on resources that are either created by infrakit or via some other means.

When a set of infrakit plugins respectively expose their relevant state / configuration data using the same metadata mechanism, the user has a unified way to access all the interesting infrastructure parameters in a global namespace. Infrakit CLI can then aggregate all this from all plugins for not only informational display but also for authoring configuration / specifications by making this metadata available to templates.

Example

1. Start up the vagrant, swarm plugins from the top level project directory:

infrakit plugin --config-url file://$PWD/examples/flavor/swarm/plugins.json  start --exec os --wait instance-vagrant flavor-swarm group-stateless

2. List the metadata that are available in this 'local' namespace:

$ infrakit metadata ls
flavor-swarm
instance-vagrant

or in the long form:

$ infrakit metadata ls -al
flavor-swarm/manager/implements/Name
flavor-swarm/manager/implements/Version
flavor-swarm/revision
flavor-swarm/version
flavor-swarm/worker/implements/Name
flavor-swarm/worker/implements/Version
instance-vagrant/implements/Name
instance-vagrant/implements/Version
instance-vagrant/revision
instance-vagrant/version

Note you can traverse these as though they are in a filesystem:

$ infrakit metadata ls flavor-swarm/manager
implements

You can get the values via cat:

$ infrakit metadata cat flavor-swarm/manager/implements
{"Name":"Flavor","Version":"0.1.0"}

Note that sometimes objects are exposed at a metadata path. You can also list the actual attributes and cat them:

$ infrakit metadata ls flavor-swarm/manager/implements
Name
Version
$ infrakit metadata cat flavor-swarm/manager/implements/Name
Flavor

This is also integrated in the template command:

$ infrakit template --url 'str://The vagrant plugin implements interface {{ metadata "instance-vagrant/implements" }}' --log 0
The vagrant plugin implements interface {"Name":"Instance","Version":"0.3.0"}

[GordonTheTurtle]

Please sign your commits following these rules:
https://github.com/docker/docker/blob/master/CONTRIBUTING.md#sign-your-work
The easiest way to do this is to amend the last commit:

$ git clone -b "metadata-spi-0" git@github.com:chungers/infrakit.git somewhere
$ cd somewhere
$ git rebase -i HEAD~842354430904
editor opens
change each 'pick' to 'edit'
save the file and quit
$ git commit --amend -s --no-edit
$ git rebase --continue # and repeat the amend for each commit
$ git push -f

Amending updates the existing PR. You DO NOT need to open a new one.

[codecov-io]

Codecov Report

Merging #396 into master will decrease coverage by -0.85%.
The diff coverage is 55.5%.

@@            Coverage Diff             @@
##           master     #396      +/-   ##
==========================================
- Coverage   62.57%   61.73%   -0.85%     
==========================================
  Files          56       62       +6     
  Lines        3137     3549     +412     
==========================================
+ Hits         1963     2191     +228     
- Misses        945     1108     +163     
- Partials      229      250      +21

Impacted Files	Coverage Δ
examples/flavor/swarm/flavor.go	47.36% <ø> (-8.31%)	❌
examples/flavor/swarm/main.go	10% <ø> (-3.16%)	❌
examples/flavor/swarm/manager.go	76.47% <100%> (ø)	✅
pkg/plugin/metadata/path.go	18.18% <18.18%> (ø)
pkg/rpc/metadata/service.go	48.91% <48.91%> (ø)
pkg/plugin/metadata/reflect.go	54.4% <54.4%> (ø)
pkg/template/funcs.go	72.91% <62.5%> (-0.14%)	❌
examples/flavor/swarm/worker.go	11.11% <66.66%> (ø)	✅
pkg/spi/metadata/path.go	78.84% <78.84%> (ø)
pkg/rpc/metadata/client.go	78.94% <78.94%> (ø)
... and 11 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 050a6a9...f2c1326. Read the comment docs.

[chungers]

Example usage -- here the swarm and vagrant plugins optionally exported some metadata (such as version and plugin interface implemented):

The ls command will list all the top level metadata 'folders' as though there's a virtual filesystem / directory. The cat command displays the value. The value is any proper JSON value, with quotes removed for strings, for readability.

~/projects/src/github.com/docker/infrakit$ infrakit metadata ls
flavor-swarm
instance-vagrant
~/projects/src/github.com/docker/infrakit$ infrakit metadata ls instance-vagrant
implements
revision
version
~/projects/src/github.com/docker/infrakit$ infrakit metadata ls instance-vagrant/implements
Name
Version
~/projects/src/github.com/docker/infrakit$ infrakit metadata cat instance-vagrant/implements
{"Name":"Instance","Version":"0.3.0"}
~/projects/src/github.com/docker/infrakit$ infrakit metadata cat instance-vagrant/implements/Name
Instance

Note that it's possible to have JSON object as value (e.g. instance-vagrant/implements) and that the fields of the struct are also represented separately: /Name and /Version.