Simple script to launch and stop docker containers and register them with etcd
Deploy docker containers and have them automatically register themselves in etcd.
Together with confd and a load balancer (e.g. HAProxy) this allows automated add/remove of containers to the load balancer.
If you're hunting for some keywords: elastic scaling, rolling updates, zero downtime deployments
- An etcd instance somewhere on the network
- A confd template for your gateway. Confd is neReded to reload the gateway server with new configuration when instances join or leave
- A docker host on the network that will run your application images
Say we have three servers:
name | private IP | packages |
---|---|---|
gw1 | 10.10.10.1 | haproxy, confd |
docker1 | 10.10.10.10 | docker, forrest |
etcd1 | 10.10.10.20 | etcd |
etcd is a key/value store, basically. It allows easy storage of configuration values in a central place, using a simple REST-API.
For example we could store our servers with a name and an IP like this:
$ curl -XPUT http://10.10.10.20:4001/v2/keys/app/servers/web1 -d value="10.10.10.10:45679"
and then get the value with
$ curl http://10.10.10.20:4001/v2/keys/app/servers/web1
We will use this to store exactly this: Our docker container IPs plus their name, so we can find them automatically as they become available to add them to our reverse proxy as well as remove them from there when they go down.
Let's start etcd:
nohup etcd/bin/etcd -bind-addr=10.10.10.20 > /var/log/etcd.log &
confd is insanely useful, yet surprisingly simple to use. Basically all it does is watch a config server (etcd or Consul) for changes.
If it detects a change, it uses a template to create a new version of a config file with these new values and run a command to reload the configuration (in our case, for instance, service haproxy reload
).
This way it allows "live" configuration changes.
For that to work we need two files: The template and the configuration. Here is the configuration:
[template]
src = "haproxy.cfg.tmpl"
dest = "/etc/haproxy/haproxy.cfg"
keys = [
"/app/servers",
]
reload_cmd = "/usr/sbin/service haproxy reload"
This defines where the file will be written to ("/etc/haproxy/haproxy.cfg"), what the template will be ("haproxy.cfg.tmpl"), what's the root key on where to find the configuration values and the command to run if the config has been rewritten.
A template for HAProxy could look like this:
defaults
log global
mode http
listen frontend 0.0.0.0:8080
mode http
stats enable
stats uri /haproxy?stats
balance roundrobin
option httpclose
option forwardfor
{{range $server := .app_servers}}
server {{Base $server.Key}} {{$server.Value}} check
{{end}}
That's a very simple HAProxy config that listens on port 8080, gathers and exposes some stats (nice for playing around with it), does a simple roundrobin load balancing.
The last three lines are interesting, as confd comes into play here...
The {{range $server := .app_servers}}
iterates over each key within etcd at the /app/servers
path and writes a config line containing server
followed by the key name itself and its value.
So if etcd has the following values:
/app/servers/web1 = "10.10.10.10:1234"
/app/servers/web2 = "10.10.10.10:1337"
/app/servers/test_web1 = "10.10.10.10:4242"
then we would get this config file lines:
server web1 10.10.10.10:1234 check
server web2 10.10.10.10:1337 check
server test_web1 10.10.10.10:4242 check
which means that HAProxy will balance requests between these 3 servers and performs a health check on each of them.
This way we can now use etcd and confd to dynamically configure our HAProxy via changes in the values stored in the etcd. Neat!
Let's start confd then:
nohup confd -verbose -interval 10 -node '10.10.10.20:4001' -confdir /etc/confd > /var/log/confd.log &
For an in-depth introduction to Docker, check out the incredibly fantastic interactive tutorial on their site.
For this scenario, I will pretend you have a working Docker host and a docker image for your application. I will call that image 'demo/myapp' for now.
The image should only expose a single port.
All forrest does is launch (or stop) a container from an image and announce the newly started container, including the Port it exposes, to the etcd.
Thus, it allows us to pick up the address:port
combination from the etcd and put it into the configuration of our HAProxy... which effectively means we're able to automatically add or remove containers from the load balancing as they go up or down.
Now this is how this would look like:
user@docker1: $ export ETCD_HOST="10.10.10.20:4001"
user@docker1: $ export ETCD_PREFIX="app/servers"
user@docker1: $ export FORREST_IP="10.10.10.10" # possibly optional
user@docker1: $ forrest launch demo/myapp
Launching demo/myapp on 10.10.10.10 ...
Announcing to 10.10.10.20:4001 ...
demo/myapp running on Port 49731 with name sharp_mccarthy
user@docker1: $
And you should now be able to reach this application through the HAProxy on your public IP or domain name.
If you want to stop a running container and remove it from etcd (and thus from the HAProxy) you do
user@docker1: $ forrest stop sharp_mccarthy
Stopping sharp_mccarthy ...
Found container 8d3299a3427b ...
Stopped.
user@docker1: $
Tada!