This project is archived. Originally it was part of the https://github.com/flokkr project, but it's no longer maintained.
Ansible and terraform scripts to install hadoop cluster based on flokkr docker images.
| Topic | Solution |
|---|---|
| Configuration management | |
| Source of config files: | Consul server (envtoconf can't be used as nomad can't use non-regular env variables) |
| Configuration preprocessing: | consync (Upload configuration to the consul server) |
| Automatic restart on config change: | Supported by the consul plugin of the baseimage. Listens on changes on the consul side |
| Provisioning and scheduling | |
| Multihost support | Yes. Nomad agents should run everywhere |
| Requirements on the hosts | Consul and Nomad agents |
| Definition of the containers per host | Nomad job descriptors |
| Scheduling (find hosts with available resource) | Yes, fine-grained constraint rules |
| Failover on host crash | N/A |
| Scale up/down: | With mondifying the Nomad job specifications |
| Multi tenancy (multiple cluster) | NO (Host network) |
| Network | |
| Network between containers | Host network |
| DNS | Yes, handled by the OS/Cloud environment (Host network) |
| Service discovery | Required for flexible scheduling (Consul based) |
| Data locality | Yes, Full |
| Availability of the ports | All of them are available (Host network) |
You need a cluster. use terraform or any other tool to start it.
- Install docker to the nodes
- Install consul to every node
- Install nomad to the nodes
- Install sigil (http://github.com/gliderlabs/sigil)
Configuration is stored in consul and downloaded by every docker image.
First you need a configuration set:
git clone https://github.com/flokkr/configuration.git
You should install the configuration upload (which uploads the configuration with additional preprcessing:
go get github.com/flokkr/consync
And now you can upload the configuration:
consync -dir ~/projects/flokkr/configuration -consul node-1 -discovery consul
Where node-1 is the hostname of a consul node
Now you can start the images with
export NOMAD_ADDR=http://node-1:4646
And finally:
./apply.sh namenode.nomad
...
Note: apply.sh is a short script to do client side templating using the sigil utility.