Add Helm chart for Airflow (fix #7)

[gsemet]

Here is a big Pull Request adding an Helm Chart for Airflow (see #7). I propose the following:

• we do the review here (I understand it can take a few time)
• I have changed some stuff in the docker files, it will requires a new image (for the moment I use mine: for testing and it works great so far docker pull stibbons31/kube-airflow:airflow-1.8.0.0-kube-1.6)
• once we agree, I can propose it to the official Helm repository if you agree
• I still have to add more docs, rework maybe the configuration of the templating, add a few more options (ingress or not, etc).

For information, I was greatly inspired by the zero-to-jupyterhub helm repository.

Usage:

helm upgrade --install --debug -f myvalues.yaml airflow /path/to/kube-airflow/airflow

Features

• add EditorConfig files. Because they simplify everything
• airflow.all.yaml is kept but Helm template directory is created in ./airflow
• Propagate server config from env variables
• statup script can receive some config by ENV variable (name of rabbitmq/postgres server), at least everything derives from the same source
• username and password for db can inherit from env
• optional airflow.cfg override by configmap support to set default environment variables for web, scheduler and worker (but still requires set of some env vars)
• allow ingress in endpoint such as /airflow and /flower (to do so, a MR#2723 is required to be merged first in Airflow). + Read the warning on the README about the way url prefix is to be integrated in your ingress.
• update to python 3 and debian stretch in container
• optional DAG synchronization using "git sync" to mirror a git repo for DAGs + warning about uncontrolled DAG updates in README
• load example true/false and flexible scheduler run numbers
• configurable PVC support
• use of statefulset for workers to logs can be accessed by the web server

To test this pull request:

• use image stibbons31/kube-airflow:helm_chart_dockerfile instead of mumoshu/kube-airflow:1.8.0.0-1.6.1
• mumoshu/kube-airflow:1.8.0.0-1.6.1 needs to be updated with the new Dockerfile.template proposed in this PR (using more recent version of Debian and python 3, new configuration files, etc)...

Please note this PR depends on a patch on airflow (#2723) for supporting url_prefix for the webservice and flower, so you cannot use a released version of Airflow so far. I maintain a fork with my patch until my Pull Request get integrated.

[lsowen]

@stibbons a helm chart would be awesome. However, do you think it might be a better fit for https://github.com/kubernetes/charts? That would allow better discoverability, review, and long term maintenance.

[gsemet]

Yes, I plan to land it to kube-airflow for testing, and if it works, merge it to kubernetes/charts

[chhetripradeep]

Typo: hostname instead of basename

[gsemet]

I wanted to emphasis it is not a full hostname. the final hostname will be prefix with airflow.prefix

[chhetripradeep]

In that case shouldn't the corresponding variable in values.yaml be using basename instead of hostname

[gsemet]

hum this part works pretty well, I don't see your point. values.yaml allows user to specify the basehostname, and here it prefixed with the generic prefix (I use it so it can be possible to start several airflow instance in the same namespace)

[chhetripradeep]

Typo: hostname instead of basename

[chhetripradeep]

@stibbons I tried your helm chart with your docker image - few things didn't work for me.

• scheduler & worker container - i got the error saying: entrypoint.sh: python3 not found
• flower container failed with error - airflow: error: argument subcommand: invalid choice: '/usr/local/airflow/flowerconfig.py' (choose from 'resetdb', 'render', 'variables', 'connections', 'pause', 'task_failed_deps', 'version', 'trigger_dag', 'initdb', 'test', 'unpause', 'dag_state', 'run', 'list_tasks', 'backfill', 'list_dags', 'kerberos', 'worker', 'webserver', 'flower', 'scheduler', 'task_state', 'pool', 'serve_logs', 'clear', 'upgradedb')

[gsemet]

You need to use stibbons31/kube-airflow:airflow-HEAD-patched-kube-1.6 instead of mumoshu/kube-airflow:1.8.0.0-1.6.1 until mumoshu/kube-airflow:1.8.0.0-1.6.1 is updated with the new Dockerfile.template (using more recent version of Debian and python 3)...

PS: I have fixed an issue today can you try again by retrieving the latest version of this PR

[gsemet]

Also, I will probably propose a version that store the complete airflow.cfg in a configmap, it should be much simpler

[chhetripradeep]

I mean shouldn't this key be basename instead of hostname.

[gsemet]

ho, yes indeed

[chhetripradeep]

@stibbons I used your image stibbons31/kube-airflow:airflow-HEAD-patched-kube-1.6 and now i am getting this exception:

Traceback (most recent call last):
  File "/usr/local/bin/airflow", line 17, in <module>
    from airflow import configuration
  File "/usr/local/lib/python3.5/dist-packages/airflow/__init__.py", line 31, in <module>
    from airflow.models import DAG
  File "/usr/local/lib/python3.5/dist-packages/airflow/models.py", line 138, in <module>
    class DagBag(BaseDagBag, LoggingMixin):
  File "/usr/local/lib/python3.5/dist-packages/airflow/models.py", line 164, in DagBag
    include_examples=configuration.getboolean('core', 'LOAD_EXAMPLES')):
  File "/usr/local/lib/python3.5/dist-packages/airflow/configuration.py", line 804, in getboolean
    return conf.getboolean(section, key)
  File "/usr/local/lib/python3.5/dist-packages/airflow/configuration.py", line 626, in getboolean
    'boolean (received "{}").'.format(section, key, val))
airflow.exceptions.AirflowConfigException: The value for configuration option "core:LOAD_EXAMPLES" is not a boolean (received "").

[gsemet]

@chhetripradeep should be fixed on latest rev.

[chhetripradeep]

@stibbons sorry the above exception is my bad. i didnt pull the latest helm chart branch.

[gsemet]

I just fixed it now :)

[chhetripradeep]

@stibbons Ack that the chart now works like a charm 🎉 🎉

[chhetripradeep]

I am also new to helm and k8s ecosystem but few feedbacks i would suggest -

• Using PVC for databases.
• Storing complete airflow.cfg in a configmap.
• Adding a flag to enable/disable ingress, Based on that we can disable/enable nodePort for service.

Overall, chart looks great.

[gsemet]

There will be probably 2 helm charts:

• 1 for kubernetes celery (this one) without my url_prefix, no configmap, maybe git-sync and python3
• 1 for kubernetes executor (the patch is still in the queue in airflow), probably in airflow 2, with configmap, pvc, git-sync as well, ...

I'll add quickly the ingress switch, i'll try pvc if i have the time, and i also wanted to switch to config map but with this image and the entrypoint.sh it will be tricky to do

[grantnicholas]

Just a note. Be careful with git-sync and airflow. If the scheduler reloads a dag in the middle of a dagrun then the dagrun will run actually start using the new version of the dag in the middle of execution. This is a known issue with airflow and it means it's unsafe in general to use a git-sync like solution with airflow without:

1. using explicit locking, ie never pull down a new dag if a dagrun is in progress
2. make dags immutable, never modify your dag always make a new one

[gsemet]

Indeed, good to know. It is just an option, and probably needs more documentation. By the way, do you know if there are official "recommendation" on airflow website, about Dag update/deployment? I don't see such information in the doc

[mumoshu]

@gsemet Hey! I finally managed to catch all the things up made until now.

Thanks for the all the your efforts and the fork, and the PR to official Kubernetes charts repository 👍

Everyone, see his awesome work at helm/charts#3959.

I'll keep this repository for mostly for a reference for a while.
But I believe - the general direction should be treat @gsemet's work as the primary, and contribute any missing parts(if any) to his k8s chart and eventually obsolete this as superseded by it.