Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add Helm chart for Airflow (fix #7) #16

Merged
merged 51 commits into from
Apr 23, 2018
Merged

Add Helm chart for Airflow (fix #7) #16

merged 51 commits into from
Apr 23, 2018

Commits on Oct 28, 2017

  1. Add Helm chart for Airflow 

    - add EditorConfig files
- airflow.all.yaml is splited in different chart templates
- Propagate server config from env variable if defined
- statup script can receive some config by ENV variable
  (name of rabbitmq/postgres server), at least everything
  derives from the same source
- username and password can inherit from env
- fix probable buggy fernet key definition, missing enum34
  with python 2 installation
- change format of template in airflow.cfg (airflow.cfg.in)
- use configmap to set default environment variables
- parameterize number of workers
- support url prefix for flower and web ui
  allow ingress in endpoint such as /airflow and /flower
- use python 3 in docker
- optional git sync to mirror a git repo for DAGs
  + warning about uncontrolled DAG update
- debian stretch (python >3.4)
- load example true/false and flexible scheduler run nums
    @gsemet
    gsemet committed Oct 28, 2017
    Configuration menu
    Copy the full SHA
    f84d458 View commit details
    Browse the repository at this point in the history

  2. requirements*.txt support + docs 

    Signed-off-by: Gaetan Semet <gaetan@xeberon.net>
    @gsemet
    gsemet committed Oct 28, 2017
    Configuration menu
    Copy the full SHA
    0c38fd8 View commit details
    Browse the repository at this point in the history

Commits on Nov 6, 2017

  1. This commit adds and icon to the Chart. 

    This is not required but still shows up in helm lint tests
    @gsemet
    Sedward authored and gsemet committed Nov 6, 2017
    Configuration menu
    Copy the full SHA
    445999d View commit details
    Browse the repository at this point in the history

  2. This commit adds a helpers to access name variables in the templates

    @gsemet
    Sedward authored and gsemet committed Nov 6, 2017
    Configuration menu
    Copy the full SHA
    5fadf86 View commit details
    Browse the repository at this point in the history

  3. Adding of optional pvc WIP 

    This commit adds an optional PVC for the dags directory.
    @gsemet
    Sedward authored and gsemet committed Nov 6, 2017
    Configuration menu
    Copy the full SHA
    ca66688 View commit details
    Browse the repository at this point in the history

  4. Refactor deployments and add a dags path variable 

    This commit breaks the deployments out into separate files.
The purpose of this is for troubleshooting when making
changes to the deployment and fixing errors when finding
which deployment and line number is causing a problem.

This change also make the dag_path which is used for
volumeMounts within deployments.
    @gsemet
    gsemet committed Nov 6, 2017
    Configuration menu
    Copy the full SHA
    2d579e5 View commit details
    Browse the repository at this point in the history

  5. Change the PVC access mode 

    Ths changes the acces mode to ReadWriteMany because it will be accessed
more than one pod

# Conflicts:
#	airflow/values.yaml
    @gsemet
    Sedward authored and gsemet committed Nov 6, 2017
    Configuration menu
    Copy the full SHA
    2b4401b View commit details
    Browse the repository at this point in the history

  6. This commit adds a more up to date version 

    of the pvc and values to go with the persistence section

# Conflicts:
#	airflow/templates/pvc.yaml
#	airflow/values.yaml
    @gsemet
    Sedward authored and gsemet committed Nov 6, 2017
    Configuration menu
    Copy the full SHA
    afa9a1a View commit details
    Browse the repository at this point in the history

Commits on Nov 7, 2017

  1. Fixes after PVC merge

    @gsemet
    gsemet committed Nov 7, 2017
    Configuration menu
    Copy the full SHA
    2a861a0 View commit details
    Browse the repository at this point in the history

  2. Protect all names 

    Signed-off-by: Gaetan Semet <gaetan@xeberon.net>
    @gsemet
    gsemet committed Nov 7, 2017
    Configuration menu
    Copy the full SHA
    b2cf5bb View commit details
    Browse the repository at this point in the history

  3. fix some issues on the charts

    @gsemet
    gsemet committed Nov 7, 2017
    Configuration menu
    Copy the full SHA
    f6d35b1 View commit details
    Browse the repository at this point in the history

  4. fix in airflow.cfg 

    and remove '.in'. It will allow to override the
configuration file by configmap more easily
    @gsemet
    gsemet committed Nov 7, 2017
    Configuration menu
    Copy the full SHA
    0930caf View commit details
    Browse the repository at this point in the history

  5. airflow.cfg configmap

    @gsemet
    gsemet committed Nov 7, 2017
    Configuration menu
    Copy the full SHA
    8cf8dd4 View commit details
    Browse the repository at this point in the history

  6. beware of the sed when replacing url_prefix 

    do not use sed s/../../
but sed s##..#...
    @gsemet
    gsemet committed Nov 7, 2017
    Configuration menu
    Copy the full SHA
    6a1e26b View commit details
    Browse the repository at this point in the history

  7. update README

    @gsemet
    gsemet committed Nov 7, 2017
    Configuration menu
    Copy the full SHA
    168c624 View commit details
    Browse the repository at this point in the history

  8. Fix credentials applying + update config.yaml example

    @gsemet
    gsemet committed Nov 7, 2017
    Configuration menu
    Copy the full SHA
    91bc107 View commit details
    Browse the repository at this point in the history

  9. Minor glitch fixes

    @gsemet
    gsemet committed Nov 7, 2017
    Configuration menu
    Copy the full SHA
    f064316 View commit details
    Browse the repository at this point in the history

  10. make helm-check

    @gsemet
    gsemet committed Nov 7, 2017
    Configuration menu
    Copy the full SHA
    229a05e View commit details
    Browse the repository at this point in the history

Commits on Nov 26, 2017

  1. Rework 

    - statefulset worker (allow access to port and so logs)
- different ingress annotations between flower and webserver
- Add /usr/bin/python symblink in Dockerfile
- ensure file overwrite with git sync
- using rolling update statefulset
- allow to pickle DAG from scheduler to worker
  so git sync on worker MIGHT be useless)
- update README
- set scheduler num runs to 1
  will restart scheduler at each execution

Signed-off-by: Gaetan Semet <gaetan@xeberon.net>
    @gsemet
    Gaetan Semet authored and gsemet committed Nov 26, 2017
    Configuration menu
    Copy the full SHA
    561a661 View commit details
    Browse the repository at this point in the history

  2. update about values.yaml

    @gsemet
    gsemet committed Nov 26, 2017
    Configuration menu
    Copy the full SHA
    e492d8c View commit details
    Browse the repository at this point in the history

  3. Values configuration

    @gsemet
    gsemet committed Nov 26, 2017
    Configuration menu
    Copy the full SHA
    2ca5bf5 View commit details
    Browse the repository at this point in the history

Commits on Nov 27, 2017

  1. statefulset: burst mode

    @gsemet
    gsemet committed Nov 27, 2017
    Configuration menu
    Copy the full SHA
    ea51bb6 View commit details
    Browse the repository at this point in the history

  2. Cleaner example

    @gsemet
    gsemet committed Nov 27, 2017
    Configuration menu
    Copy the full SHA
    8bb9e41 View commit details
    Browse the repository at this point in the history

Commits on Nov 28, 2017

  1. Rolling update

    @gsemet
    gsemet committed Nov 28, 2017
    Configuration menu
    Copy the full SHA
    c73d0bc View commit details
    Browse the repository at this point in the history

  2. Add web service liveness probe

    @gsemet
    gsemet committed Nov 28, 2017
    Configuration menu
    Copy the full SHA
    1206199 View commit details
    Browse the repository at this point in the history

  3. no wait

    @gsemet
    Gaetan Semet authored and gsemet committed Nov 28, 2017
    Configuration menu
    Copy the full SHA
    8884c02 View commit details
    Browse the repository at this point in the history

Commits on Feb 21, 2018

  1. Removed embedded postgresql pieces.

    @jpds
    jpds committed Feb 21, 2018
    Configuration menu
    Copy the full SHA
    948ff3e View commit details
    Browse the repository at this point in the history

  2. Added dependency on postgresql chart.

    @jpds
    jpds committed Feb 21, 2018
    Configuration menu
    Copy the full SHA
    9845a39 View commit details
    Browse the repository at this point in the history

  3. Leverage postgres dependency in configmaps.

    @jpds
    jpds committed Feb 21, 2018
    Configuration menu
    Copy the full SHA
    27178fe View commit details
    Browse the repository at this point in the history

  4. airflow/values.yaml: Removed concourse reference.

    @jpds
    jpds committed Feb 21, 2018
    Configuration menu
    Copy the full SHA
    31bcf03 View commit details
    Browse the repository at this point in the history

  5. Merge pull request #13 from jpds/postgresql-dependency 

    Added a dependency on main PostgreSQL chart
    @gsemet
    gsemet authored Feb 21, 2018
    Configuration menu
    Copy the full SHA
    1cd1e32 View commit details
    Browse the repository at this point in the history

  6. Use puckel image and split out imageTag.

    @jpds
    jpds committed Feb 21, 2018
    Configuration menu
    Copy the full SHA
    abb90e2 View commit details
    Browse the repository at this point in the history

  7. Removed airflow_cfg and all references to it.

    @jpds
    jpds committed Feb 21, 2018
    Configuration menu
    Copy the full SHA
    07faee4 View commit details
    Browse the repository at this point in the history

  8. Use database environment variables for all services.

    @jpds
    jpds committed Feb 21, 2018
    Configuration menu
    Copy the full SHA
    b018c1d View commit details
    Browse the repository at this point in the history

  9. Integrated redis chart and variables.

    @jpds
    jpds committed Feb 21, 2018
    Configuration menu
    Copy the full SHA
    47af54a View commit details
    Browse the repository at this point in the history

  10. deployments-web.yaml: Fixed health check endpoint.

    @jpds
    jpds committed Feb 21, 2018
    Configuration menu
    Copy the full SHA
    421bc57 View commit details
    Browse the repository at this point in the history

  11. values.yaml: Removed airflow_cfg.

    @jpds
    jpds committed Feb 21, 2018
    Configuration menu
    Copy the full SHA
    52ab63b View commit details
    Browse the repository at this point in the history

  12. Removed RabbitMQ references.

    @jpds
    jpds committed Feb 21, 2018
    Configuration menu
    Copy the full SHA
    7049217 View commit details
    Browse the repository at this point in the history

  13. services.yaml: Don't use NodePorts to allow for multiple deployments on 

    same cluster.
    @jpds
    jpds committed Feb 21, 2018
    Configuration menu
    Copy the full SHA
    036b700 View commit details
    Browse the repository at this point in the history

  14. templates/configmaps-env.yaml: Removed.

    @jpds
    jpds committed Feb 21, 2018
    Configuration menu
    Copy the full SHA
    03fd6c7 View commit details
    Browse the repository at this point in the history

  15. Adapted deployment labels to helm create defaults.

    @jpds
    jpds committed Feb 21, 2018
    Configuration menu
    Copy the full SHA
    29ac02b View commit details
    Browse the repository at this point in the history

  16. Replaced prefix option with dynamically generated helm release name.

    @jpds
    jpds committed Feb 21, 2018
    Configuration menu
    Copy the full SHA
    b8f942b View commit details
    Browse the repository at this point in the history

  17. Added airflow.config option for dynamic airflow variable setting.

    @jpds
    jpds committed Feb 21, 2018
    Configuration menu
    Copy the full SHA
    e784966 View commit details
    Browse the repository at this point in the history

  18. airflow/values.yaml: config value for dynamic airflow env assignment.

    @jpds
    jpds committed Feb 21, 2018
    Configuration menu
    Copy the full SHA
    94d7d74 View commit details
    Browse the repository at this point in the history

  19. statefulsets-workers: Specify imageTag.

    @jpds
    jpds committed Feb 21, 2018
    Configuration menu
    Copy the full SHA
    974b2ab View commit details
    Browse the repository at this point in the history

  20. statefulsets-workers: Fixed name field for lint.

    @jpds
    jpds committed Feb 21, 2018
    Configuration menu
    Copy the full SHA
    e7ae900 View commit details
    Browse the repository at this point in the history

Commits on Feb 22, 2018

  1. services.yaml: Match ingress serviceNames.

    @jpds
    jpds committed Feb 22, 2018
    Configuration menu
    Copy the full SHA
    d2bebb8 View commit details
    Browse the repository at this point in the history

  2. ingresses.yml: Match other chart label annotations and use subdomain for 

    flower instead of path prefixes.
    @jpds
    jpds committed Feb 22, 2018
    Configuration menu
    Copy the full SHA
    39664a5 View commit details
    Browse the repository at this point in the history

  3. values.yaml: Removed rabbitmq section.

    @jpds
    jpds committed Feb 22, 2018
    Configuration menu
    Copy the full SHA
    83c82e3 View commit details
    Browse the repository at this point in the history

Commits on Mar 1, 2018

  1. install airflow HEAD now my patch has been integrated

    @gsemet
    gsemet committed Mar 1, 2018
    Configuration menu
    Copy the full SHA
    539696c View commit details
    Browse the repository at this point in the history

Commits on Mar 2, 2018

  1. Merge pull request #14 from jpds/puckel-image 

    Use puckel/docker-airflow image
    @gsemet
    gsemet authored Mar 2, 2018
    Configuration menu
    Copy the full SHA
    74059f2 View commit details
    Browse the repository at this point in the history