-
-
Notifications
You must be signed in to change notification settings - Fork 204
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- add EditorConfig files - airflow.all.yaml is splited in different chart templates - Propagate server config from env variable if defined - statup script can receive some config by ENV variable (name of rabbitmq/postgres server), at least everything derives from the same source - username and password can inherit from env - fix probable buggy fernet key definition, missing enum34 with python 2 installation - change format of template in airflow.cfg (airflow.cfg.in) - use configmap to set default environment variables - parameterize number of workers - support url prefix for flower and web ui allow ingress in endpoint such as /airflow and /flower - use python 3 in docker - optional git sync to mirror a git repo for DAGs - debian stretch (python >3.4) - load example true/false and flexible scheduler run nums
- Loading branch information
Showing
16 changed files
with
662 additions
and
37 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,19 @@ | ||
| root = true | ||
|
|
||
| [*] | ||
| indent_style = space | ||
| indent_size = 4 | ||
| end_of_line = lf | ||
| charset = utf-8 | ||
| trim_trailing_whitespace = true | ||
| insert_final_newline = true | ||
|
|
||
| [*.md] | ||
| trim_trailing_whitespace = false | ||
|
|
||
| [*.yaml] | ||
| indent_size = 2 | ||
|
|
||
| [Makefile] | ||
| indent_style = tab | ||
| indent_size = 4 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,2 +1,3 @@ | ||
| *~ | ||
| build | ||
| rootfs |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,4 @@ | ||
| apiVersion: v1 | ||
| description: Airflow installation | ||
| name: airflow | ||
| version: v0.1.0 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,29 @@ | ||
| apiVersion: v1 | ||
| kind: ConfigMap | ||
| metadata: | ||
| name: "{{ .Values.airflow.prefix -}}airflow-cfg" | ||
| data: | ||
| AIRFLOW_HOME: "/usr/local/airflow" | ||
| {{- if .Values.airflow.fernet_key }} | ||
| FERNET_KEY: "{{- .Values.airflow.fernet_key -}}" | ||
| {{- end }} | ||
| RABBITMQ_HOST: "{{- .Values.airflow.prefix -}}{{- .Values.db.rabbitmq.basename -}}" | ||
| {{- if .Values.db.rabbitmq.user }} | ||
| RABBITMQ_CREDS: "{{- .Values.db.rabbitmq.user -}}:{{- .Values.db.rabbitmq.password -}}" | ||
| {{- end }} | ||
| POSTGRES_HOST: "{{- .Values.airflow.prefix -}}{{- .Values.db.postgres.basename -}}" | ||
| {{- if .Values.db.postgres.user }} | ||
| POSTGRES_CREDS: "{{- .Values.db.postgres.user -}}:{{- .Values.db.postgres.password -}}" | ||
| {{- end }} | ||
| {{- if .Values.airflow.init_retry_loop }} | ||
| TRY_LOOP: "{{ .Values.airflow.init_retry_loop -}}" | ||
| {{- end }} | ||
| FLOWER_URL_PREFIX: "{{ .Values.flower.url_prefix -}}" | ||
| AIRFLOW_URL_PREFIX: "{{ .Values.airflow.url_prefix -}}" | ||
| {{ if .Values.dags.git_sync_enabled }} | ||
| GIT_SYNC_REPO: "{{ .Values.dags.git_repo }}" | ||
| GIT_SYNC_BRANCH: "{{ .Values.dags.git_branch }}" | ||
| GIT_SYNC_WAIT: "{{ .Values.dags.poll_interval_sec }}" | ||
| GIT_SYNC_DEBUG: "{{ .Values.dags.git_sync_debug }}" | ||
| {{ end }} | ||
| LOAD_DAGS_EXAMPLES: "{{ .Values.dags.load_examples }}" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,155 @@ | ||
| apiVersion: extensions/v1beta1 | ||
| kind: Deployment | ||
| metadata: | ||
| name: {{ .Values.airflow.prefix -}}postgres | ||
| spec: | ||
| replicas: 1 | ||
| template: | ||
| metadata: | ||
| labels: | ||
| app: airflow | ||
| tier: db | ||
| spec: | ||
| containers: | ||
| - name: {{ .Values.airflow.prefix -}}postgres | ||
| image: postgres | ||
| ports: | ||
| - name: {{ .Values.airflow.prefix -}}postgres | ||
| containerPort: 5432 | ||
| env: | ||
| - name: POSTGRES_USER | ||
| value: "{{ .Values.db.postgres.user -}}" | ||
| - name: POSTGRES_PASSWORD | ||
| value: "{{ .Values.db.postgres.password -}}" | ||
| - name: POSTGRES_DB | ||
| value: "{{ .Values.db.postgres.database -}}" | ||
| --- | ||
| apiVersion: extensions/v1beta1 | ||
| kind: Deployment | ||
| metadata: | ||
| name: {{ .Values.airflow.prefix -}}rabbitmq | ||
| spec: | ||
| replicas: 1 | ||
| template: | ||
| metadata: | ||
| labels: | ||
| app: airflow | ||
| tier: rabbitmq | ||
| spec: | ||
| restartPolicy: Always | ||
| containers: | ||
| - name: {{ .Values.airflow.prefix -}}rabbitmq | ||
| image: rabbitmq:3-management | ||
| ports: | ||
| - name: {{ .Values.airflow.prefix -}}management | ||
| containerPort: 15672 | ||
| - name: {{ .Values.airflow.prefix -}}node | ||
| containerPort: 5672 | ||
| env: | ||
| - name: RABBITMQ_DEFAULT_USER | ||
| value: "{{ .Values.db.rabbitmq.user -}}" | ||
| - name: RABBITMQ_DEFAULT_PASS | ||
| value: "{{ .Values.db.rabbitmq.password -}}" | ||
| - name: RABBITMQ_DEFAULT_VHOST | ||
| value: "airflow" | ||
| --- | ||
| apiVersion: extensions/v1beta1 | ||
| kind: Deployment | ||
| metadata: | ||
| name: {{ .Values.airflow.prefix -}}web | ||
| spec: | ||
| replicas: 1 | ||
| template: | ||
| metadata: | ||
| labels: | ||
| app: airflow | ||
| tier: web | ||
| spec: | ||
| restartPolicy: Always | ||
| containers: | ||
| - name: {{ .Values.airflow.prefix -}}web | ||
| image: {{ .Values.airflow.image }} | ||
| imagePullPolicy: {{ .Values.airflow.image_pull_policy}} | ||
| envFrom: | ||
| - configMapRef: | ||
| name: {{ .Values.airflow.prefix }}airflow-cfg | ||
| ports: | ||
| - name: {{ .Values.airflow.prefix -}}web | ||
| containerPort: 8080 | ||
| args: ["webserver"] | ||
| --- | ||
| apiVersion: extensions/v1beta1 | ||
| kind: Deployment | ||
| metadata: | ||
| name: {{ .Values.airflow.prefix -}}flower | ||
| spec: | ||
| replicas: 1 | ||
| template: | ||
| metadata: | ||
| labels: | ||
| app: airflow | ||
| tier: flower | ||
| spec: | ||
| restartPolicy: Always | ||
| containers: | ||
| - name: {{ .Values.airflow.prefix -}}flower | ||
| image: {{ .Values.airflow.image }} | ||
| imagePullPolicy: {{ .Values.airflow.image_pull_policy}} | ||
| env: | ||
| - name: FLOWER_PORT | ||
| value: "5555" | ||
| envFrom: | ||
| - configMapRef: | ||
| name: {{ .Values.airflow.prefix }}airflow-cfg | ||
| ports: | ||
| - name: {{ .Values.airflow.prefix -}}flower | ||
| containerPort: 5555 | ||
| args: ["flower"] | ||
| --- | ||
| apiVersion: extensions/v1beta1 | ||
| kind: Deployment | ||
| metadata: | ||
| name: {{ .Values.airflow.prefix -}}scheduler | ||
| spec: | ||
| replicas: 1 | ||
| template: | ||
| metadata: | ||
| labels: | ||
| app: airflow | ||
| tier: scheduler | ||
| spec: | ||
| restartPolicy: Always | ||
| containers: | ||
| - name: {{ .Values.airflow.prefix -}}scheduler | ||
| image: {{ .Values.airflow.image }} | ||
| imagePullPolicy: {{ .Values.airflow.image_pull_policy}} | ||
| # volumes: | ||
| # - /localpath/to/dags:/usr/local/airflow/dags | ||
| envFrom: | ||
| - configMapRef: | ||
| name: {{ .Values.airflow.prefix }}airflow-cfg | ||
| args: ["scheduler", "-n", "{{ .Values.airflow.scheduler_num_runs }}"] | ||
| --- | ||
| apiVersion: extensions/v1beta1 | ||
| kind: Deployment | ||
| metadata: | ||
| name: {{ .Values.airflow.prefix -}}worker | ||
| spec: | ||
| replicas: {{ .Values.celery.num_workers }} | ||
| template: | ||
| metadata: | ||
| labels: | ||
| app: airflow | ||
| tier: worker | ||
| spec: | ||
| restartPolicy: Always | ||
| containers: | ||
| - name: {{ .Values.airflow.prefix -}}worker | ||
| image: {{ .Values.airflow.image }} | ||
| imagePullPolicy: {{ .Values.airflow.image_pull_policy}} | ||
| # volumes: | ||
| # - /localpath/to/dags:/usr/local/airflow/dags | ||
| envFrom: | ||
| - configMapRef: | ||
| name: {{ .Values.airflow.prefix }}airflow-cfg | ||
| args: ["worker"] |
Oops, something went wrong.