Use puckel/docker-airflow image

[jpds]

Use puckel/docker-airflow image, allows everything to be configured via environment variables and deploy redis too.

[gsemet]

why have you removed it? Did you found a way to specify the cfg file without this kung foo?

[gsemet]

won't accept this patch sorry. Why do you get rid of the custom configuration file?

[jpds]

Because all of Airflow can be configured through environment variables instead: https://airflow.apache.org/configuration.html

You can also set options with environment variables by using this format: $AIRFLOW__{SECTION}__{KEY}

I'll look into adding a option in the values so that envs can be dynamically set.

[jpds]

Added airflow.config to values.yaml, which let's one do:

-  config: {}
+  config:
+    AIRFLOW__CORE__SQL_ALCHEMY_CONN: my_conn_string

Which results in:

$ helm install --namespace testing-airflow ./airflow  --debug --dry-run
...
          env:
            - name: POSTGRES_HOST
              value: ...
            ...
            - name: AIRFLOW__CORE__SQL_ALCHEMY_CONN
              value: my_conn_string

I'm not sure if we need to split out variables between the different services, though this will be easy.

[gsemet]

maybe we can put all these env var in a configmap ?

[gsemet]

ok, this anwser suits me. I actually prefer to configure using environment variable. Can you add a few words on this feature on the main README?

For the configmap, I prefere using them for readability sake, but their update do not trigger a new deployment without some hacks. What is you opinion on it?

[gsemet]

note: we need to use a more recent version of airflow to support mount in subpath (apache/airflow#2952). Since airflow 2.0 is not released yet, we are stuck with custom build for this feature. Until then, only root path is supported.

[jpds]

Ah, I wasn't aware of custom features in the other docker image build.

[gsemet]

don't worry, I'll document the limitations.
Also, I think we should, at one point, move to the official helm repository. I'll be glad if you can help me on this :)

[jpds]

ok, this anwser suits me. I actually prefer to configure using environment variable. Can you add a few words on this feature on the main README?

Will do.

For the configmap, I prefere using them for readability sake, but their update do not trigger a new deployment without some hacks. What is you opinion on it?

I have come across a few charts that have this issue. They don't usually leverage the feature at:

• https://github.com/kubernetes/helm/blob/master/docs/charts_tips_and_tricks.md#automatically-roll-deployments-when-configmaps-or-secrets-change

But the helm upgrade command has a --recreate-pods option to force it.

[gsemet]

I like this annotation trick!