Configuration & Environment Overrides

The cluster definition allows overriding configuration properties and environment variables per role or role group, with role group overrides taking precedence.

Overriding operator-set properties (e.g. HTTP port) may cause issues. Additionally, ensure consistent configurations across roles. Not all roles use each setting, but some things — such as external endpoints — need to be consistent to avoid problems.

Configuration Properties

Airflow exposes an environment variable for every Airflow configuration setting, a list of which can be found in the Configuration Reference.

As Airflow can be configured with python code too, arbitrary code can be added to the webserver_config.py. You can use either EXPERIMENTAL_FILE_HEADER to add code to the top or EXPERIMENTAL_FILE_FOOTER to add to the bottom.

This is an experimental feature.

webservers:
  configOverrides:
    webserver_config.py:
      CSV_EXPORT: "{'encoding': 'utf-8'}"
      EXPERIMENTAL_FILE_HEADER: |
        from modules.my_module import my_class
      EXPERIMENTAL_FILE_FOOTER: |
        import logging
        from airflow.security import AirflowSecurityManager

        class myCustomSecurityManger(AirflowSecurityManager):
          def __init__():
            init()

        CUSTOM_SECURITY_MANAGER = myCustomSecurityManger
  roleGroups:
    default:
      config: {}

Although Kubernetes can override these settings in one of two ways (Configuration overrides, or Environment Variable overrides), the effect is the same and currently only the latter is implemented. This is described in the following section.

Environment Variables

These can be set — or overwritten — at either the role level:

webservers:
  envOverrides:
    AIRFLOW__WEBSERVER__AUTO_REFRESH_INTERVAL: "8"
  roleGroups:
    default:
      replicas: 1

Or per role group:

webservers:
  roleGroups:
    default:
      envOverrides:
        AIRFLOW__WEBSERVER__AUTO_REFRESH_INTERVAL: "8"
      replicas: 1

In both examples, the UI DAG refresh default (3s) is replaced with 8s. All override property values must be strings.

Pod overrides

The Airflow operator also supports Pod overrides, enabling you to modify any Kubernetes Pod property. Refer to the Pod overrides documentation for details.