Security

Secure Apache Airflow by configuring user authentication and authorization. Airflow provides built-in user and role management, but can also connect to an LDAP server or an OIDC provider to manage users centrally instead.

Authentication

Users need to authenticate themselves before using Airflow, and there are several ways in which this can be set up.

Multiple authentication methods

Only one authentication method is supported at a time, and in case of LDAP, only one authentication class is allowed. This means, it is not possible to configure both LDAP and OIDC authentication methods at the same time, but it is possible to configure multiple OIDC classes or one LDAP authentication class.

Built-in user management

By default, users are manually set up in the Webserver UI. Use the blue "+" button to create users.

LDAP

Airflow supports user authentication against a single LDAP server. Set up an AuthenticationClass for the LDAP server and reference it in the Airflow Stacklet resource as shown:

apiVersion: airflow.stackable.tech/v1alpha1
kind: AirflowCluster
metadata:
  name: airflow-with-ldap
spec:
  image:
    productVersion: 2.10.2
  clusterConfig:
    authentication:
      - authenticationClass: ldap    (1)
        userRegistrationRole: Admin  (2)

1	The reference to an AuthenticationClass called `ldap`
2	The default role that all users are assigned to

Users that log in with LDAP are assigned to a default role which is specified with the userRegistrationRole property.

You can follow the Authentication with OpenLDAP tutorial to learn how to set up an AuthenticationClass for an LDAP server, as well as consulting the AuthenticationClass reference .

The users and roles can be viewed as before in the Webserver UI, but the blue "+" button is not available when authenticating via LDAP:

OpenID Connect

An OpenID Connect provider can be used for authentication. Unfortunately, there is no generic support for OpenID Connect built into Airflow. This means that only specific OpenID Connect providers can be configured.

Airflow deployments on the Stackable Data Platform only support Keycloak.

apiVersion: airflow.stackable.tech/v1alpha1
kind: AirflowCluster
metadata:
  name: airflow-with-oidc
spec:
  image:
    productVersion: 2.10.2
  clusterConfig:
    authentication:
    - authenticationClass: keycloak                       (1)
      oidc:
        clientCredentialsSecret: airflow-keycloak-client  (2)
      userRegistrationRole: User                          (3)

1	The reference to an AuthenticationClass called `keycloak`
2	The reference to the Secret containing the Airflow client credentials
3	The default role to which all users are assigned

Users that log in with OpenID Connect are assigned to a default role which is specified with the userRegistrationRole property.

The Secret containing the Airflow client credentials:

apiVersion: v1
kind: Secret
metadata:
  name: airflow-keycloak-client
stringData:
  clientId: airflow                    (1)
  clientSecret: airflow_client_secret  (2)

1	The client ID of Airflow as defined in Keycloak
2	The client secret as defined in Keycloak

A minimum client configuration in Keycloak for this example looks like this:

{
  "clientId": "airflow",
  "enabled": true,
  "clientAuthenticatorType": "client-secret", (1)
  "secret": "airflow_client_secret",
  "redirectUris": [
    "*"
  ],
  "webOrigins": [
    "*"
  ],
  "standardFlowEnabled": true, (2)
  "protocol": "openid-connect" (3)
}

1	Sets the OIDC type to confidential access type.
2	Enables the OAuth2 "Authorization Code Flow".
3	Enables OpenID Connect and OAuth2 support.

Further information for specifying an AuthenticationClass for an OIDC provider can be found at the concepts page.

Authorization

The Airflow Webserver delegates the handling of user access control to the Flask AppBuilder. The AuthManager in the Flask AppBuilder can be configured to fetch the user roles from the authentication backend, e.g. LDAP. Instead of using the integrated authorization, the Stackable Data Platform also provides an AuthManager which delegates the authorization requests to an Open Policy Agent (OPA).

Integrated authorization

Webinterface

You can view, add to, and assign the roles displayed in the Airflow Webserver UI to existing users.

LDAP

Airflow supports assigning Roles to users based on their LDAP group membership, though this is not yet supported by the Stackable operator. All the users logging in via LDAP get assigned to the same role which you can configure via the attribute authenticationConfig.userRegistrationRole on the AirflowCluster object:

apiVersion: airflow.stackable.tech/v1alpha1
kind: AirflowCluster
metadata:
  name: airflow-with-ldap
spec:
  clusterConfig:
    authentication:
      - authenticationClass: ldap (1)
        userRegistrationRole: Admin  (2)

1	The reference to an AuthenticationClass called `ldap`
2	All users are assigned to the `Admin` role

OpenID Connect

The mechanism for assigning roles to users described in the LDAP section also applies to OpenID Connect. Airflow supports assigning Roles to users based on their OpenID Connect scopes, though this is not yet supported by the Stackable operator. All the users logging in via OpenID Connect get assigned to the same role which you can configure via the attribute authentication[*].userRegistrationRole on the AirflowCluster object:

apiVersion: airflow.stackable.tech/v1alpha1
kind: AirflowCluster
metadata:
  name: airflow-with-oidc
spec:
  clusterConfig:
    authentication:
    - authenticationClass: keycloak
      oidc:
        clientCredentialsSecret: airflow-keycloak-client
      userRegistrationRole: Admin  (1)

1	All users are assigned to the `Admin` role

Open Policy Agent

Authorization with an Open Policy Agent can be enabled with the following cluster configuration:

apiVersion: airflow.stackable.tech/v1alpha1
kind: AirflowCluster
metadata:
  name: airflow-with-opa
spec:
  clusterConfig:
    authorization:
      opa:
        configMapName: opa      (1)
        package: airflow        (2)
        cache:                  (3)
          entryTimeToLive: 10s  (4)
          maxEntries: 100000    (5)

1	The service discovery ConfigMap for the OPA instance containing the URL of the OPA API
2	The Rego rule package with the authorization rules
3	A cache for authorization requests to the Open Policy Agent to reduce the load on OPA and to bridge restarts of the OPA pods; Defaults are used, if not set explicitly.
4	Time to live per cached authorization request; Defaults to 30 seconds; Changes in the Rego rules may not be effective within the given duration.
5	Maximum number of cached authorization requests in the cache; Defaults to 10,000 entries; If this limit is reached then the least recently used entry is removed and the metric `airflow_opa_cache_limit_reached` is increased by one. The cache size should probably be increased if this metric is constantly raised.

The Rego rule package defined in the configuration must contain specific rules which are true or false dependent on the input which differs slightly between the rules. The following list contains the rule names as well as a specification of the possible input:

is_authorized_configuration

Returns whether the user is authorized to perform a given action on configuration.

"input": {
  "method": "<the method to perform>",          (1)
  "details": {
    "section": "<name of the section or null>"  (2)
  },
  "user": {
    "id": "<the user ID in Airflow>",
    "name": "<the user name>"
  }
}

1	One of "GET", "POST", "PUT", "DELETE" or "MENU"
2	`null` if the action is performed on all configuration sections

is_authorized_connection

Returns whether the user is authorized to perform a given action on a connection.

"input": {
  "method": "<the method to perform>",             (1)
  "details": {
    "conn_id": "<name of the connection or null>"  (2)
  },
  "user": {
    "id": "<the user ID in Airflow>",
    "name": "<the user name>"
  }
}

1	One of "GET", "POST", "PUT", "DELETE" or "MENU"
2	`null` if the action is performed on all connections

is_authorized_dag

Returns whether the user is authorized to perform a given action on a DAG.

"input": {
  "method": "<the method to perform>",       (1)
  "access_entity": "<DAG entity or null>",   (2)
  "details": {
    "dag_id": "<the ID of the DAG or null>"  (3)
  },
  "user": {
    "id": "<the user ID in Airflow>",
    "name": "<the user name>"
  }
}

1	One of "GET", "POST", "PUT", "DELETE" or "MENU"
2	The kind of DAG information the authorization request is about. If not provided, the authorization request is about the DAG itself. One of "AUDIT_LOG", "CODE", "DEPENDENCIES", "RUN", "SLA_MISS", "TASK", "TASK_INSTANCE", "TASK_RESCHEDULE", "TASK_LOGS", "WARNING" or "XCOM"
3	`null` if the action is performed on all DAGs

is_authorized_dataset

Returns whether the user is authorized to perform a given action on a dataset.

"input": {
  "method": "<the method to perform>",         (1)
  "details": {
    "uri": "<the URI of the dataset or null>"  (2)
  },
  "user": {
    "id": "<the user ID in Airflow>",
    "name": "<the user name>"
  }
}

1	One of "GET", "POST", "PUT", "DELETE" or "MENU"
2	`null` if the action is performed on all datasets

is_authorized_pool

Returns whether the user is authorized to perform a given action on a pool.

"input": {
  "method": "<the method to perform>",        (1)
  "details": {
    "name": "<the name of the pool or null>"  (2)
  },
  "user": {
    "id": "<the user ID in Airflow>",
    "name": "<the user name>"
  }
}

1	One of "GET", "POST", "PUT", "DELETE" or "MENU"
2	`null` if the action is performed on all pools

is_authorized_variable

Returns whether the user is authorized to perform a given action on a variable.

"input": {
  "method": "<the method to perform>",          (1)
  "details": {
    "key": "<the key of the variable or null>"  (2)
  },
  "user": {
    "id": "<the user ID in Airflow>",
    "name": "<the user name>"
  }
}

1	One of "GET", "POST", "PUT", "DELETE" or "MENU"
2	`key` if the action is performed on all variables

is_authorized_view

Returns whether the user is authorized to access a read-only state of the installation.
```
"input": {
  "access_view": "<the variant of the view>",  (1)
  "user": {
    "id": "<the user ID in Airflow>",
    "name": "<the user name>"
  }
}
```
1 The specific read-only view/state the authorization request is about. One of "CLUSTER_ACTIVITY", "DOCS", "IMPORT_ERRORS", "JOBS", "PLUGINS", "PROVIDERS", "TRIGGERS" or "WEBSITE".
is_authorized_custom_view

Returns whether the user is authorized to perform a given action on a custom view.

A custom view can be a view defined as part of the auth manager. This view is then only available when the auth manager is used as part of the environment. It can also be a view defined as part of a plugin defined by a user.
```
"input": {
  "method": "<the method to perform>", (1)
  "resource_name": "<the name of the resource>",
  "user": {
    "id": "<the user ID in Airflow>",
    "name": "<the user name>"
  }
}
```
1 Usually one of "GET", "POST", "PUT", "DELETE" or "MENU", but the method can also be a string if the action has been defined in a plugin. In that case, the action can be anything.

The roles defined in Airflow are not used when the authorization is performed by OPA. Therefore, it makes sense to hide the menu entry "List Roles" simply by not allowing access to the custom view "List Roles".

A ConfigMap with the Rego rules could look as follows:

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: airflow-rules
  labels:
    opa.stackable.tech/bundle: "true"
data:
  airflow.rego: |

    # The Rego rule package which must be defined in the cluster
    # configuration (spec.clusterConfig.authorization.opa.package).
    package airflow

    import rego.v1

    default is_authorized_configuration := false
    default is_authorized_connection := false
    default is_authorized_dag := false
    default is_authorized_dataset := false
    default is_authorized_pool := false
    default is_authorized_variable := false
    default is_authorized_view := false
    default is_authorized_custom_view := false

    # Allow everybody to get the DAG runs of example_trigger_target_dag
    is_authorized_dag if {
        input.method == "GET"
        input.access_entity == "RUN"
        input.details.id == "example_trigger_target_dag"
    }

    # Allow the administrator to access all custom views but hide the
    # menu "List Roles" and disallow the access to the Roles resource.
    is_authorized_custom_view if {
        input.resource_name != "List Roles"
        input.resource_name != "Roles"

        input.user.name == "admin"
    }

The User Info Fetcher can be used to fetch the groups in which the user is a member of. The Rego rule can then grant access based on the group membership.