OpenLineage

Airflow can emit OpenLineage lineage events describing the datasets your DAGs read and write. This is implemented by the Airflow OpenLineage provider, which must be installed in the product image.

The apache-airflow-providers-openlineage provider package must be present in the Airflow image. The Stackable images include it; if you build a custom image, make sure the provider is installed.

Enabling OpenLineage

Set spec.clusterConfig.lineage and provide a connection to your OpenLineage backend. The connection can be defined inline:

apiVersion: airflow.stackable.tech/v1alpha2
kind: AirflowCluster
metadata:
  name: airflow
spec:
  clusterConfig:
    lineage:
      connection:
        inline:
          http:              (1)
            host: marquez    (2)
            port: 5000       (3)
      namespace: production  (4)

1	The transport used to publish events. `http` is currently the only supported transport.
2	Host of the OpenLineage backend, without protocol or port.
3	Port the backend listens on.
4	OpenLineage namespace lineage is reported under. Defaults to `default`.

The operator configures the transport via the OpenLineage client environment variables (OPENLINEAGETRANSPORT*) and sets AIRFLOWOPENLINEAGENAMESPACE. Lineage events are emitted to \http(s)://<host>:<port><path>, where path defaults to /api/v1/lineage.

Referencing an OpenLineageConnection

To share a backend definition across multiple stacklets, create an OpenLineageConnection resource and reference it by name. The referenced resource must live in the same namespace as the AirflowCluster.

apiVersion: lineage.stackable.tech/v1alpha1
kind: OpenLineageConnection
metadata:
  name: marquez
spec:
  http:
    host: marquez
    port: 5000

apiVersion: airflow.stackable.tech/v1alpha2
kind: AirflowCluster
metadata:
  name: airflow
spec:
  clusterConfig:
    lineage:
      connection:
        reference: marquez  (1)

1	Name of the `OpenLineageConnection` resource.

TLS

Configure TLS on the connection to reach a backend over HTTPS:

lineage:
  connection:
    inline:
      http:
        host: marquez
        port: 5000
        tls:
          verification:
            server:
              caCert:
                secretClass: openlineage-tls  (1)

1	A SecretClass providing the CA certificate used to verify the backend.

When a SecretClass CA is configured, the CA certificate is mounted into the Airflow pods and the OpenLineage client is pointed at it via REQUESTS_CA_BUNDLE.

REQUESTS_CA_BUNDLE is set on the whole Airflow container, not just for the OpenLineage client. The Python requests library reads it for every call that verifies certificates, and it replaces the default certifi CA bundle instead of extending it. This might break DAGs and Airflow itself. If this is a problem, build a CA bundle that contains both the certifi CAs and the OpenLineage backend CA.

Authentication

If the backend requires a bearer token, set credentialsSecretName on the connection to the name of a Secret containing the token. The Secret must contain the token under the fixed key apiKey and must live in the same namespace as the cluster:

apiVersion: v1
kind: Secret
metadata:
  name: openlineage-credentials
stringData:
  apiKey: "<your-token>"  (1)

1	The bearer token, under the fixed key `apiKey`.

lineage:
  connection:
    inline:
      http:
        host: marquez
        port: 5000
        credentialsSecretName: openlineage-credentials  (1)

1	Name of the Secret holding the API key.

The token is injected into the Airflow pods from the Secret via a secretKeyRef environment variable. It never appears in the CRD, a ConfigMap, or a plaintext environment value.

Endpoint path

By default events are sent to /api/v1/lineage on the backend. Set path on the transport to target a different endpoint:

lineage:
  connection:
    inline:
      http:
        host: marquez
        port: 5000
        path: /custom/lineage  (1)

1	URL path of the endpoint events are sent to. Defaults to `/api/v1/lineage`.

Overriding injected configuration

The OpenLineage environment variables are applied to every role (and the Kubernetes-executor pod template) before user envOverrides, so you can override any of them. For example, to disable emission for a single role group without removing the lineage block:

webservers:
  roleGroups:
    default:
      envOverrides:
        AIRFLOW__OPENLINEAGE__DISABLED: "true"

See Configuration & Environment Overrides for more on overrides.