Get started (free)

OpenLineage

OpenLineage is an open standard for data lineage collection. Trino ships an OpenLineage event listener as a core plugin, which emits lineage events for the queries run on the cluster. The spec.clusterConfig.lineage block in the TrinoCluster custom resource configures the OpenLineage event listener and its connection to a backend.

When configured, the operator:

  • writes an event-listener.properties file on the coordinator selecting the OpenLineage plugin and its HTTP transport

  • points the transport at the backend endpoint (<scheme>://<host>:<port>)

  • records this cluster’s trino.uri and a lineage namespace

  • (optionally) imports a backend CA certificate into the coordinator trust store and injects a bearer token for authenticated backends

The event listener runs on the coordinator only; workers are not affected.

By default, SELECT queries do not emit any lineage events, so a cluster that only reads data produces no lineage at all. To also capture reads, set openlineage-event-listener.trino.include-query-types as shown in Emitting lineage for SELECT queries.

Enabling OpenLineage

OpenLineage is configured through spec.clusterConfig.lineage.connection, which either inlines an OpenLineage connection or references a standalone OpenLineageConnection resource by name:

apiVersion: trino.stackable.tech/v1alpha1
kind: TrinoCluster
metadata:
  name: trino
spec:
  clusterConfig:
    lineage: (1)
      connection: (2)
        inline:
          http:
            host: marquez
            port: 5000
            # path: /api/v1/lineage (4)
            # tls:
            #   verification:
            #     server:
            #       caCert:
            #         secretClass: marquez-ca
      # namespace: trino-lineage (3)
      # jobNameFormat: "$QUERY_ID" (5)
      # datasetNamespaceUri: https://trino-prod (6)
  ...
1 Adding the lineage block enables OpenLineage event emission. Omit it entirely to leave OpenLineage off (the default).
2 Required. The OpenLineage backend connection. Use inline (shown here) to define the connection directly, or reference: <name> to point at an OpenLineageConnection resource in the same namespace.
3 Optional. The OpenLineage namespace events are reported under. Defaults to default.
4 Optional. URL path of the endpoint events are POSTed to. Defaults to /api/v1/lineage.
5 Optional. Sets the OpenLineage job name format. Accepts an arbitrary string with optional $QUERY_ID, $USER, $SOURCE and $CLIENT_IP substitution variables. Defaults to $QUERY_ID. See the Trino OpenLineage event listener documentation .
6 Optional. Sets openlineage-event-listener.trino.uri, from which Trino derives the namespace of every dataset it reports. The URI is only an identifier and is never connected to. Defaults to the coordinator Service address <scheme>://<coordinator service name>:<port> (http and 8080 when server TLS is disabled) when empty or unset, which yields the same dataset namespace (trino://<coordinator service name>:8443) that other OpenLineage producers, such as Airflow, report when they connect to that Service. Set it explicitly if they connect via a different host or port.

The OpenLineageConnection resource (lineage.stackable.tech) is a reusable definition of a connection to an OpenLineage backend, provided by the Stackable operator library. Reference it to share one backend definition across multiple clusters:

apiVersion: lineage.stackable.tech/v1alpha1
kind: OpenLineageConnection
metadata:
  name: marquez
spec:
  http:
    host: marquez
    port: 5000

With TLS server verification, the host must match a Subject Alternative Name in the backend’s certificate. Use the exact hostname the certificate was issued for (for an in-cluster backend, typically its full service FQDN such as marquez.<namespace>.svc.cluster.local), otherwise the TLS handshake fails and no events are delivered.

Authentication

Backends that require authentication (for example DataHub, or a Marquez instance behind a gateway) expect a bearer token on each request. Configure this by setting credentialsSecretName on the connection to the name of a Secret holding the token:

apiVersion: trino.stackable.tech/v1alpha1
kind: TrinoCluster
metadata:
  name: trino
spec:
  clusterConfig:
    lineage:
      connection:
        inline:
          http:
            host: marquez
            port: 5000
            credentialsSecretName: openlineage-token (1)
  ...
---
apiVersion: v1
kind: Secret
metadata:
  name: openlineage-token
  namespace: default (2)
stringData:
  apiKey: "s3cr3t-bearer-token" (3)
1 Name of the Secret holding the token. If omitted, no authentication is used. The Secret must exist in the same namespace as the TrinoCluster.
2 The Secret must be in the cluster’s namespace so it can be projected into the coordinator.
3 The token must be stored under the fixed key apiKey. The token is mounted as a file into the coordinator and resolved into the event listener configuration at startup, so it never appears in the ConfigMap or the pod’s environment.

The operator translates this into the OpenLineage HTTP transport’s openlineage-event-listener.transport.api-key (bearer) authentication. Combine it with the tls block above when the backend also needs TLS server verification.

Overriding the generated configuration

The generated event-listener.properties can be extended or overridden per role group through config overrides, using the event-listener.properties file name. Overrides are merged last and therefore take precedence over the operator-generated values:

spec:
  coordinators:
    configOverrides:
      event-listener.properties:
        openlineage-event-listener.transport.timeout: "10s"

Event listeners run on the coordinator, so event-listener.properties overrides only take effect on the coordinator role.

Emitting lineage for SELECT queries

openlineage-event-listener.trino.include-query-types controls which query types emit lineage events. It replaces the default list, so repeat the default types when adding SELECT:

spec:
  coordinators:
    configOverrides:
      event-listener.properties:
        openlineage-event-listener.trino.include-query-types: "SELECT,INSERT,DELETE,MERGE,UPDATE,ALTER_TABLE_EXECUTE,DATA_DEFINITION"

Every SELECT then produces a START and a COMPLETE (or FAIL) event, which can noticeably increase the load on the lineage backend for clusters serving many interactive or dashboard queries.