Log aggregation

The logs can be forwarded to a Vector log aggregator by providing a discovery ConfigMap for the aggregator and by enabling the log agent:

The task handler is responsible for showing the task logs in the UI. Unfortunately, the log level of the task handler cannot be specified. To avoid that all logs are emitted to the UI, the log level of the airflow.task logger is set explicitly to INFO. You can change the log level as shown below.

spec:
  clusterConfig:
    vectorAggregatorConfigMapName: vector-aggregator-discovery
  webservers:
    config:
      logging:
        enableVectorAgent: true
        containers:
          airflow:
            loggers:
              "flask_appbuilder":
                level: WARN
  celeryExecutors:
    config:
      logging:
        enableVectorAgent: true
        containers:
          airflow:
            loggers:
              "airflow.processor":
                level: INFO
              "airflow.task":
                level: DEBUG
  schedulers:
    config:
      logging:
        enableVectorAgent: true
        containers:
          airflow:
            loggers:
              "airflow.processor_manager":
                level: INFO

Further information on how to configure logging, can be found in Logging.