Monitoring

In November 2024, Apache NiFi released a new major version 2.0.0.

The NiFi 2.0.0 release changed the way of exposing Prometheus metrics significantly. The following steps explain on how to expose Metrics in NiFi versions 1.x.x and 2.x.x.

Configure metrics in NiFi 1.x.x

For NiFi versions 1.x.x, the operator automatically configures NiFi to export Prometheus metrics. This is done by creating a Job that connects to NiFi and configures a Prometheus Reporting Task.

Network access from the Job to NiFi is required. If you are running a Kubernetes with restrictive NetworkPolicies, make sure to allow access from the Job to NiFi.

See Monitoring for more details.

Disabling create-reporting-task Job

It can be helpful to disable the Job, e.g. when you configOverride an authentication mechanism, which the Job currently cannot use to authenticate against NiFi.

To achieve this use the following configuration:

spec:
  clusterConfig:
    createReportingTaskJob:
      enabled: false

Configure metrics in NiFi 2.x.x

The Prometheus Reporting Task was removed in NiFi 2.x.x in NIFI-13507. Metrics are now always exposed and can be scraped using the NiFi metrics Service and the HTTP path /nifi-api/flow/metrics/prometheus.

For a deployed NiFi cluster called simple-nifi, containing a rolegroup called default, the metrics endpoint is reachable under:

https://simple-nifi-node-default-metrics.<namespace>.svc.cluster.local:8443/nifi-api/flow/metrics/prometheus
The above URL connects to one of the Pods, reachable through the specified Service, therefore scraping metrics produced by that Pod only. To scrape metrics from a particular Pod, the FQDN of the Pod and the headless Service need to be used. For example: https://simple-nifi-node-default-0.simple-nifi-node-default-headless.<namespace>.svc.cluster.local:8443/nifi-api/flow/metrics/prometheus
If NiFi is configured to do any user authentication, requests to the metrics endpoint must be authenticated and authorized.

Authentication with NiFi 2.x.x

To authenticate against the NiFi 2.x.x API, you can configure mTLS between NiFi and the client calling NiFi. For more information about authentication between Kubernetes Pods, check out the Secret Operator documentation.

The following example illustrates the configuration of a Prometheus scraper for NiFi, using the aforementioned method of configuring mTLS and utilizing the internally available tls SecretClass.

To generate a client certificate signed by the tls SecretClass CA trusted in NiFi, add the following volume and volumeMount to the Prometheus Pod.

If the Prometheus Operator is used to deploy Prometheus, there is currently a known bug, which prevents adding an additional Volume containing annotations on the volumeClaimTemplate. The bug is tracked in the Prometheus Operator Issues. The annotations are necessary to configure the behavior of the Secret Operator. As a current workaround, until the issue is resolved, one could deploy an additional Pod only responsible for creating a TLS certificate as a Secret, which then can be used by the ServiceMonitor. This workaround is illustrated in the monitoring Stack. 
---
prometheus: (1)
  prometheusSpec:
    volumes:
      - name: tls
        ephemeral:
          volumeClaimTemplate:
            metadata:
              annotations:
                secrets.stackable.tech/class: tls (2)
                secrets.stackable.tech/scope: pod,service=prometheus-kube-prometheus-prometheus (3)
            spec:
              storageClassName: secrets.stackable.tech
              accessModes:
                - ReadWriteOnce
              resources:
                requests:
                  storage: "1"
    volumeMounts:
      - name: tls
        mountPath: /stackable/tls (4)
1 This given configuration is set in the Prometheus resource for the Prometheus Operator
2 The tls SecretClass created by the Secret Operator, storing its CA in a Kubernetes Secret. Any other SecretClass can be used as well
3 The service=prometheus-kube-prometheus-prometheus scope is added to include the subjectAlternateName of the Prometheus Service in the generated TLS certificate. This particular Service name, used here, refers to the Prometheus Service deployed by the Prometheus Operator
4 The path where the mTLS certificates are mounted inside the Prometheus Pod

If you want to use a ServiceMonitor you’d need to configure it as follows:

---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: scrape-nifi-2
  labels:
    stackable.tech/vendor: Stackable
    release: prometheus
spec:
  endpoints:
  - path: /nifi-api/flow/metrics/prometheus
    port: https
    scheme: https
    tlsConfig: (1)
      caFile: /stackable/tls/ca.crt
      certFile: /stackable/tls/tls.crt
      keyFile: /stackable/tls/tls.key
    relabelings: (2)
      - sourceLabels:
          - __meta_kubernetes_pod_name
          - __meta_kubernetes_service_name
          - __meta_kubernetes_namespace
          - __meta_kubernetes_pod_container_port_number
        targetLabel: __address__
        replacement: ${1}.${2}-headless.${3}.svc.cluster.local:${4} (3)
        regex: (.+);(.+?)(?:-metrics)?;(.+);(.+)
  selector:
    matchLabels:
      stackable.tech/vendor: Stackable
      prometheus.io/scrape: "true"
  namespaceSelector:
    any: true
  jobLabel: app.kubernetes.io/instance
1 In the TLS configuration of the ServiceMonitor, specify the location of the cert and key files mounted into the Prometheus Pod
2 Relabel address to be a FQDN rather then the IP-Address of the target Pod. This is currently necessary to scrape NiFi, since it requires a DNS name to address the NiFi REST API
3 Currently, the NiFi StatefulSet only offers using FQDNs for NiFi Pods through the headless Service, which is why we use the headless Service instead of the metrics Service to scrape NiFi metrics
The SDP exposes a dedicated metrics Service since the Listener integration.

The described example is part of the Prometheus and ServiceMonitor manifests used in the monitoring stack of the demos repository.