Monitoring

In November 2024, Apache NiFi released a new major version 2.0.0.

The NiFi 2.0.0 release changed the way of exposing Prometheus metrics significantly. The following steps explain on how to expose Metrics in NiFi versions 1.x.x and 2.x.x.

Configure metrics in NiFi 1.x.x

For NiFi versions 1.x.x, the operator automatically configures NiFi to export Prometheus metrics. This is done by creating a Job that connects to NiFi and configures a Prometheus Reporting Task.

Network access from the Job to NiFi is required. If you are running a Kubernetes with restrictive NetworkPolicies, make sure to allow access from the Job to NiFi.

See Monitoring for more details.

Disabling create-reporting-task Job

It can be helpful to disable the Job, e.g. when you configOverride an authentication mechanism, which the Job currently cannot use to authenticate against NiFi.

To achieve this use the following configuration:

spec:
  clusterConfig:
    createReportingTaskJob:
      enabled: false

Configure metrics in NiFi 2.x.x

The Prometheus Reporting Task was removed in NiFi 2.x.x in NIFI-13507. Metrics are now always exposed and can be scraped using the NiFi Pod FQDN and the HTTP path /nifi-api/flow/metrics/prometheus.

For a deployed single node NiFi cluster called simple-nifi, containing a rolegroup called default, the metrics endpoint is reachable under:

https://simple-nifi-node-default-0.simple-nifi-node-default.<namespace>.svc.cluster.local:8443/nifi-api/flow/metrics/prometheus
If NiFi is configured to do any user authentication, requests to the metric endpoint must be authenticated and authorized.

Authentication with NiFi 2.x.x

=== The NiFi metrics endpoints are behind a strong authentication mechanism which require credentials for each individual pod. ===

To authenticate, you can use a bearer token created by your NiFi instance e.g.

curl -X POST https://simple-nifi-node-default-0.simple-nifi-node-default.<namespace>.svc.cluster.local:8443/nifi-api/access/token -d 'username=<user>&password=<password>' -k

where -k equals verify=false to allow self-signed certificates. The reply is your bearer token.

The following example shows how to configure the Prometheus scraper to use the bearer token to authenticate against a NiFi pod.

---
authorization: (1)
  type: Bearer
  credentials: "<Bearer Token>" (2)
tls_config:
  insecure_skip_verify: true
static_configs:
  - targets:
    - '<pod>.<statefulset>.svc.cluster.local:8443' (3)
metrics_path: '/nifi-api/flow/metrics/prometheus'
scheme: https
1 Use the authorization property instead if the basic_auth.
2 Add the previously obtained token here.
3 Static targets only scrapes one pod.

or use it in a NiFi secret which should look like

---
apiVersion: v1
kind: Secret
metadata:
  name: nifi-authorization-secret
type: Opaque
stringData:
  nifi_token: "<Bearer_token>"

If you want to use a ServiceMonitor you’d need to configure it as follows:

---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: scrape-nifi2
  labels:
    stackable.tech/vendor: Stackable
    release: prometheus
spec:
  endpoints:
  - port: https
    path: 'nifi-api/flow/metrics/prometheus'
    scheme: https
    interval: 5s
    tlsConfig:
      insecureSkipVerify: true
    authorization:
      credentials: (1)
        key: "nifi_token"
        name: "nifi-authorization-secret"
        optional: false
      type: "Bearer"
    relabelings: (2)
      - sourceLabels:
          - __meta_kubernetes_pod_name
          - __meta_kubernetes_service_name
          - __meta_kubernetes_namespace
          - __meta_kubernetes_pod_container_port_number
        targetLabel: __address__
        replacement: ${1}.${2}.${3}.svc.cluster.local:${4}
        regex: (.+);(.+?)(?:-headless)?;(.+);(.+)
  selector:
    matchLabels:
      prometheus.io/scrape: "true"
  namespaceSelector:
    any: true
  jobLabel: app.kubernetes.io/instance
1 Authorization via Bearer Token stored in a secret
2 Relabel __address__ to be a FQDN rather then the IP-Address of target pod
As of Listener integration, SDP exposes a Service with -headless thus we need to regex this suffix.