Monitoring
In November 2024, Apache NiFi released a new major version 2.0.0
.
The NiFi 2.0.0
release changed the way of exposing Prometheus metrics significantly.
The following steps explain on how to expose Metrics in NiFi versions 1.x.x
and 2.x.x
.
Configure metrics in NiFi 1.x.x
For NiFi versions 1.x.x
, the operator automatically configures NiFi to export Prometheus metrics.
This is done by creating a Job that connects to NiFi and configures a Prometheus Reporting Task.
Network access from the Job to NiFi is required. If you are running a Kubernetes with restrictive NetworkPolicies, make sure to allow access from the Job to NiFi. |
See Monitoring for more details.
Disabling create-reporting-task Job
It can be helpful to disable the Job, e.g. when you configOverride an authentication mechanism, which the Job currently cannot use to authenticate against NiFi.
To achieve this use the following configuration:
spec:
clusterConfig:
createReportingTaskJob:
enabled: false
Configure metrics in NiFi 2.x.x
The Prometheus Reporting Task was removed in NiFi 2.x.x
in NIFI-13507.
Metrics are now always exposed and can be scraped using the NiFi Pod FQDN and the HTTP path /nifi-api/flow/metrics/prometheus
.
For a deployed single node NiFi cluster called simple-nifi
, containing a rolegroup called default
, the metrics endpoint is reachable under:
https://simple-nifi-node-default-0.simple-nifi-node-default.<namespace>.svc.cluster.local:8443/nifi-api/flow/metrics/prometheus
If NiFi is configured to do any user authentication, requests to the metric endpoint must be authenticated and authorized. |
Authentication with NiFi 2.x.x
=== The NiFi metrics endpoints are behind a strong authentication mechanism which require credentials for each individual pod. === |
To authenticate, you can use a bearer token created by your NiFi instance e.g.
curl -X POST https://simple-nifi-node-default-0.simple-nifi-node-default.<namespace>.svc.cluster.local:8443/nifi-api/access/token -d 'username=<user>&password=<password>' -k
where -k
equals verify=false
to allow self-signed certificates. The reply is your bearer token.
The following example shows how to configure the Prometheus scraper to use the bearer token to authenticate against a NiFi pod.
---
authorization: (1)
type: Bearer
credentials: "<Bearer Token>" (2)
tls_config:
insecure_skip_verify: true
static_configs:
- targets:
- '<pod>.<statefulset>.svc.cluster.local:8443' (3)
metrics_path: '/nifi-api/flow/metrics/prometheus'
scheme: https
1 | Use the authorization property instead if the basic_auth . |
2 | Add the previously obtained token here. |
3 | Static targets only scrapes one pod. |
or use it in a NiFi secret which should look like
---
apiVersion: v1
kind: Secret
metadata:
name: nifi-authorization-secret
type: Opaque
stringData:
nifi_token: "<Bearer_token>"
If you want to use a ServiceMonitor
you’d need to configure it as follows:
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: scrape-nifi2
labels:
stackable.tech/vendor: Stackable
release: prometheus
spec:
endpoints:
- port: https
path: 'nifi-api/flow/metrics/prometheus'
scheme: https
interval: 5s
tlsConfig:
insecureSkipVerify: true
authorization:
credentials: (1)
key: "nifi_token"
name: "nifi-authorization-secret"
optional: false
type: "Bearer"
relabelings: (2)
- sourceLabels:
- __meta_kubernetes_pod_name
- __meta_kubernetes_service_name
- __meta_kubernetes_namespace
- __meta_kubernetes_pod_container_port_number
targetLabel: __address__
replacement: ${1}.${2}.${3}.svc.cluster.local:${4}
regex: (.+);(.+?)(?:-metrics)?;(.+);(.+)
selector:
matchLabels:
prometheus.io/scrape: "true"
namespaceSelector:
any: true
jobLabel: app.kubernetes.io/instance
1 | Authorization via Bearer Token stored in a secret |
2 | Relabel __address__ to be a FQDN rather then the IP-Address of target pod |
As of Listener integration, SDP exposes a Service with -metrics thus we need to regex this suffix.
|