First steps

After going through the Installation section and having installed all the operators, you will now deploy a NiFi cluster and the required dependencies. Afterwards you can verify that it works by querying the REST API.

Setup

Two things need to be installed to create a NiFi cluster:

  • A ZooKeeper cluster for internal use by NiFi

  • The NiFi cluster itself

We will create them in this order, each one is created by applying a manifest file. The operators you just installed will then create the resources according to the manifest.

Apache ZooKeeper

To create a ZooKeeper instance run the following command:

kubectl apply -f - <<EOF
---
apiVersion: zookeeper.stackable.tech/v1alpha1
kind: ZookeeperCluster
metadata:
  name: simple-zk
spec:
  image:
    productVersion: 3.8.0
    stackableVersion: "23.4.0"
  servers:
    roleGroups:
      default:
        replicas: 3
EOF

Create a Znode object:

kubectl apply -f - <<EOF
---
apiVersion: zookeeper.stackable.tech/v1alpha1
kind: ZookeeperZnode
metadata:
  name: simple-nifi-znode
spec:
  clusterRef:
    name: simple-zk
EOF

The ZNode makes sure that the NiFi cluster will operate in its own separated directory in ZooKeeper.

Apache NiFi

The NiFi cluster requires authentication. Create a set of credentials for this purpose:

kubectl apply -f - <<EOF
---
apiVersion: v1
kind: Secret
metadata:
  name: nifi-admin-credentials-simple
stringData:
  username: admin
  password: admin
EOF

Finally create a NiFi instance:

kubectl apply -f - <<EOF
---
apiVersion: nifi.stackable.tech/v1alpha1
kind: NifiCluster
metadata:
  name: simple-nifi
spec:
  image:
    productVersion: 1.18.0
    stackableVersion: "23.4.0"
  clusterConfig:
    listenerClass: external-unstable
    authentication:
      method:
        singleUser:
          adminCredentialsSecret: nifi-admin-credentials-simple
          autoGenerate: true
    sensitiveProperties:
      keySecret: nifi-sensitive-property-key
      autoGenerate: true
    zookeeperConfigMapName: simple-nifi-znode
  nodes:
    roleGroups:
      default:
        replicas: 2
EOF

Verify that it works

First, make sure all pods are ready:

kubectl wait -l statefulset.kubernetes.io/pod-name=simple-nifi-node-default-0 \
--for=condition=ready pod --timeout=1200s && \
kubectl wait -l statefulset.kubernetes.io/pod-name=simple-nifi-node-default-1 \
--for=condition=ready pod --timeout=1200s

Then make sure the StatefulSets are ready:

kubectl get statefulset

The output should show all pods ready:

NAME                                 READY   AGE
simple-nifi-node-default             2/2     5m
simple-zk-server-default             3/3     7m

Congratulations! You successfully created your first NiFi cluster!

Access the NiFi web interface

You can retrieve the URL for the NiFi cluster web interface via stackablectl or kubectl.

stackablectl

Use the service command of stackablectl to get a list of all available endpoints:

stackablectl svc list

which should return something like this:

 PRODUCT    NAME         NAMESPACE  ENDPOINTS                       EXTRA INFOS

 nifi       simple-nifi  default    https https://172.18.0.3:32595

 zookeeper  simple-zk    default    zk    172.18.0.3:30173

You can also use the json output and parse the endpoint:

nifi_url=$(stackablectl svc list -o json | jq --raw-output '.nifi[0].endpoints.https')

Then connect to https://172.18.0.3:32595/nifi and you should see the NiFi web login. After providing the username admin and password admin you are redirected to the NiFi web interface.

nifi web ui

Via kubectl

Extracting the IP and port via kubectl is cumbersome. We recommend using stackablectl instead. The following kubectl commands store their output for further use in a variable and write its content to stdout afterwards. Make sure to run these commands in the same terminal:

nifi_node_name=$(kubectl get endpoints simple-nifi --output=jsonpath='{.subsets[0].addresses[0].nodeName}') && \
echo "NodeName: $nifi_node_name"

which should output a single node name where a NiFi pod is scheduled:

NodeName: kind-worker

Retrieve the IP of that node:

nifi_node_ip=$(kubectl get nodes -o jsonpath="{.items[?(@.metadata.name==\"$nifi_node_name\")].status.addresses[?(@.type==\"InternalIP\")].address}") && \
echo "NodeIp: $nifi_node_ip"

which should output the internal IP of that node:

NodeIp: 172.18.0.3

You might need to replace InternalIP with ExternalIP depending on how you connect to your Kubernetes cluster.

Finally, retrieve the NodePort of the simple-nifi service:

nifi_service_port=$(kubectl get service -o jsonpath="{.items[?(@.metadata.name==\"simple-nifi\")].spec.ports[?(@.name==\"https\")].nodePort}") && \
echo "NodePort: $nifi_service_port"

which should output the NodePort:

NodePort: 32595

Now build the full URL:

nifi_url="https://$nifi_node_ip:$nifi_service_port" && \
echo "NiFi web interface: $nifi_url"

which should output a URL to connect to the NiFi web interface:

NiFi web interface: https://172.18.0.3:32595

Then connect to https://172.18.0.3:32595/nifi and you should see the NiFi web login. After providing the username admin and password admin you are redirected to the NiFi web interface.

What’s next

Have a look at the Usage guide page to find out more about the features of the NiFi Operator.