Installation

Install the Stackable Spark operator as well as the commons, secret and listener operators which are required by all Stackable operators.

Dependencies

Spark applications almost always require dependencies like database drivers, REST api clients and many others. These dependencies must be available on the classpath of each executor (and in some cases of the driver, too). There are multiple ways to provision Spark jobs with such dependencies: some are built into Spark itself while others are implemented at the operator level. In this guide we are going to keep things simple and look at executing a Spark job that has a minimum of dependencies.

More information about the different ways to define Spark jobs and their dependencies is given on the following pages:

Stackable Operators

There are multiple ways to install the Stackable Operator for Apache Spark. stackablectl is the preferred way, but Helm is also supported. OpenShift users may prefer installing the operator from the RedHat Certified Operator catalog using the OpenShift web console.

  • stackablectl

  • Helm

stackablectl is the command line tool to interact with Stackable operators and our recommended way to install operators. Follow the installation steps for your platform.

After you have installed stackablectl run the following command to install the Spark operator:

stackablectl operator install \
  commons=24.11.1-rc2 \
  secret=24.11.1-rc2 \
  listener=24.11.1-rc2 \
  spark-k8s=24.11.1-rc2

The tool shows

Installed commons=24.11.1-rc2 operator
Installed secret=24.11.1-rc2 operator
Installed listener=24.11.1-rc2 operator
Installed spark-k8s=24.11.1-rc2 operator
Consult the Quickstart to learn more about how to use stackablectl. For example, you can use the --cluster kind flag to create a Kubernetes cluster with kind.

Add the Stackable Helm repository:

helm repo add stackable-stable https://repo.stackable.tech/repository/helm-stable/

Install the Stackable Operators:

helm install --wait commons-operator stackable-stable/commons-operator --version 24.11.1-rc2
helm install --wait secret-operator stackable-stable/secret-operator --version 24.11.1-rc2
helm install --wait listener-operator stackable-stable/listener-operator --version 24.11.1-rc2
helm install --wait spark-k8s-operator stackable-stable/spark-k8s-operator --version 24.11.1-rc2

Helm will deploy the operators in a Kubernetes Deployment and apply the CRDs for the SparkApplication (as well as the CRDs for the required operators).

What’s next

Execute a Spark Job and verify that it works by inspecting the pod logs.