On this page you will install the Stackable Spark-on-Kubernetes operator as well as the commons, secret and listener operators which are required by all Stackable operators.
Spark applications almost always require dependencies like database drivers, REST api clients and many others. These
dependencies must be available on the
classpath of each executor (and in some cases of the driver, too). There are
multiple ways to provision Spark jobs with such dependencies: some are built into Spark itself while others are
implemented at the operator level. In this guide we are going to keep things simple and look at executing a Spark job
that has a minimum of dependencies.
More information about the different ways to define Spark jobs and their dependencies is given on the following pages:
There are 2 ways to install Stackable operators
Using a Helm chart
stackablectl is the command line tool to interact with Stackable operators and our recommended way to install
Operators. Follow the installation steps for your platform.
After you have installed
stackablectl run the following command to install the Spark-k8s operator:
stackablectl operator install \ commons=0.0.0-dev \ secret=0.0.0-dev \ listener=0.0.0-dev \ spark-k8s=0.0.0-dev
The tool will show
[INFO ] Installing commons operator [INFO ] Installing secret operator [INFO ] Installing listener operator [INFO ] Installing spark-k8s operator
You can also use Helm to install the operator. Add the Stackable Helm repository:
helm repo add stackable-dev https://repo.stackable.tech/repository/helm-dev/
Then install the Stackable Operators:
helm install --wait commons-operator stackable-dev/commons-operator --version 0.0.0-dev helm install --wait secret-operator stackable-dev/secret-operator --version 0.0.0-dev helm install --wait listener-operator stackable-dev/listener-operator --version 0.0.0-dev helm install --wait spark-k8s-operator stackable-dev/spark-k8s-operator --version 0.0.0-dev
Helm will deploy the operators in a Kubernetes Deployment and apply the CRDs for the
SparkApplication (as well as the
CRDs for the required operators). You are now ready to create a Spark job.