Stackable Operator for Apache Spark

The operator manages two custom resource kinds: The SparkApplication and the SparkHistoryServer.

The SparkApplication resource is the main point of interaction with the operator. Unlike other Stackable operator custom resources, the SparkApplication does not have roles. An exhaustive list of options is given in the SparkApplication CRD reference .

The SparkHistoryServer has a single node role. It is used to deploy a Spark history server that displays application logs from S3 buckets. Of course, your applications need to write their logs to the same buckets.

For every SparkApplication deployed to the cluster the operator creates a Job, A ServiceAccout and a few ConfigMaps.

The Job runs spark-submit in a Pod which then creates a Spark driver Pod. The driver creates its own Executors based on the configuration in the SparkApplication. The Job, driver and executors all use the same image, which is configured in the SparkApplication resource.

The two main ConfigMaps are the <name>-driver-pod-template and <name>-executor-pod-template which define how the driver and executor Pods should be created.

The Spark history server deploys like other Stackable-supported applications: A Statefulset is created for every role group. A role group can have multiple replicas (Pods). A ConfigMap supplies the necessary configuration, and there is a service to connect to.

The Spark-Kubernetes RBAC documentation describes what is needed for spark-submit jobs to run successfully: minimally a role/cluster-role to allow the driver pod to create and manage executor pods.

However, to add security each spark-submit job launched by the operator will be assigned its own ServiceAccount.

During the operator installation, a cluster role named spark-k8s-clusterrole is created with pre-defined permissions.

When a new Spark application is submitted, the operator creates a new service account with the same name as the application and binds this account to the cluster role spark-k8s-clusterrole .