Release notes for the Stackable Data Platform

The Stackable platform consists of multiple operators that work together. Periodically a platform release is made, including all components of the platform at a specific version.

Release 24.7

New / extended platform features

Vulnerability management

In this release we have worked on significantly reducing the number of vulnerabilities in our product binaries. This will be more manageable going forward as we now build all Java-based binaries from source, which gives us greater flexibility when creating patches. We tackle this task at different levels:

  • bump base images (in this release we are using the UBI9 line of images in place of UBI8)

  • bump product versions to take advantage of product security improvements (see the section below for more details)

  • introduce our own patches (to e.g. bump individual dependencies or make changes to the code)

  • exclude product modules that are not relevant (e.g. build Hive to include only the Metastore and not the Hive server)

In this release we have eliminated 75% of all vulnerabilities that were present in the product binaries that were part of release 24.3.

This work will continue in the next version and the progress made in this release enables us to do more work in the future (workflow improvements, tooling etc.).

Build products from source

All Java-based product binaries are now built from source instead of packaging them from the official releases. The status is summarised below:

  • Apache Airflow: official release (Python-based)

  • Apache Druid: built from source from release 24.7

  • Apache HBase: built from source from release 24.3

  • Apache Hadoop HDFS: built from source from release 24.3

  • Apache Hive: built from source from release 24.7

  • Apache Kafka: built from source from release 24.7

  • Apache NiFi: built from source from release 24.7

  • Open Policy Agent: built from source from release 24.7

  • Trino: built from source from release 24.7

  • Apache Spark: built from source from release 24.7

  • Apache Superset: official release (Python-based)

  • Apache ZooKeeper: built from source from release 24.7

Multi-platform images

This release is the first multi-platform release of the Stackable Data Platform, supporting AMD64 and ARM64 architectures. Each image has a manifest list which wraps the architecture-specific image. The status is still experimental, as we work to fine-tune the necessary workflows.

Security

Support for OIDC with/without TLS has been added to Apache Druid in this release.

SDP now provides OIDC-support for Druid, Superset and Trino

In this release we provide experimental HBase 2.6.0 support with a new experimental policy based authorizer (with OPA). Check the documentation on HBase authorization with OPA but do not rely on its API as it might change in breaking ways during the experimental phase.

In an upcoming release we will enable authentication and encryption by default where possible. To ensure a smooth transition to future releases, we strongly encourage you to enable security features wherever possible in your deployments.
Documentation

  • Apache Hive and Trino operators: we have provided non-trivial sample Rego rules for these operators, together with an in-depth explanation and links

  • Apache Hive: there is now a tutorial on how to load and use external database drivers

  • Apache Spark: documentation has been added showing how to provision Spark dependencies

  • Open Policy Agent: N.B. As mentioned in the release 24.3, we will be actively building out the backends supported by the User Info Fetcher. This feature should be therefore be treated as experimental as we continue to extend and consolidate back-end handling and fine-tune the tool in general.

Other product features

The following are selected product features provided by new versions available in this release:

  • Apache Airflow: support for modularized DAGs

There is currently a known problem with using git-sync credentials in 24.7. This has been corrected in this PR and the fix is available in the nightly build and will be released in the next version.

  • Apache Druid: support for specifying and loading additional extensions

  • Apache HBase: support for exporting snapshots to S3. The HBase image now depends on the Hadoop image and the required AWS JARs are copied from there to the HBase image. See the documentation for more information.

  • Apache Hive: we now only supply the Hive Metastore. For most users this is an internal change, but is breaking for users with custom logging configurations

  • Listener operator: allow users to configure the external traffic policy, which is than passed to the created Service

  • Apache NiFi: support specifying the SecretClass that is used to obtain TLS certificates

  • Open Policy Agent: support enabling decision logs, this was an often requested feature and helps implement audit logging for authorization decisions

  • Secret operator: reduce CA default lifetime to one year and log when secrets are created

  • Trino: support for row filters and column masks in Rego rules

  • Apache ZooKeeper: allow the overriding of the ZNode path by setting status.znodePath

    Bugfixes

  • Apache Druid: move the DB credentials user and password out of the CRD into a secret containing the keys username and password

  • Apache Hive: move the metastore user and password DB credentials out of the CRD into a Secret containing the keys username and password

  • Apache Kafka: remove field/arg controller_config from kafka_controller::Ctx struct and create_controller function

  • Apache NiFi: use config-utils for text-replacement of variables in configs. This fixes escaping problems, especially when special characters are included in the password

  • Secret operator: for OpenShift clusters, the TLS CA Secret is now installed into the Namespace of the operator (typically stackable-operators), rather than default

  • Apache Spark: CPU resources are now applied correctly (instead of being rounding to the next whole number). This might affect existing jobs, as they may have e.g. only 200m CPU resources requested instead of the 1000m it had thus far, meaning they might slow down significantly

  • Apache Superset: admin credentials are not printed during startup

  • Trino: change the username which triggers graceful shutdown from admin to graceful-shutdown-user for greater clarity (e.g. in the Trino policies)

Product versions

As with previous SDP releases, many product images have been updated to their latest versions. The LTS version has in many cases also been adjusted in line with our support policy.

New versions

The following new product versions are now supported:

We ship Apache Hadoop 3.4.0 as a preview only and do NOT support upgrading from 3.3.x to 3.4.0 at the moment. Please test version 3.4.0 on fresh installations but do not attempt to upgrade to 3.4.0 if you are currently using 3.3.x.

Deprecated versions

The following product versions are deprecated and will be removed in a later release:

  • Apache Airflow: 2.6.3, 2.8.1, 2.8.4

  • Apache Druid: 28.0.1

  • Apache HBase: 2.4.17

  • Apache Kafka: 3.4.1, 3.6.1

  • Apache NiFi: 1.21.0

  • Open Policy Agent: 0.61.0

  • Apache Spark: 3.4.2, 3.4.3

  • Apache Superset: 2.1.3, 3.1.0, 3.1.3

  • Trino: 414, 442

  • Apache ZooKeeper: 3.8.4

N.B. in some cases a newly supported version is also immediately marked as deprecated. This is done to allow an update path from the latest patch of a minor version (e.g. Kafka 2.8.2 -→ 3.4.1).

Removed versions

The following product versions are no longer supported (although images for released product versions remain available here):

  • Apache Airflow: 2.7.2, 2.7.3

  • Apache Druid: 27.0.0

  • Apache Kafka: 3.5.2

  • Apache NiFi: 1.23.2

  • Open Policy Agent: 0.57.0

  • Apache Spark: 3.4.1, 3.5.0

  • Apache Superset: 2.1.1, 3.0.1, 3.0.3

  • Trino: 428

  • Apache ZooKeeper: 3.8.3

stackablectl

The following changes have been made to stackablectl:

  • a new experimental debug command

  • a pre-built binary for aarch64-unknown-linux-gnu is now available

  • complete error messages are now shown (remedying the truncation of some details in previous releases)

  • use of the latest Go and Rust versions and respective dependencies

Supported Kubernetes versions

This release supports the following Kubernetes versions:

  • 1.30

  • 1.29

  • 1.28

These Kubernetes versions are no longer supported:

  • 1.27

Supported OpenShift versions

This release is available in the RedHat Certified Operator Catalog for the following OpenShift versions:

  • 4.15

  • 4.14

  • 4.13

  • 4.12

These OpenShift versions are no longer supported:

  • 4.11

Breaking changes

Of the changes mentioned above, the following are breaking (or could lead to breaking behaviour), and you will need to adapt your existing CRDs accordingly:

Stackable Operator for Apache Druid

Breaking changes details

  • spec.metadataStorageDatabase.user: This field has been removed.

  • spec.metadataStorageDatabase.password: This field has been removed.

  • spec.metadataStorageDatabase.credentialsSecret: Name of the secret containing the credentials for the database (i.e. containing username and password fields).

Stackable Operator for Apache Hadoop

YARN, Map-reduce and other dependencies not needed by the HDFS operator have been removed from the 24.7.0 images. If these dependencies are needed - e.g. for distcp commands - then please use the older 24.3.0 images which still contain these libraries.

Stackable Operator for Apache Hive

Breaking changes details

  • spec.clusterConfig.database.user: This field has been removed.

  • spec.clusterConfig.database.password: This field has been removed.

  • spec.clusterConfig.database.credentialsSecret: Name of the secret containing the credentials for the database (i.e. containing username and password fields).

Stackable Secret Operator

Stackable Operator for Apache Spark

Stackable Operator for Trino

Upgrade from 24.3

Using stackablectl

Uninstall the 24.3 release

$ stackablectl release uninstall 24.3

Uninstalled release '24.3'

Use "stackablectl release list" to list available releases.
# ...

Afterwards you will need to upgrade the CustomResourceDefinitions (CRDs) installed by the Stackable Platform. The reason for this is that helm will uninstall the operators but not the CRDs. This can be done using kubectl replace.

The cluster name for the hello-world operator has been changed in this release so the CRD cannot be patched in-place. For this reason in the snipets below the CRD for this operator will be subject to a delete command (plus an apply as part of the operator rollout in the new release) instead of a replace. 
kubectl replace -f https://raw.githubusercontent.com/stackabletech/airflow-operator/24.7.0/deploy/helm/airflow-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/commons-operator/24.7.0/deploy/helm/commons-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/druid-operator/24.7.0/deploy/helm/druid-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/hbase-operator/24.7.0/deploy/helm/hbase-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/hdfs-operator/24.7.0/deploy/helm/hdfs-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/hive-operator/24.7.0/deploy/helm/hive-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/kafka-operator/24.7.0/deploy/helm/kafka-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/listener-operator/24.7.0/deploy/helm/listener-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/nifi-operator/24.7.0/deploy/helm/nifi-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/opa-operator/24.7.0/deploy/helm/opa-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/secret-operator/24.7.0/deploy/helm/secret-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/spark-k8s-operator/24.7.0/deploy/helm/spark-k8s-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/superset-operator/24.7.0/deploy/helm/superset-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/trino-operator/24.7.0/deploy/helm/trino-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/zookeeper-operator/24.7.0/deploy/helm/zookeeper-operator/crds/crds.yaml
# N.B. due to change of name
kubectl delete -f https://raw.githubusercontent.com/stackabletech/hello-world-operator/24.3.0/deploy/helm/hello-world-operator/crds/crds.yaml
customresourcedefinition.apiextensions.k8s.io "airflowclusters.airflow.stackable.tech" replaced
customresourcedefinition.apiextensions.k8s.io "airflowdbs.airflow.stackable.tech" replaced
customresourcedefinition.apiextensions.k8s.io "authenticationclasses.authentication.stackable.tech" replaced
customresourcedefinition.apiextensions.k8s.io "s3connections.s3.stackable.tech" replaced
...

If relevant, copy secrets to the operator’s namespace. E.g. for secret-provisioner-tls-ca:

kubectl get secrets secret-provisioner-tls-ca --output=yaml | \
    sed 's/namespace: .*/namespace: stackable-operators/' | \
    kubectl create --filename=-

Install the 24.7 release

$ stackablectl release install 24.7

Installed release '24.7'

Use "stackablectl operator installed" to list installed operators.

Using Helm

Use helm list to list the currently installed operators.

You can use the following command to uninstall all operators that are part of the 24.3 release:

$ helm uninstall airflow-operator commons-operator druid-operator hbase-operator hdfs-operator hello-world-operator hive-operator kafka-operator listener-operator nifi-operator opa-operator secret-operator spark-k8s-operator superset-operator trino-operator zookeeper-operator
release "airflow-operator" uninstalled
release "commons-operator" uninstalled
...

Afterward you will need to upgrade the CustomResourceDefinitions (CRDs) installed by the Stackable Platform. The reason for this is that helm will uninstall the operators but not the CRDs. This can be done using kubectl replace:

kubectl replace -f https://raw.githubusercontent.com/stackabletech/airflow-operator/24.7.0/deploy/helm/airflow-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/commons-operator/24.7.0/deploy/helm/commons-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/druid-operator/24.7.0/deploy/helm/druid-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/hbase-operator/24.7.0/deploy/helm/hbase-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/hdfs-operator/24.7.0/deploy/helm/hdfs-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/hive-operator/24.7.0/deploy/helm/hive-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/kafka-operator/24.7.0/deploy/helm/kafka-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/listener-operator/24.7.0/deploy/helm/listener-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/nifi-operator/24.7.0/deploy/helm/nifi-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/opa-operator/24.7.0/deploy/helm/opa-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/secret-operator/24.7.0/deploy/helm/secret-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/spark-k8s-operator/24.7.0/deploy/helm/spark-k8s-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/superset-operator/24.7.0/deploy/helm/superset-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/trino-operator/24.7.0/deploy/helm/trino-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/zookeeper-operator/24.7.0/deploy/helm/zookeeper-operator/crds/crds.yaml
# N.B. due to change of name
kubectl delete -f https://raw.githubusercontent.com/stackabletech/hello-world-operator/24.3.0/deploy/helm/hello-world-operator/crds/crds.yaml

If relevant, copy secrets to the operator’s namespace. E.g. for secret-provisioner-tls-ca:

kubectl get secrets secret-provisioner-tls-ca --output=yaml | \
    sed 's/namespace: .*/namespace: stackable-operators/' | \
    kubectl create --filename=-

Install the 24.7 release

helm repo add stackable-stable https://repo.stackable.tech/repository/helm-stable/
helm repo update stackable-stable
helm install --wait airflow-operator stackable-stable/airflow-operator --version 24.7.0
helm install --wait commons-operator stackable-stable/commons-operator --version 24.7.0
helm install --wait druid-operator stackable-stable/druid-operator --version 24.7.0
helm install --wait hbase-operator stackable-stable/hbase-operator --version 24.7.0
helm install --wait hdfs-operator stackable-stable/hdfs-operator --version 24.7.0
helm install --wait hive-operator stackable-stable/hive-operator --version 24.7.0
helm install --wait kafka-operator stackable-stable/kafka-operator --version 24.7.0
helm install --wait listener-operator stackable-stable/listener-operator --version 24.7.0
helm install --wait hello-world-operator stackable-stable/hello-world-operator --version 24.7.0
helm install --wait nifi-operator stackable-stable/nifi-operator --version 24.7.0
helm install --wait opa-operator stackable-stable/opa-operator --version 24.7.0
helm install --wait secret-operator stackable-stable/secret-operator --version 24.7.0
helm install --wait spark-k8s-operator stackable-stable/spark-k8s-operator --version 24.7.0
helm install --wait superset-operator stackable-stable/superset-operator --version 24.7.0
helm install --wait trino-operator stackable-stable/trino-operator --version 24.7.0
helm install --wait zookeeper-operator stackable-stable/zookeeper-operator --version 24.7.0

Known upgrade issues

In the case of the breaking changes detailed above it will be necessary to update the custom resources and re-apply them.

Copy any secrets from the default namespace to that used by the operator, as shown above.

Please note that Java-based products that use the JMX exporter for Prometheus are now built with version 1.0.1, which has breaking changes relevant for any dashboards displaying JVM metrics.

Apache Nifi

To upgrade from 1.27 to the 2.x series, the following config overrides are necessary in the new cluster’s manifest:

spec:
  image:
    productVersion: 2.0.0-M4
  nodes:
    configOverrides:
      nifi.properties:
        nifi.flow.configuration.file: /stackable/data/database/flow.json.gz
Do not override this property for the 1.27 cluster version.

This is necessary because the 2.x versions do not support the XML format for flow definitions anymore. Support for the JSON format has been addded in version 1.16 and both formats have been maintained up to (excluding) version 2.0. The next SDP release 24.11 will automatically take care of this step for you.

Release 24.3

New / extended platform features

Support for many new product versions

Almost all product images have been updated to their latest versions. Some notable examples are Apache Airflow 2.8.1, Trino 442 and Apache Spark 3.5.1. In addition, we started building some product binaries from source instead of packaging them from the official releases. This enables greater control over the features and the security aspects of the products. Currently, Apache Hadoop and Apache HBase are built from source and others will follow in coming releases.

Security

We put a special emphasis on security in this release. For this purpose we addressed the following topics:

  • Authorization

  • Authentication with Kerberos

  • OpenID Connect integration

  • Vulnerability management

Starting with this release, user authorization is consistent across all products. The Open Policy Agent (OPA) has been a core component of the platform since the beginning and in this release we enhanced its capability with the inclusion of a new component called user-info-fetcher. It allows for authorization policies to be built on many attributes like organizational group membership, resource assignment and much more. The first major identity provider supported by the user-info-fetcher is Keycloak but others will follow. For more information, refer to the User Info Fetcher Usage Guide.

A lot of effort was spent on enabling policy based authorization (with OPA) within HDFS. This was in important milestone in the platform evolution and is a unique feature that has long been missed by HDFS users and administrators. It requires a Kerberos-enabled cluster as well as the HDFS extension which includes an OPA authorizer and group mapper. This is already bundled in Stackable image of HDFS and can be used by following the documentation. More details are available in the pull request.

Kerberos is the most widely used authentication protocol in the enterprise world. We added support for it in Apache Hive and Apache HBase. We also added examples for running Apache Spark applications in a Kerberos enabled environment.

OpenID Connect is the de-facto authorization standard on the Web, and it’s making its way into enterprise environments. Our platform now supports it for Apache Superset (documentation) and Trino (documentation), others will follow.

A core component of SDP is the Secret Operator. In this release, the secret operator will automatically rotate certificates it generates. The certificate lifetime is also jittered to avoid all pods restarting around the same time.

This is also the first release we’re publishing SBOMs (Software Bill of Materials) in the CycloneDX format for. We publish these for both our operators and our product images. The work on this was partially funded by the Sovereign Tech Fund. These SBOMS are published to our OCI registry as signed in-toto attestations and we have written a guide on how you can use it. To make it easier to use we have written a SBOM browser which allows you to download the raw CycloneDX JSON files as well.

In an upcoming release we will enable authentication and encryption by default where possible. To ensure a smooth transition to future releases, we strongly encourage you to enable security features wherever possible in your deployments.
Storage

HDFS deployments now support rack awareness. This is another unique feature that brings the SDP platform closer to feature parity with bare metal HDFS deployments. Of course, the exact meaning of rack is different in Kubernetes environments, but the effect is the same: DataNodes are brought closer to the data they are reading and writing thus improving performance and reliability. A new topology provider is bundled with the HDFS image that maps Kubernetes labels to a cluster topology.

Documentation

We are constantly working on improving the platform documentation and custom resource definitions are a significant part of that. The CRD documentation is now generated automatically and can be found at https://crds.stackable.tech.

Command line tools

The stackablectl command line tool has been overhauled and can now list endpoints provided by the listener operator. Also operator installation is parallelized, which considerably speeds up the process of setting up SDP on fresh Kubernetes clusters.

Custom labels for Helm charts

In the past, Helm users could not assign custom labels to stacklets. This is now possible and it enables better component management with third party tools.

Bugfixes

  • Apache Airflow Operator: Using git-sync with the KubernetesExecutor is now possible.

  • Apache Hadoop Operator:

    • Kerberos principals are now included in the discovery ConfigMap.

    • Environment variables can now be overridden with the role group’s envOverrides property.

  • Apache Spark Operator:

    • Applications can now be provisioned dynamically without having to fiddle with classpath settings.

    • RBAC permissions have been updated to allow the deletion of ConfigMaps when cleaning up applications.

  • Trino Operator: Add HDFS configuration files to the hive.config.resources property when connecting to a HDFS cluster.

Product features

The following are selected product features provided by new versions available in this release:

  • Apache Airflow: Introducing Airflow Object Storage and Listener hooks for Datasets plus various bug fixes.

  • Apache Druid: SQL compliance & engine enhancements, ingestion improvements, concurrent data handling.

  • Apache Kafka: Bug fixes.

  • Apache NiFi:

    • Improvements and bugfixes.

    • Over 270 issues fixed since version 1.23.2.

    • Adds new components for Slack and Zendesk integration among others.

  • Open Policy Agent:

    • Performance improvements, bugfixes and security fixes for third-party libraries.

    • Tooling to help prepare existing policies for the upcoming OPA 1.0 release, which will include a new version of the Rego language.

  • Apache Spark:

    • 3.5.0 (SDP ships with 3.5.1) added first class support for distributed training and inference support, and enhancement of compatibility for Structured streaming, Spark SQL improvements.

    • Spark Connect is not yet supported in this release.

  • Apache Superset:

    • Latest patch release for the Superset 2.x lineage.

    • Apache Superset 3.1 includes various smaller new features/optimizations e.g. waterfall chart visualization, ECharts bubble chart, improved data set selectors, automatically format SQL queries, and country map visualization improvements.

  • Trino:

    • Lots of improvements and optimization since release 428.

    • Most notably we would like to highlight support for access control with the Open Policy Agent that we ourselves contributed in release 438 (#19532).

    • Also, starting from release 440, there is now row filtering and column masking in Open Policy Agent.

  • Apache ZooKeeper: Security and bug fixes.

Support for the ARM architecture

During the development of this release, we started introducing support for the arm64 architecture. Currently support is experimental, and we only provide arm64 images for the previous release (23.11). For more information on how to use the ARM images, refer to the documentation.

Product versions

New versions

The following new product versions are now supported:

Deprecated versions

The following product versions are deprecated and will be removed in a later release:

  • Apache Airflow: 2.7.2, 2.7.3

  • Apache Druid: 27.0.0

  • Apache Kafka: 3.5.1, 3.5.2, 3.6.2

  • Apache NiFi: 1.23.2

  • Apache Spark: 3.4.1, 3.5.0

  • Apache Superset: 2.1.1, 3.0.1. 3.0.3

  • Trino: 428

  • Apache ZooKeeper: 3.8.3

  • Open Policy Agent: 0.57.0

N.B. in some cases a newly supported version is also immediately marked as deprecated. This is done to allow an update path from the latest patch of a minor version (e.g. Kafka 2.8.2 -→ 3.4.1).

Removed versions

The following product versions are no longer supported (although images for released product versions remain available here):

  • Apache Airflow: 2.6.1, 2.6.3

  • Apache Druid: 27.0.0

  • Apache Hadoop: 3.2.2, 3.2.4

  • Apache HBase: 2.4.12

  • Apache Kafka: 2.8.1, 2.8.2, 3.4.1

  • Open Policy Agent: 0.51.0

  • Apache Spark: 3.4.0

  • Apache Superset: 2.1.0

  • Apache ZooKeeper: 3.8.1

Cockpit and stackablectl

A new project called Stackable Cockpit has been started. It is a web-based management tool that allows users to interact with the Stackable data platform. The repository also contains the stackablectl command line tool, which has been refactored for performance and stability.

Supported Kubernetes versions

This release supports the following Kubernetes versions:

  • 1.29

  • 1.28

  • 1.27

These Kubernetes versions are no longer supported:

  • 1.26

  • 1.25

Supported OpenShift versions

This release is available in the RedHat Certified Operator Catalog for the following OpenShift versions:

  • 4.15

  • 4.14

  • 4.13

  • 4.12

  • 4.11

Breaking changes

You will need to adapt your existing CRDs due to the following breaking changes detailed below.

Stackable Operator for Apache Airflow

Breaking changes details

The following fields used to be optional but are now mandatory: * spec.clusterConfig.credentialsSecret: Name of the secret containing the credentials for the database. * spec.clusterConfig.exposeConfig: Set to true to export the AIRFLOWWEBSERVEREXPOSE__CONFIG environment variable. * spec.clusterConfig.loadExamples: Set to true to load example DAGs into the Airflow cluster.

Breaking changes details

Assigning role groups to node selector was deprecated in 23.11 and has been removed in this release.

To assign roles and role groups to nodes, use the config.affinity.nodeAffinity property as described in the documentation.

Apache HBase

Breaking changes details

Assigning role groups to node selector was deprecated in SDP 23.11 and has been removed in this release.

To assign roles and role groups to nodes, use the config.affinity.nodeAffinity property as described in the documentation.

Stackable Operator for Apache Hadoop HDFS

Breaking changes details

This requires a change from e.g.

apiVersion: hdfs.stackable.tech/v1alpha1
kind: HdfsCluster
metadata:
  name: hdfs
spec:
  clusterConfig:
    listenerClass: external-unstable (1)
    ...

to:

apiVersion: hdfs.stackable.tech/v1alpha1
kind: HdfsCluster
metadata:
  name: hdfs
spec:
  clusterConfig:
    ...
  nameNodes:
    config:
      listenerClass: external-unstable (2)
      ...
    ...
  dataNodes:
    config:
      listenerClass: external-unstable (3)
      ...
  journalNodes:
    config:
      ...
1 Remove the cluster-wide listenerClass
2 Add the external-unstable listenerClass to the nameNodes role. You can set these at the role-group level too.
3 Add the external-unstable listenerClass to the dataNodes role. You can set these at the role-group level too.

It should be noted that this change is not necessary if you are using the default spec.clusterConfig.listenerClass: cluster-internal.

Unfortunately, it is not possible to patch existing HDFS stacklets in place. It will be necessary to delete and recreate the HDFS stacklet. No data will be lost during this process.

Open Policy Agent Operator

Breaking changes details

Assigning role groups to node selector was deprecated in SDP 23.11 and has been removed in this release.

To assign roles and role groups to nodes, use the config.affinity.nodeAffinity property as described in the documentation.

Secret operator

This changes the format of the CA secrets. Old secrets will be migrated automatically, but manual intervention will be required to downgrade back to 23.11.x.

Stackable Operator for Apache Spark

Breaking changes details

The spec.version field has been removed.

The spec.mode field is now required and must be set to cluster.

The spec.mainClass field is now required and must point to a location on ths file system or S3 where the main class is located.

Breaking changes details

This is an experimental feature that was introduced to support logging in XML format. The side effect of this removal is that the vector agent cannot aggregate output from the spark-submit containers. On the other hand, it enables dynamic provisioning of java packages (such as Delta Lake) with Stackable stock images, which we consider more important.

Stackable Operator for Apache Superset

The configuration for the Superset authentication, operations and listener class is specified within the spec.clusterConfig field. This field used to be optional but it is now required.

Upgrade from 23.11

Using stackablectl

Uninstall the 23.11 release

$ stackablectl release uninstall 23.11
[INFO ] Uninstalling release 23.11
[INFO ] Uninstalling airflow operator
[INFO ] Uninstalling commons operator
# ...

Afterwards you will need to upgrade the CustomResourceDefinitions (CRDs) installed by the Stackable Platform. The reason for this is that helm will uninstall the operators but not the CRDs. This can be done using kubectl replace:

kubectl replace -f https://raw.githubusercontent.com/stackabletech/airflow-operator/24.3.0/deploy/helm/airflow-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/commons-operator/24.3.0/deploy/helm/commons-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/druid-operator/24.3.0/deploy/helm/druid-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/hbase-operator/24.3.0/deploy/helm/hbase-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/hdfs-operator/24.3.0/deploy/helm/hdfs-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/hello-world-operator/24.3.0/deploy/helm/hello-world-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/hive-operator/24.3.0/deploy/helm/hive-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/kafka-operator/24.3.0/deploy/helm/kafka-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/listener-operator/24.3.0/deploy/helm/listener-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/nifi-operator/24.3.0/deploy/helm/nifi-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/opa-operator/24.3.0/deploy/helm/opa-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/secret-operator/24.3.0/deploy/helm/secret-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/spark-k8s-operator/24.3.0/deploy/helm/spark-k8s-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/superset-operator/24.3.0/deploy/helm/superset-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/trino-operator/24.3.0/deploy/helm/trino-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/zookeeper-operator/24.3.0/deploy/helm/zookeeper-operator/crds/crds.yaml
customresourcedefinition.apiextensions.k8s.io "airflowclusters.airflow.stackable.tech" replaced
customresourcedefinition.apiextensions.k8s.io "airflowdbs.airflow.stackable.tech" replaced
customresourcedefinition.apiextensions.k8s.io "authenticationclasses.authentication.stackable.tech" replaced
customresourcedefinition.apiextensions.k8s.io "s3connections.s3.stackable.tech" replaced
...

Install the 24.3 release

$ stackablectl release install 24.3
[INFO ] Installing release 23.11
[INFO ] Installing airflow operator in version 24.3.0
[INFO ] Installing commons operator in version 24.3.0
[INFO ] Installing druid operator in version 24.3.0
[INFO ] Installing hbase operator in version 24.3.0
[INFO ] Installing hdfs operator in version 24.3.0
[INFO ] Installing hive operator in version 24.3.0
[INFO ] Installing kafka operator in version 24.3.0
[INFO ] Installing listener operator in version 24.3.0
[INFO ] Installing hello-world operator in version 24.3.0
[INFO ] Installing nifi operator in version 24.3.0
[INFO ] Installing opa operator in version 24.3.0
[INFO ] Installing secret operator in version 24.3.0
[INFO ] Installing spark-k8s operator in version 24.3.0
[INFO ] Installing superset operator in version 24.3.0
[INFO ] Installing trino operator in version 24.3.0
[INFO ] Installing zookeeper operator in version 24.3.0

Using Helm

Use helm list to list the currently installed operators.

You can use the following command to uninstall all operators that are part of the 23.11 release:

$ helm uninstall airflow-operator commons-operator druid-operator hbase-operator hdfs-operator hive-operator kafka-operator listener-operator nifi-operator opa-operator secret-operator spark-k8s-operator superset-operator trino-operator zookeeper-operator
release "airflow-operator" uninstalled
release "commons-operator" uninstalled
# ...

Afterward you will need to upgrade the CustomResourceDefinitions (CRDs) installed by the Stackable Platform. The reason for this is that helm will uninstall the operators but not the CRDs. This can be done using kubectl replace:

kubectl replace -f https://raw.githubusercontent.com/stackabletech/airflow-operator/24.3.0/deploy/helm/airflow-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/commons-operator/24.3.0/deploy/helm/commons-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/druid-operator/24.3.0/deploy/helm/druid-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/hbase-operator/24.3.0/deploy/helm/hbase-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/hdfs-operator/24.3.0/deploy/helm/hdfs-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/hello-world-operator/24.3.0/deploy/helm/hello-world-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/hive-operator/24.3.0/deploy/helm/hive-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/kafka-operator/24.3.0/deploy/helm/kafka-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/listener-operator/24.3.0/deploy/helm/listener-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/nifi-operator/24.3.0/deploy/helm/nifi-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/opa-operator/24.3.0/deploy/helm/opa-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/secret-operator/24.3.0/deploy/helm/secret-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/spark-k8s-operator/24.3.0/deploy/helm/spark-k8s-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/superset-operator/24.3.0/deploy/helm/superset-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/trino-operator/24.3.0/deploy/helm/trino-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/zookeeper-operator/24.3.0/deploy/helm/zookeeper-operator/crds/crds.yaml

Install the 24.3 release

helm repo add stackable-stable https://repo.stackable.tech/repository/helm-stable/
helm repo update stackable-stable
helm install --wait airflow-operator stackable-stable/airflow-operator --version 24.3.0
helm install --wait commons-operator stackable-stable/commons-operator --version 24.3.0
helm install --wait druid-operator stackable-stable/druid-operator --version 24.3.0
helm install --wait hbase-operator stackable-stable/hbase-operator --version 24.3.0
helm install --wait hdfs-operator stackable-stable/hdfs-operator --version 24.3.0
helm install --wait hive-operator stackable-stable/hive-operator --version 24.3.0
helm install --wait kafka-operator stackable-stable/kafka-operator --version 24.3.0
helm install --wait listener-operator stackable-stable/listener-operator --version 24.3.0
helm install --wait hello-world-operator stackable-stable/hello-world-operator --version 24.3.0
helm install --wait nifi-operator stackable-stable/nifi-operator --version 24.3.0
helm install --wait opa-operator stackable-stable/opa-operator --version 24.3.0
helm install --wait secret-operator stackable-stable/secret-operator --version 24.3.0
helm install --wait spark-k8s-operator stackable-stable/spark-k8s-operator --version 24.3.0
helm install --wait superset-operator stackable-stable/superset-operator --version 24.3.0
helm install --wait trino-operator stackable-stable/trino-operator --version 24.3.0
helm install --wait zookeeper-operator stackable-stable/zookeeper-operator --version 24.3.0

Known upgrade issues

In the case of the breaking changes detailed above it will be necessary to update the custom resources and re-apply them.

Additionally, please note the following:

All operators

If you used node selectors to assign Pods to nodes, this will not work anymore. Use the config.affinity.nodeAffinity property instead and follow the instructions in the documentation.

Release 23.11

This release introduces two further elements of platform stability - pod disruption budgets and graceful shutdown specifications - as well as a new management tool and updated product version support. Any known issues related to this release are listed in the Known issues section below.

New / extended platform features

The following new major platform features were added:

PodDisruptionBudgets

Kubernetes has mechanisms to ensure minimal planned downtime. Our product operators deploy so-called PodDisruptionBudget (PDB) resources alongside the products. For every role that you specify (e.g. HDFS namenodes or Trino workers) a PDB is created. This will determine the extent to which roles for a given application may be inactive at any given time. See the documentation for more details.

Graceful shutdown

Graceful shutdown refers to the managed, controlled shutdown of service instances in the manner intended by the software authors. Typically, an instance will receive a signal indicating the intent for the server to shut down, and it will initiate a controlled shutdown. Our operators configure a sensible amount of time Pods are granted to properly shut down without disrupting the availability of the product. See the documentation for more details.

Signed SDP product images

As of this release all Stackable product images are signed (the signing of operator images was delivered in SDP 23.7). Please see this tutorial for more information.

Airflow KubernetesExecutor

Airflow clusters can now be configured to use Kubernetes executors, whereby pods are spun up for job tasks and terminated when complete, thus offering an alternative way to use resources without the need for job queuing.

Overridable Java security settings

For JVM-based products (i.e. Druid, HBase, HDFS, Hive, Kafka, NiFi, Spark, Trino and ZooKeeper) it is now possible to provide custom security settings that override the default values. This allows the user to control things such as DNS lookup caches.

Stackable Cockpit

This release includes a very early preview version of Stackable Cockpit, a browser-based management tool which interacts with the Stackable data platform to display e.g. deployed stacklets and their status.

stackablectl

Our command line tool has been re-worked to use the same backbone as Stackable Cockpit: you can find out about the recent enhancements by visiting the online documentation.

Listener operator

The listener-operator was introduced in release 23.1 and the associated ServiceType field in 23.4. In this release we introduce configurable ListenerClass presets that map to the service types appropriate for different environments. This is discussed in more detail in the documentation.

Openshift certification

All Stackable operators in the 23.11 release have been certified for Openshift versions 4.11-4.13 and can be installed directly from the OperatorHub UI: OperatorHub in Openshift portal

Product features

Additionally, there are some other individual product features that are noteworthy:

Product Versions

New Versions

The following new product versions are now supported:

Deprecated Versions

The following product versions are deprecated and will be removed in a later release:

  • Airflow: 2.6.1

  • HBase: 2.4.12

  • HDFS: 3.2.4, 3.3.4

  • Kafka: 2.8.2, 3.4.0

  • OpenPolicyAgent: 0.51.0

  • Spark: 3.4.0

  • Superset: 2.1.0

  • ZooKeeper: 3.8.1

N.B. in some cases a newly supported version is also immediately marked as deprecated. This is done to allow an update path from the latest patch of a minor version (e.g. Kafka 2.8.2 -→ 3.4.1).

Removed Versions

The following product versions are no longer supported (although images for released product versions remain available here):

  • Airflow: 2.2.3, 2.2.4, 2.2.5, 2.4.1

  • Druid: 0.23.0, 24.0.0

  • HBase: 2.4.6, 2.4.8, 2.4.9, 2.4.11

  • HDFS: 3.3.1, 3.3.3

  • Hive: 2.3.9

  • Kafka: 2.7.1, 3.1.0, 3.2.0, 3.3.1

  • NiFi: 1.15.x, 1.16.x, 1.18.0, 1.20.0

  • Opa: 0.27.1, 0.28.0, 0.37.2, 0.41.0, 0.45.0

  • Spark: 3.2.1, 3.3.0

  • Superset: 1.3.2, 1.4.1, 1.4.2, 1.5.1, 1.5.3, 2.0.1

  • Trino: 377, 387, 395, 396, 403

  • ZooKeeper: 3.5.8, 3.6.3, 3.7.0, 3.8.0

stackablectl

The following demo has been added to stackablectl:

Signal processing demo

This demonstrates the ingestion of streamed data into a Timescale time-series database, augmented by a moving window of anomaly detection measurements that are plotted alongside the raw data in Grafana.

This link lists the available demos.

Supported Kubernetes versions

This release supports the following Kubernetes versions:

  • 1.28

  • 1.27

  • 1.26

These Kubernetes versions are no longer supported:

  • 1.25

  • 1.24

Supported OpenShift versions

This release supports the following OpenShift versions:

  • 4.13

  • 4.12

  • 4.11

Breaking changes

You will need to adapt your existing CRDs due to the following breaking changes detailed below.

For all operators: the field spec.image.stackableVersion is no longer needed when the operator and product stackable versions match e.g. both are 23.11. If this field has been previously set this will not happen: in such cases spec.image.stackableVersion should be set explicitly, either to a release version or to null.

Stackable Operator for Apache Airflow

  • Removed AirflowDB .spec.clusterConfig.databaseInitialization was removed from the CRD. This allowed configuring logging for the database initialization job, which does not exist anymore. Instead, database initialization is done by the scheduler pod, which has its own logging configuration. This was necessary to remove the AirflowDB Custom Resource, which caused problems when upgrading or reinstalling Airflow clusters. Additionally, the AirflowDB custom resource is not used anymore. Those CRs should be deleted.

  • Implement KubernetesExecutor As an alternative to the CeleryExecutor we now also support the KubernetesExecutor, whereby pods are spun up for job tasks and are terminated afterwards. This removes the need for a Redis job queue and offers an alternative approach to resource management.

  • Rename service port name The service port name has been renamed from airflow to http for consistency reasons. This change requires that the statefulset be removed before upgrading. There might also be some e.g. Ingresses that rely on the port name that will need to be updated.

  • AuthenticationClass references spec.clusterConfig.authenticationConfig has been consolidated to spec.clusterConfig.authentication which takes a vector of AuthenticationClass references.

These consolidated changes would require a change from e.g.

Breaking changes details 
apiVersion: airflow.stackable.tech/v1alpha1
kind: AirflowCluster
metadata:
  name: airflow
spec:
  image:
    productVersion: "2.6.1"
    stackableVersion: "23.7" (1)
  clusterConfig:
    executor: CeleryExecutor (2)
    ...
    databaseInitialization: (3)
      logging:
        enableVectorAgent: false
        containers:
          ...
    authenticationConfig:  (4)
      authenticationClass: server-tls
      userRegistrationRole: Admin
  webservers:
    ...
  workers:  (5)
    roleGroups:
      default:
        replicas: 1
    ...

to:

apiVersion: airflow.stackable.tech/v1alpha1
kind: AirflowCluster
metadata:
  name: airflow
spec:
  image:
    productVersion: "2.6.1"
  clusterConfig:
    authentication:  (4)
      - authenticationClass: server-tls
        userRegistrationRole: Admin
  webservers:
    roleGroups:
      default:
        replicas: 1
  celeryExecutors:  (5)
    roleGroups:
      default:
        replicas: 1
    ...
1 this field is no longer needed if the product and operator image versions match
2 field removed
3 section removed
4 authenticationConfig replaced with a list of authentication classes
5 workers replaced with either celeryExecutors or kubernetesExecutors
it will be necessary to remove the stateful sets before updating the Airflow resource due to the change to the name of container port. Any existing AirflowDB jobs should be deleted as well. This will allow a database update to be followed through where necessary.

Stackable Operator for Apache HDFS

Breaking changes details

This requires a change from e.g.

apiVersion: hdfs.stackable.tech/v1alpha1
kind: HdfsCluster
metadata:
  name: hdfs
spec:
  image:
    productVersion: "3.3.4"
    stackableVersion: "23.7" (1)
  clusterConfig:
    zookeeperConfigMapName: hdfs-zk
    autoFormatFs: False (2)
    ...

to:

apiVersion: hdfs.stackable.tech/v1alpha1
kind: HdfsCluster
metadata:
  name: hdfs
spec:
  image:
    productVersion: "3.3.4"
  clusterConfig:
    zookeeperConfigMapName: hdfs-zk
    ...
1 this field is no longer needed if the product and operator image versions match.
2 field removed.

Stackable Operator for Apache Kafka

  • Certificate conversion The secret-operator now handles certificate conversion. This allows for the removal of the prepare init container but means that you can’t configure the log level for this container anymore. You will need to remove the field spec.brokers.config.logging.container.prepare in case it is specified.

Breaking changes details

This requires a change from e.g.

apiVersion: kafka.stackable.tech/v1alpha1
kind: KafkaCluster
metadata:
  name: kafka
spec:
  image:
    productVersion: "3.4.0"
    stackableVersion: "23.7" (1)
  clusterConfig:
    ...
  brokers:
    config:
      logging:
        containers:
          prepare: (2)
            console:
              level: INFO
        ...

to:

apiVersion: kafka.stackable.tech/v1alpha1
kind: KafkaCluster
metadata:
  name: kafka
spec:
  image:
    productVersion: "3.4.0"
  clusterConfig:
    ...
  brokers:
    config:
      logging:
        ...
1 this field is no longer needed if the product and operator image versions match.
2 section removed for prepare container.
for details about how Kafka uses a PVC to persist a reference to its ZNode, and how this may be relevant to upgrade scenarios, please read the documentation here.

Stackable Operator for Apache NiFi

Breaking changes details

This requires a change from e.g.

---
apiVersion: nifi.stackable.tech/v1alpha1
kind: NifiCluster
metadata:
  name: test-nifi
spec:
  image:
    productVersion: "1.21.0"
    stackableVersion: "23.7" (1)
  clusterConfig:
    zookeeperConfigMapName: test-zk
    authentication: (2)
      allowAnonymousAccess: False  (3)
      method:
        singleUser:
          adminCredentialsSecret: nifi-admin-credentials-simple
          autoGenerate: False  (4)
    sensitiveProperties:
      keySecret: nifi-sensitive-property-key
  nodes:
    roleGroups:
      default:
        replicas: 1

to:

---
apiVersion: nifi.stackable.tech/v1alpha1
kind: NifiCluster
metadata:
  name: test-nifi
spec:
  image:
    productVersion: "1.23.2"
  clusterConfig:
    zookeeperConfigMapName: test-zk
    authentication: (2)
      - authenticationClass: nifi-users  (5)
    sensitiveProperties:
      keySecret: nifi-sensitive-property-key
  nodes:
    roleGroups:
      default:
        replicas: 1
---
apiVersion: authentication.stackable.tech/v1alpha1
kind: AuthenticationClass
metadata:
  name: nifi-users  (5)
spec:
  provider:
    static:
      userCredentialsSecret:
        name: nifi-admin-credentials
---
apiVersion: v1
kind: Secret
metadata:
  name: nifi-admin-credentials
stringData:
  admin: supersecretpassword
1 this field is no longer needed if the product and operator image versions match.
2 this section has been changed to take a list of authentication classes.
3 functionality has been removed.
4 functionality has been removed.
5 the authentication class referenced by the NiFi cluster.

Stackable Operator for Apache Spark

Breaking changes details

This requires a change from e.g.

apiVersion: spark.stackable.tech/v1alpha1
kind: SparkApplication
metadata:
  name: spark-examples
spec:
  version: "1.0"
  sparkImage: "docker.stackable.tech/stackable/spark-k8s:3.4.0-stackable23.7" (1)
  mode: cluster
  mainClass: org.apache.spark.examples.SparkALS
  mainApplicationFile: "local:///stackable/spark/examples/jars/spark-examples.jar"
  job: (2)
    logging:
      enableVectorAgent: False
  driver: (2)
    logging:
      enableVectorAgent: False
  executor: (2)
    instances: 1
    logging:
      enableVectorAgent: False

to:

apiVersion: spark.stackable.tech/v1alpha1
kind: SparkApplication
metadata:
  name: spark-examples-2
spec:
  version: "1.0"
  sparkImage: (1)
    productVersion: "3.5.0"
  mode: cluster
  mainClass: org.apache.spark.examples.SparkALS
  mainApplicationFile: "local:///stackable/spark/examples/jars/spark-examples.jar"
  job:
    config: (2)
      logging:
        enableVectorAgent: False
  driver:
    config: (2)
      logging:
        enableVectorAgent: False
  executor:
    replicas: 1
    config: (2)
      logging:
        enableVectorAgent: False
1 this field has been changed to be consistent with product image selection, documented here.
2 this section has been changed to be consistent with common configuration definitions used for other operators.

Stackable Operator for Apache Superset

  • Remove SupersetDB .spec.clusterConfig.loadExamplesOnInit was removed from the CRD. Already loaded examples in Superset will not be removed by this change. Additionally, the SupersetDB custom resource is not used anymore. Those CRs should be deleted. Loading examples is still supported, the process is now described in the documentation.

  • Rename service port name The service port name has been renamed from superset to http for consistency reasons. This change requires that the statefulset be removed before upgrading. There might also be some e.g. Ingresses that rely on the port name that will need to be updated.

Breaking changes details

This requires a change from e.g.

apiVersion: superset.stackable.tech/v1alpha1
kind: SupersetCluster
metadata:
  name: superset
spec:
  image:
    productVersion: "2.1.0"
    stackableVersion: "23.7" (1)
  clusterConfig:
    credentialsSecret: superset-credentials
    loadExamplesOnInit: false (2)
  nodes:
    roleGroups:
      default:
        replicas: 1
    ...

to:

apiVersion: superset.stackable.tech/v1alpha1
kind: SupersetCluster
metadata:
  name: superset
spec:
  image:
    productVersion: "2.1.0"
  clusterConfig:
    credentialsSecret: superset-credentials
  nodes:
    roleGroups:
      default:
        replicas: 1
    ...
1 this field is no longer needed if the product and operator image versions match.
2 this field has been removed.
it will be necessary to remove the stateful sets before updating the Superset resource due to the change to the name of container port.

Stackable Operator for Trino

Breaking changes details

This requires a change from e.g.

apiVersion: trino.stackable.tech/v1alpha1
kind: TrinoCluster
metadata:
  name: trino
spec:
  image:
    productVersion: "414"
  clusterConfig:
    ...

to:

---
apiVersion: trino.stackable.tech/v1alpha1
kind: TrinoCluster
metadata:
  name: trino
spec:
  image:
    productVersion: "428"
  clusterConfig:
    ...
---

To adapt the rego rules to work with the new authorizer visit the documentation here. For example, you can upgrade to the last version of Open Policy Agent and enter the relevant syntax in a ConfigMap like this:

---
apiVersion: opa.stackable.tech/v1alpha1
kind: OpaCluster
metadata:
  name: opa
spec:
  image:
    productVersion: "0.57.0"
    stackableVersion: 23.11
  servers:
    roleGroups:
      default: {}
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: simple-trino-opa-bundle
  labels:
    opa.stackable.tech/bundle: "trino"
data:
  trino.rego: |
    package trino

    import future.keywords.in

    default allow = false

    allow {
        is_admin
    }
    extended[i] {
        some i
        input.action.filterResources[i]
        is_admin
    }

    allow {
        input.action.operation in ["ExecuteQuery", "AccessCatalog"]
        is_bob
    }
    extended[i] {
        input.action.operation in ["FilterCatalogs"]
        some i
        input.action.filterResources[i]
        is_bob
    }

    is_admin() {
      input.context.identity.user == "admin"
    }

    is_bob() {
      input.context.identity.user == "bob"
    }

Upgrade from 23.7

Using stackablectl

To uninstall the 23.7 release run

$ stackablectl release uninstall 23.7
[INFO ] Uninstalling release 23.7
[INFO ] Uninstalling airflow operator
[INFO ] Uninstalling commons operator
# ...

Afterwards you will need to upgrade the CustomResourceDefinitions (CRDs) installed by the Stackable Platform. The reason for this is that helm will uninstall the operators but not the CRDs. This can be done using kubectl replace:

kubectl replace -f https://raw.githubusercontent.com/stackabletech/airflow-operator/23.11.0/deploy/helm/airflow-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/commons-operator/23.11.0/deploy/helm/commons-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/druid-operator/23.11.0/deploy/helm/druid-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/hbase-operator/23.11.0/deploy/helm/hbase-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/hdfs-operator/23.11.0/deploy/helm/hdfs-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/hello-world-operator/23.11.0/deploy/helm/hello-world-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/hive-operator/23.11.0/deploy/helm/hive-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/kafka-operator/23.11.0/deploy/helm/kafka-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/listener-operator/23.11.0/deploy/helm/listener-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/nifi-operator/23.11.0/deploy/helm/nifi-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/opa-operator/23.11.0/deploy/helm/opa-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/secret-operator/23.11.0/deploy/helm/secret-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/spark-k8s-operator/23.11.0/deploy/helm/spark-k8s-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/superset-operator/23.11.0/deploy/helm/superset-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/trino-operator/23.11.0/deploy/helm/trino-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/zookeeper-operator/23.11.0/deploy/helm/zookeeper-operator/crds/crds.yaml
customresourcedefinition.apiextensions.k8s.io "airflowclusters.airflow.stackable.tech" replaced
customresourcedefinition.apiextensions.k8s.io "airflowdbs.airflow.stackable.tech" replaced
customresourcedefinition.apiextensions.k8s.io "authenticationclasses.authentication.stackable.tech" replaced
customresourcedefinition.apiextensions.k8s.io "s3connections.s3.stackable.tech" replaced
...

To install the 23.11 release run

$ stackablectl release install 23.11
[INFO ] Installing release 23.11
[INFO ] Installing airflow operator in version 23.11.0
[INFO ] Installing commons operator in version 23.11.0
[INFO ] Installing druid operator in version 23.11.0
[INFO ] Installing hbase operator in version 23.11.0
[INFO ] Installing hdfs operator in version 23.11.0
[INFO ] Installing hive operator in version 23.11.0
[INFO ] Installing kafka operator in version 23.11.0
[INFO ] Installing listener operator in version 23.11.0
[INFO ] Installing hello-world operator in version 23.11.0
[INFO ] Installing nifi operator in version 23.11.0
[INFO ] Installing opa operator in version 23.11.0
[INFO ] Installing secret operator in version 23.11.0
[INFO ] Installing spark-k8s operator in version 23.11.0
[INFO ] Installing superset operator in version 23.11.0
[INFO ] Installing trino operator in version 23.11.0
[INFO ] Installing zookeeper operator in version 23.11.0

Using helm

Use helm list to list the currently installed operators.

You can use the following command to uninstall all operators that are part of the 23.7 release:

$ helm uninstall airflow-operator commons-operator druid-operator hbase-operator hdfs-operator hive-operator kafka-operator listener-operator nifi-operator opa-operator secret-operator spark-k8s-operator superset-operator trino-operator zookeeper-operator
release "airflow-operator" uninstalled
release "commons-operator" uninstalled
# ...

Afterwards you will need to upgrade the CustomResourceDefinitions (CRDs) installed by the Stackable Platform. The reason for this is that helm will uninstall the operators but not the CRDs. This can be done using kubectl replace:

kubectl replace -f https://raw.githubusercontent.com/stackabletech/airflow-operator/23.11.0/deploy/helm/airflow-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/commons-operator/23.11.0/deploy/helm/commons-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/druid-operator/23.11.0/deploy/helm/druid-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/hbase-operator/23.11.0/deploy/helm/hbase-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/hdfs-operator/23.11.0/deploy/helm/hdfs-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/hello-world-operator/23.11.0/deploy/helm/hello-world-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/hive-operator/23.11.0/deploy/helm/hive-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/kafka-operator/23.11.0/deploy/helm/kafka-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/listener-operator/23.11.0/deploy/helm/listener-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/nifi-operator/23.11.0/deploy/helm/nifi-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/opa-operator/23.11.0/deploy/helm/opa-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/secret-operator/23.11.0/deploy/helm/secret-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/spark-k8s-operator/23.11.0/deploy/helm/spark-k8s-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/superset-operator/23.11.0/deploy/helm/superset-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/trino-operator/23.11.0/deploy/helm/trino-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/zookeeper-operator/23.11.0/deploy/helm/zookeeper-operator/crds/crds.yaml

To install the 23.11 release run

helm repo add stackable-stable https://repo.stackable.tech/repository/helm-stable/
helm repo update stackable-stable
helm install --wait airflow-operator stackable-stable/airflow-operator --version 23.11.0
helm install --wait commons-operator stackable-stable/commons-operator --version 23.11.0
helm install --wait druid-operator stackable-stable/druid-operator --version 23.11.0
helm install --wait hbase-operator stackable-stable/hbase-operator --version 23.11.0
helm install --wait hdfs-operator stackable-stable/hdfs-operator --version 23.11.0
helm install --wait hive-operator stackable-stable/hive-operator --version 23.11.0
helm install --wait kafka-operator stackable-stable/kafka-operator --version 23.11.0
helm install --wait listener-operator stackable-stable/listener-operator --version 23.11.0
helm install --wait hello-world-operator stackable-stable/hello-world-operator --version 23.11.0
helm install --wait nifi-operator stackable-stable/nifi-operator --version 23.11.0
helm install --wait opa-operator stackable-stable/opa-operator --version 23.11.0
helm install --wait secret-operator stackable-stable/secret-operator --version 23.11.0
helm install --wait spark-k8s-operator stackable-stable/spark-k8s-operator --version 23.11.0
helm install --wait superset-operator stackable-stable/superset-operator --version 23.11.0
helm install --wait trino-operator stackable-stable/trino-operator --version 23.11.0
helm install --wait zookeeper-operator stackable-stable/zookeeper-operator --version 23.11.0

Known upgrade issues

In the case of the breaking changes detailed above it will be necessary to update the custom resources and re-apply them.

Additionally, please note the following:

All operators

  • If the default PVC size has been changed, then the StatefulSet must be deleted: it is not possible to change the PVC in the StatefulSet specification.

Known issues

Release 23.7

This release introduces the specification of resource quotas and pod overrides and updates the product versions supported by SDP.

New / extended platform features

The following new major platform features were added:

Resource Quotas

Explicit resources are now applied to all containers, for both operators and products. This allows running the Stackable Data Platform on Kubernetes clusters with a ResourceQuota or LimitRange set. Where these are not specified directly, defaults will be used. See this issue for more information.

Pod Overrides

It is now possible to add custom settings which specify elements of a pod template (Service, StatefulSet etc.) on roles or rolegroups, which the operator then merges with the objects it writes before actually applying them. This provides the user with a possibility for specifying any property that can be set on a regular Kubernetes Pod, but which is not directly exposed via the Stackable custom resource definition. Have a look at the documentation for more details.

For example, with HDFS:

    roleGroups:
      default:
        replicas: 1
        podOverrides:
          spec:
            containers:
              - name: journalnode
                resources:
                  requests:
                    cpu: 110m
                  limits:
                    cpu: 410m
Openshift certification

OLM bundles - a pre-requisite for the Openshift certification process - have been created for each operator. All 15 SDP operators in release 23.4.1 are now Openshift-certified and deployable directly from within an Openshift cluster.

Signed SDP operator images

As of this release all Stackable operator images are signed (this feature will be added to product images in a subsequent release). More information about this, including how to verify the image signatures, can be found in this guide.

New Versions

The following new product versions are now supported:

Deprecated Versions

The following product versions are deprecated and will be removed in a later release:

  • Airflow: 2.2.3, 2.2.4, 2.2.5, 2.4.1

  • Druid: 0.23.0, 24.0.0

  • HBase: 2.4.6, 2.4.8, 2.4.9, 2.4.11

  • HDFS: 3.2.2, 3.3.1, 3.3.3

  • Hive: 2.3.9

  • Kafka: 2.7.1, 2.8.1, 3.1.0, 3.2.0, 3.3.1

  • Nifi: 1.15.0, 1.15.1, 1.15.2, 1.15.3, 1.16.0, 1.16.1, 1.16.2, 1.16.3, 1.18.0

  • Opa: 0.27.1, 0.28.0, 0.37.2, 0.41.0, 0.45.0

  • Spark: 3.2.1, 3.3.0

  • Superset: 1.3.2, 1.4.1, 1.5.1

  • Trino: 377, 387, 395, 396, 403

  • Zookeeper: 3.5.8, 3.6.3, 3.7.0, 3.8.0

Removed Versions

No product versions have been removed.

Product features

Additionally, there are some individual product features that are noteworthy:

stackablectl

There are no new demos in this platform release.

Supported Kubernetes versions

This release supports the following Kubernetes versions:

  • 1.26

  • 1.25

  • 1.24

This Kubernetes version is no longer supported:

  • 1.23

Supported OpenShift versions

This release supports the following OpenShift versions:

  • 4.11

  • 4.10

Breaking changes

The re-structuring of configuration definitions in certain operators will require you to adapt your existing CRDs as shown below.

Stackable Operator for Apache Airflow

CRDs should be changed from e.g.

spec:
  ...
  executor: CeleryExecutor
  loadExamples: true
  exposeConfig: false
  credentialsSecret: test-airflow-credentials
  ...

to:

spec:
  ...
  clusterConfig:
    executor: CeleryExecutor
    loadExamples: true
    exposeConfig: false
    credentialsSecret: test-airflow-credentials
    ...

Stackable Operator for Apache Superset

CRDs should be changed from e.g.

spec:
  ...
  credentialsSecret: superset-credentials
  loadExamplesOnInit: false
  vectorAggregatorConfigMapName: vector-aggregator-discovery
  ...

to:

spec:
  ...
  clusterConfig:
    credentialsSecret: superset-credentials
    loadExamplesOnInit: false
    vectorAggregatorConfigMapName: vector-aggregator-discovery
    ...

Stackable Operator for Trino

CRDs should be changed from e.g.

spec:
  ...
  clusterConfig:
    authentication:
      method:
        multiUser:
          userCredentialsSecret:
            name: trino-users
  ...

referencing a Secret with bcrypt-ed data:

---
apiVersion: v1
kind: Secret
metadata:
  name: trino-users
type: kubernetes.io/opaque
stringData:
  # admin:admin
  admin: $2y$10$89xReovvDLacVzRGpjOyAOONnayOgDAyIS2nW9bs5DJT98q17Dy5i
  # alice:alice
  alice: $2y$10$HcCa4k9v2DRrD/g7e5vEz.Bk.1xg00YTEHOZjPX7oK3KqMSt2xT8W
  # bob:bob
  bob: $2y$10$xVRXtYZnYuQu66SmruijPO8WHFM/UK5QPHTr.Nzf4JMcZSqt3W.2.

to:

spec:
  ...
  clusterConfig:
    authentication:
      - authenticationClass: trino-users-auth
    ...

referencing an AuthenticationClass (which references a Secret with plain data):

---
apiVersion: authentication.stackable.tech/v1alpha1
kind: AuthenticationClass
metadata:
    name: trino-users-auth
spec:
  provider:
    static:
      userCredentialsSecret:
        name: trino-users
---
apiVersion: v1
kind: Secret
metadata:
  name: trino-users
type: kubernetes.io/opaque
stringData:
  admin: admin
  alice: alice
  bob: bob

Upgrade from 23.4

Using stackablectl

To uninstall the 23.4 release run

$ stackablectl release uninstall 23.4
[INFO ] Uninstalling release 23.4
[INFO ] Uninstalling airflow operator
[INFO ] Uninstalling commons operator
# ...

Afterwards you will need to upgrade the CustomResourceDefinitions (CRDs) installed by the Stackable Platform. The reason for this is that helm will uninstall the operators but not the CRDs. This can be done using kubectl replace:

kubectl replace -f https://raw.githubusercontent.com/stackabletech/airflow-operator/23.7.0/deploy/helm/airflow-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/commons-operator/23.7.0/deploy/helm/commons-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/druid-operator/23.7.0/deploy/helm/druid-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/hbase-operator/23.7.0/deploy/helm/hbase-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/hdfs-operator/23.7.0/deploy/helm/hdfs-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/hive-operator/23.7.0/deploy/helm/hive-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/kafka-operator/23.7.0/deploy/helm/kafka-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/listener-operator/23.7.0/deploy/helm/listener-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/nifi-operator/23.7.0/deploy/helm/nifi-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/opa-operator/23.7.0/deploy/helm/opa-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/secret-operator/23.7.0/deploy/helm/secret-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/spark-k8s-operator/23.7.0/deploy/helm/spark-k8s-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/superset-operator/23.7.0/deploy/helm/superset-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/trino-operator/23.7.0/deploy/helm/trino-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/zookeeper-operator/23.7.0/deploy/helm/zookeeper-operator/crds/crds.yaml
customresourcedefinition.apiextensions.k8s.io "airflowclusters.airflow.stackable.tech" replaced
customresourcedefinition.apiextensions.k8s.io "airflowdbs.airflow.stackable.tech" replaced
customresourcedefinition.apiextensions.k8s.io "authenticationclasses.authentication.stackable.tech" replaced
customresourcedefinition.apiextensions.k8s.io "s3connections.s3.stackable.tech" replaced
...

To install the 23.7 release run

$ stackablectl release install 23.7
[INFO ] Installing release 23.7
[INFO ] Installing airflow operator in version 23.7.0
[INFO ] Installing commons operator in version 23.7.0
[INFO ] Installing druid operator in version 23.7.0
[INFO ] Installing hbase operator in version 23.7.0
[INFO ] Installing hdfs operator in version 23.7.0
[INFO ] Installing hive operator in version 23.7.0
[INFO ] Installing kafka operator in version 23.7.0
[INFO ] Installing listener operator in version 23.7.0
[INFO ] Installing nifi operator in version 23.7.0
[INFO ] Installing opa operator in version 23.7.0
[INFO ] Installing secret operator in version 23.7.0
[INFO ] Installing spark-k8s operator in version 23.7.0
[INFO ] Installing superset operator in version 23.7.0
[INFO ] Installing trino operator in version 23.7.0
[INFO ] Installing zookeeper operator in version 23.7.0

Using helm

Use helm list to list the currently installed operators.

You can use the following command to uninstall all operators that are part of the 23.4 release:

$ helm uninstall airflow-operator commons-operator druid-operator hbase-operator hdfs-operator hive-operator kafka-operator listener-operator nifi-operator opa-operator secret-operator spark-k8s-operator superset-operator trino-operator zookeeper-operator
release "airflow-operator" uninstalled
release "commons-operator" uninstalled
# ...

Afterwards you will need to upgrade the CustomResourceDefinitions (CRDs) installed by the Stackable Platform. The reason for this is that helm will uninstall the operators but not the CRDs. This can be done using kubectl replace:

kubectl replace -f https://raw.githubusercontent.com/stackabletech/airflow-operator/23.7.0/deploy/helm/airflow-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/commons-operator/23.7.0/deploy/helm/commons-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/druid-operator/23.7.0/deploy/helm/druid-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/hbase-operator/23.7.0/deploy/helm/hbase-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/hdfs-operator/23.7.0/deploy/helm/hdfs-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/hive-operator/23.7.0/deploy/helm/hive-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/kafka-operator/23.7.0/deploy/helm/kafka-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/listener-operator/23.7.0/deploy/helm/listener-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/nifi-operator/23.7.0/deploy/helm/nifi-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/opa-operator/23.7.0/deploy/helm/opa-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/secret-operator/23.7.0/deploy/helm/secret-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/spark-k8s-operator/23.7.0/deploy/helm/spark-k8s-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/superset-operator/23.7.0/deploy/helm/superset-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/trino-operator/23.7.0/deploy/helm/trino-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/zookeeper-operator/23.7.0/deploy/helm/zookeeper-operator/crds/crds.yaml

To install the 23.7 release run

helm repo add stackable-stable https://repo.stackable.tech/repository/helm-stable/
helm repo update stackable-stable
helm install --wait airflow-operator stackable-stable/airflow-operator --version 23.7.0
helm install --wait commons-operator stackable-stable/commons-operator --version 23.7.0
helm install --wait druid-operator stackable-stable/druid-operator --version 23.7.0
helm install --wait hbase-operator stackable-stable/hbase-operator --version 23.7.0
helm install --wait hdfs-operator stackable-stable/hdfs-operator --version 23.7.0
helm install --wait hive-operator stackable-stable/hive-operator --version 23.7.0
helm install --wait kafka-operator stackable-stable/kafka-operator --version 23.7.0
helm install --wait listener-operator stackable-stable/listener-operator --version 23.7.0
helm install --wait nifi-operator stackable-stable/nifi-operator --version 23.7.0
helm install --wait opa-operator stackable-stable/opa-operator --version 23.7.0
helm install --wait secret-operator stackable-stable/secret-operator --version 23.7.0
helm install --wait spark-k8s-operator stackable-stable/spark-k8s-operator --version 23.7.0
helm install --wait superset-operator stackable-stable/superset-operator --version 23.7.0
helm install --wait trino-operator stackable-stable/trino-operator --version 23.7.0
helm install --wait zookeeper-operator stackable-stable/zookeeper-operator --version 23.7.0

Known upgrade issues

In the case of the breaking changes detailed above it will be necessary to update the custom resources for Airflow, Superset and Trino clusters and re-apply them.

Additionally, please note the following:

All operators

  • If the default PVC size has been changed, then the StatefulSet must be deleted: it is not possible to change the PVC in the StatefulSet specification.

    • The error message is similar to: StatefulSet.apps "trino-worker-default" is invalid: spec: Forbidden: updates to `StatefulSet spec for fields other than 'replicas', 'template', 'updateStrategy', […​]`

ZooKeeper operator

  • The ZooKeeper operator in this release expects a product image with the same version. An existing ZooKeeper cluster resource should be deleted and re-applied with the corresponding stackableVersion e.g.:

spec:
  image:
    productVersion: 3.8.0
    stackableVersion: "23.7"

Release 23.4

The focus in this platform release is on the support of default/custom affinities and the status field, as well as the rollout of log aggregation across the remaining operators. Additionally, all operators have been updated and tested for compatibility with OpenShift clusters (versions 4.10 and 4.11). Several operators from the 23.1 platform release were already certified against OpenShift.

Release 23.4.0

This was the first release in the 23.4 release line. It is recommended to install Release 23.4.1 instead, as it contains relevant bugfixes.

Release 23.4.1

This is a bugfix/patch-level release that fixes the following issues:

  • Fix missing custom resource defaults that are required for a release update. See here.

  • Specify the security context to run as a member of the root group (this has been implemented for the Stackable operators where it had not previously been implemented i.e. Apache HBase, Apache HDFS, Apache ZooKeeper and Apache Spark on Kubernetes). This is required by Openshift clusters so that the product can be run with a random UID. This is a requirement for at least Airflow, but is Openshift policy as described here and here.

  • Automatically migrate the name used for the bundle-builder container for OPA daemonsets. See here.

  • Automatically shorten the registration socket path used in listener-operator for Microk8s compatibility, migrated during upgrade. See here.

New / extended platform features

The following new major platform features were added:

Cluster Operation

The first part of Cluster operations was rolled out in every applicable Stackable Operator. This supports pausing the cluster reconciliation and stopping the cluster completely. Pausing reconciliation will not apply any changes to the Kubernetes resources (e.g. when changing the custom resource). Stopping the cluster will set all replicas of StatefulSets, Deployments or DaemonSets to zero and therefore result in the deletion of all Pods belonging to that cluster (not the PVCs)

Status Field

Operators of the Stackable Data Platform create, manage and delete Kubernetes resources: in order to easily query the health state of the products - and react accordingly - Stackable Operators use several predefined condition types to capture different aspects of a product’s availability. See this ADR for more information.

Default / Custom Affinities

In Kubernetes there are different ways to influence how Pods are assigned to Nodes. In some cases it makes sense to co-locate certain services that communicate a lot with each other, such as HBase regionservers with HDFS datanodes. In other cases it makes sense to distribute the Pods among as many Nodes as possible. There may also be additional requirements e.g. placing important services - such as HDFS namenodes - in different racks, datacenter rooms or even datacenters. This release implements default affinities that should suffice for many scenarios out-of-the box, while also allowing for custom affinity rules at a role and/or role-group level. See this ADR for more information.

Log Aggregation

The logging framework (added to the platform in Release 23.1) offers a consistent custom resource configuration and a separate, persisted sink (defaulting to OpenSearch). This has now been rolled out across all products. See this ADR and this concepts page for more information.

Service Type

The Service type can now be specified in all products. This currently differentiates between the internal ClusterIP and the external NodePort and is forward compatible with the ListenerClass for the automatic exposure of Services via the Listener Operator. This change is not backwards compatible with older platform releases. For security reasons, the default is set to the cluster-internal (ClusterIP) ListenerClass. A cluster can be exposed outside of Kubernetes by setting clusterConfig.listenerClass to external-unstable (NodePort) or external-stable (LoadBalancer).

New Versions

No new product versions are supported in this platform release.

Deprecated Versions

No product versions have been deprecated in this platform release.

Product features

Additionally, there are some individual product features that are noteworthy:

stackablectl

The following have been added to stackablectl:

Trino-iceberg demo

This is a condensed form of the data-lakehouse-iceberg-trino-spark demo focusing on using the lakehouse to store and modify data. It demonstrates how to integrate Trino and Iceberg and should run on a local workstation.

Jupyterhub/Spark demo

This demo showcases the integration between Jupyter and Apache Hadoop deployed on the Stackable Data Platform (SDP) Kubernetes cluster. This demo can be installed on most cloud managed Kubernetes clusters as well as on premise or on a reasonably provisioned laptop.

The quickstart guide shows how to get started with stackablectl. This link lists the available demos.

Supported Kubernetes versions

This release supports the following Kubernetes versions:

  • 1.26

  • 1.25

  • 1.24

  • 1.23 (it is planned to discontinue support for this version in the next release)

Supported OpenShift versions

This release supports the following OpenShift versions:

  • 4.11

  • 4.10

Breaking changes

You will need to adapt your existing CRDs due to the following breaking changes detailed below.

All Stackable Operators

As mentioned above, specifying the service type is a breaking change for all operators. The default value is set to the cluster-internal ListenerClass: if the cluster requires external access outside of Kubernetes then set clusterConfig.listenerClass to external-unstable or external-stable:

spec:
  image:
    productVersion: "396"
    stackableVersion: "23.4.1"
  clusterConfig:
    listenerClass: external-unstable

This is an example for Trino, but the pattern is the same across all operators.

Stackable Operator for Apache Airflow

Existing Airflow clusters need to be deleted and recreated. Airflow metadata held in the database and DAGs saved on disk are not affected.

This is required because the UID of the Airflow user has changed to be in line with the rest of the platform.

Stackable Operator for Apache HBase

CRDs should be changed from e.g.

spec:
  ...
  hdfsConfigMapName: simple-hdfs
  zookeeperConfigMapName: simple-znode

to:

spec:
  ...
  clusterConfig:
    hdfsConfigMapName: simple-hdfs
    zookeeperConfigMapName: simple-znode

Stackable Operator for Apache Hadoop

CRDs should be changed from e.g.

spec:
  ...
  zookeeperConfigMapName: simple-hdfs-znode
  dfsReplication: 3
  vectorAggregatorConfigMapName: vector-aggregator-discovery

to:

spec:
  ...
  clusterConfig:
    zookeeperConfigMapName: simple-hdfs-znode
    dfsReplication: 1
    vectorAggregatorConfigMapName: vector-aggregator-discovery

Stackable Operator for Apache Nifi

CRDs should be changed from e.g.

spec:
  ...
  zookeeperConfigMapName: simple-nifi-znode

to:

spec:
  ...
  clusterConfig:
    zookeeperConfigMapName: simple-nifi-znode

Stackable Operator for Apache Spark-k8s

CRDs should be changed from e.g.

spec:
  ...
  driver:
    nodeSelector:

to:

spec:
  ...
  driver:
    affinity:

Stackable Operator for Apache Trino

CRDs should be changed from e.g.

spec:
  ...
  opa:
    configMapName: simple-opa
    package: trino
  authentication:
    method:
      multiUser:
        userCredentialsSecret:
          name: simple-trino-users-secret
  catalogLabelSelector:
    matchLabels:
      trino: simple-trino
  vectorAggregatorConfigMapName: vector-aggregator-discovery

to:

spec:
  ...
  clusterConfig:
    authentication:
      method:
        multiUser:
          userCredentialsSecret:
            name: simple-trino-users-secret
    authorization:
      opa:
        configMapName: simple-opa
        package: trino
    catalogLabelSelector:
      matchLabels:
        trino: simple-trino
    vectorAggregatorConfigMapName: vector-aggregator-discovery

Upgrade from 23.1

Using stackablectl

You can list the available releases as follows

$ stackablectl release list

RELEASE            RELEASE DATE   DESCRIPTION
23.4               2023-04-25     Fifth release focusing on affinities and product status
23.1               2023-01-27     Fourth release focusing on image selection and logging
22.11              2022-11-08     Third release focusing on resource management
22.09              2022-09-09     Second release focusing on security and OpenShift support
22.06              2022-06-30     First official release of the Stackable Data Platform

To uninstall the 23.1 release run

$ stackablectl release uninstall 23.1
[INFO ] Uninstalling release 23.1
[INFO ] Uninstalling airflow operator
[INFO ] Uninstalling commons operator
# ...

Afterwards you will need to upgrade the CustomResourceDefinitions (CRDs) installed by the Stackable Platform. The reason for this is that helm will uninstall the operators but not the CRDs. This can be done using kubectl replace:

kubectl replace -f https://raw.githubusercontent.com/stackabletech/airflow-operator/23.4.1/deploy/helm/airflow-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/commons-operator/23.4.1/deploy/helm/commons-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/druid-operator/23.4.1/deploy/helm/druid-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/hbase-operator/23.4.1/deploy/helm/hbase-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/hdfs-operator/23.4.1/deploy/helm/hdfs-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/hive-operator/23.4.1/deploy/helm/hive-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/kafka-operator/23.4.1/deploy/helm/kafka-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/listener-operator/23.4.1/deploy/helm/listener-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/nifi-operator/23.4.1/deploy/helm/nifi-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/opa-operator/23.4.1/deploy/helm/opa-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/secret-operator/23.4.1/deploy/helm/secret-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/spark-k8s-operator/23.4.1/deploy/helm/spark-k8s-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/superset-operator/23.4.1/deploy/helm/superset-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/trino-operator/23.4.1/deploy/helm/trino-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/zookeeper-operator/23.4.1/deploy/helm/zookeeper-operator/crds/crds.yaml
customresourcedefinition.apiextensions.k8s.io "airflowclusters.airflow.stackable.tech" replaced
customresourcedefinition.apiextensions.k8s.io "airflowdbs.airflow.stackable.tech" replaced
customresourcedefinition.apiextensions.k8s.io "authenticationclasses.authentication.stackable.tech" replaced
customresourcedefinition.apiextensions.k8s.io "s3connections.s3.stackable.tech" replaced
...

To install the 23.4 release run

$ stackablectl release install 23.4
[INFO ] Installing release 23.4
[INFO ] Installing airflow operator in version 23.4.1
[INFO ] Installing commons operator in version 23.4.1
[INFO ] Installing druid operator in version 23.4.1
[INFO ] Installing hbase operator in version 23.4.1
[INFO ] Installing hdfs operator in version 23.4.1
[INFO ] Installing hive operator in version 23.4.1
[INFO ] Installing kafka operator in version 23.4.1
[INFO ] Installing listener operator in version 23.4.1
[INFO ] Installing nifi operator in version 23.4.1
[INFO ] Installing opa operator in version 23.4.1
[INFO ] Installing secret operator in version 23.4.1
[INFO ] Installing spark-k8s operator in version 23.4.1
[INFO ] Installing superset operator in version 23.4.1
[INFO ] Installing trino operator in version 23.4.1
[INFO ] Installing zookeeper operator in version 23.4.1

Using helm

Use helm list to list the currently installed operators.

You can use the following command to uninstall all operators that are part of the 23.1 release:

$ helm uninstall airflow-operator commons-operator druid-operator hbase-operator hdfs-operator hive-operator kafka-operator listener-operator nifi-operator opa-operator secret-operator spark-k8s-operator superset-operator trino-operator zookeeper-operator
release "airflow-operator" uninstalled
release "commons-operator" uninstalled
# ...

Afterwards you will need to upgrade the CustomResourceDefinitions (CRDs) installed by the Stackable Platform. The reason for this is that helm will uninstall the operators but not the CRDs. This can be done using kubectl replace:

kubectl replace -f https://raw.githubusercontent.com/stackabletech/airflow-operator/23.4.1/deploy/helm/airflow-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/commons-operator/23.4.1/deploy/helm/commons-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/druid-operator/23.4.1/deploy/helm/druid-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/hbase-operator/23.4.1/deploy/helm/hbase-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/hdfs-operator/23.4.1/deploy/helm/hdfs-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/hive-operator/23.4.1/deploy/helm/hive-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/kafka-operator/23.4.1/deploy/helm/kafka-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/listener-operator/23.4.1/deploy/helm/listener-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/nifi-operator/23.4.1/deploy/helm/nifi-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/opa-operator/23.4.1/deploy/helm/opa-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/secret-operator/23.4.1/deploy/helm/secret-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/spark-k8s-operator/23.4.1/deploy/helm/spark-k8s-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/superset-operator/23.4.1/deploy/helm/superset-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/trino-operator/23.4.1/deploy/helm/trino-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/zookeeper-operator/23.4.1/deploy/helm/zookeeper-operator/crds/crds.yaml

To install the 23.4 release run

helm repo add stackable-stable https://repo.stackable.tech/repository/helm-stable/
helm repo update stackable-stable
helm install --wait airflow-operator stackable-stable/airflow-operator --version 23.4.1
helm install --wait commons-operator stackable-stable/commons-operator --version 23.4.1
helm install --wait druid-operator stackable-stable/druid-operator --version 23.4.1
helm install --wait hbase-operator stackable-stable/hbase-operator --version 23.4.1
helm install --wait hdfs-operator stackable-stable/hdfs-operator --version 23.4.1
helm install --wait hive-operator stackable-stable/hive-operator --version 23.4.1
helm install --wait kafka-operator stackable-stable/kafka-operator --version 23.4.1
helm install --wait listener-operator stackable-stable/listener-operator --version 23.4.1
helm install --wait nifi-operator stackable-stable/nifi-operator --version 23.4.1
helm install --wait opa-operator stackable-stable/opa-operator --version 23.4.1
helm install --wait secret-operator stackable-stable/secret-operator --version 23.4.1
helm install --wait spark-k8s-operator stackable-stable/spark-k8s-operator --version 23.4.1
helm install --wait superset-operator stackable-stable/superset-operator --version 23.4.1
helm install --wait trino-operator stackable-stable/trino-operator --version 23.4.1
helm install --wait zookeeper-operator stackable-stable/zookeeper-operator --version 23.4.1

Release 23.1

This release marks a major change in the way operator and product images are versioned. Up until now, operators were versioned independently of each other and a platform release was a loosely coupled set of operator versions. This had major disadvantages both technical and organisational.

On the technical side, a multi-dimensional matrix of versions had to be tested, documentation cross-references had to be maintained and coordinating a platform release was extremely difficult.

Organizationally the biggest challenge was communication and coordination within the teams as well as to and with users. As a result, starting with this release, all operator and product images are versioned in lock-step. This platform release is marked 23.1 and all included components are tagged with 23.1.0. Eventual patch versions of the components that might follow and will be tagged with 23.1.1, 23.1.2 and so on.

The focus in this platform release is on the support of offline (or on-premise) product images and the partial rollout of logging support.

New platform features

The following new major platform features were added:

Product image selection

Product image selection has been expanded to cover different scenarios:

  • Stackable-provided product images, defined with the repository, the product version and the stackable tag

  • As above, but without the stackable tag (whereby the most recent tagged image will be taken)

  • The product version and a full repository path (this allows fully-customized product images)

These options are described in more detail in this ADR and on this concepts page.

N.B. this is a breaking change across all operators as spec.version has been replaced by spec.image.

Logging Aggregation

Component activity within the platform is logged in a way that makes it difficult to find, persist and consolidate this information. Log configuration is also a challenge. To address these two issues a logging framework has been added to the platform, offering a consistent custom resource configuration and a separate, persisted sink (the current implementation support OpenSearch). This is discussed in more detail in this ADR and on this concepts page.

In this release this has been added to the following components:

Support for other products will be added in future releases.

New Versions

The following new product version is now supported:

Deprecated Versions

The following product version is no longer supported:

Product features

Additionally, there are some individual product features that are noteworthy

stackablectl

The following have been added to stackablectl:

Logging demo

This illustrates how to set up logging for Zookeeper and browse the results in an Open Search dashboard. This has been implemented for HBase, Hadoop and Zookeeper and will eventually be available for all Stackable operators.

LDAP stack and tutorial

LDAP support has now been added to multiple products. An explanation of the overall approach is given here but in order to make the configuration steps a little clearer a tutorial has been added that uses a dedicated Stackable stack for OpenLDAP and shows its usage.

The quickstart guide shows how to get started with stackablectl. This link lists the available demos.

Supported Kubernetes versions

This release supports the following Kubernetes versions:

  • 1.25

  • 1.24

  • 1.23

  • 1.22

Breaking changes

This release brings with it several breaking changes needed to future-proof the platform. You will need to adapt your existing CRDs due to the following breaking changes:

All Stackable Operators

As mentioned above, product image selection is a breaking for all operators. Previously the product image was declared using spec.version:

spec:
  version: 396-stackable23.1

(example for Trino)

This must now be replaced with spec.image:

spec:
  image:
    productVersion: 396
    stackableVersion: 23.1

This is the same pattern across operators. so for Hive the change would look like this. From:

spec:
  version: 3.1.3-stackable23.1

to

spec:
  image:
    productVersion: 3.1.3
    stackableVersion: 23.1

Stackable Operator for Apache Druid

This means a stackable version >= 23.1 has to be used for the product image.

Deep storage, Ingestion spec, discovery config maps, authentication etc. are now subfields of spec.clusterConfig instead of being top level under spec. Change the resource from e.g.:

  zookeeperConfigMapName: simple-druid-znode
  metadataStorageDatabase:
    dbType: derby
    connString: jdbc:derby://localhost:1527/var/druid/metadata.db;create=true
    host: localhost
    port: 1527
  deepStorage:
    hdfs:
      configMapName: simple-hdfs
      directory: /data

to

  clusterConfig:
    deepStorage:
      hdfs:
        configMapName: simple-hdfs
        directory: /data
    metadataStorageDatabase:
      dbType: derby
      connString: jdbc:derby://localhost:1527/var/druid/metadata.db;create=true
      host: localhost
      port: 1527
    tls: null
    zookeeperConfigMapName: simple-druid-znode

Stackable Operator for Apache Hadoop

As part of the change mentioned above we also did some code cleanup that allowed us to remove arbitrary hard-coded values from the operator. This change affects the directory structure the operator creates inside of the PersistentVolumes used for permanent storage.

The old folder naming was:

  • DataNode → data

  • JournalNode → journal

  • NameNode → name

which has now been adopted to match the actual rolename:

  • DataNode → datanode

  • JournalNode → journalnode

  • NameNode → namenode

Unfortunately, this means that for cluster that where initially rolled out with an older operator version, a one-time migration step becomes necessary to rename these directories.

You can either do this manually by attaching the PVs to a pod and performing the rename (cluster needs to be stopped for this) or use the script provided below.

Please be aware that if this script runs after the cluster was already restarted with the newer operator version it will delete any data that was written to the empty post-upgrade HDFS that was stood up by the new operator. 
#!/usr/bin/env bash

if [ $# -ne 1 ] ; then
     echo "Usage: $0 CLUSTER_NAME"
     exit 1
else
    HDFS_CLUSTER_NAME=$1
fi

kubectl get pvc -l app.kubernetes.io/name=hdfs -l app.kubernetes.io/instance="$HDFS_CLUSTER_NAME"
for pvc in $(kubectl get pvc -l app.kubernetes.io/name=hdfs -l app.kubernetes.io/instance="$HDFS_CLUSTER_NAME" -l app.kubernetes.io/component=journalnode -o name | sed -e 's#persistentvolumeclaim/##'); do
kubectl apply -f - << EOF
apiVersion: batch/v1
kind: Job
metadata:
  name: migrate-journalnode-${pvc}
spec:
  template:
    spec:
      containers:
        - name: migrate
          image: docker.stackable.tech/stackable/hadoop:3.3.4-stackable23.1.0
          command: ["bash", "-c", "ls -la /stackable/data && if [ -d /stackable/data/journal ]; then echo Removing might existing target dir && rm -rf /stackable/data/journalnode && echo Renaming folder && mv /stackable/data/journal /stackable/data/journalnode; else echo Nothing to do; fi"]
          volumeMounts:
            - name: data
              mountPath: /stackable/data
      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: ${pvc}
      restartPolicy: Never
  backoffLimit: 1
EOF
done
for pvc in $(kubectl get pvc -l app.kubernetes.io/name=hdfs -l app.kubernetes.io/instance="$HDFS_CLUSTER_NAME" -l app.kubernetes.io/component=namenode -o name | sed -e 's#persistentvolumeclaim/##'); do
kubectl apply -f - << EOF
apiVersion: batch/v1
kind: Job
metadata:
  name: migrate-namenode-${pvc}
spec:
  template:
    spec:
      containers:
        - name: migrate
          image: docker.stackable.tech/stackable/hadoop:3.3.4-stackable23.1.0
          command: ["bash", "-c", "ls -la /stackable/data && if [ -d /stackable/data/name ]; then echo Removing might existing target dir && rm -rf /stackable/data/namenode && echo Renaming folder && mv /stackable/data/name /stackable/data/namenode; else echo Nothing to do; fi"]
          volumeMounts:
            - name: data
              mountPath: /stackable/data
      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: ${pvc}
      restartPolicy: Never
  backoffLimit: 1
EOF
done
for pvc in $(kubectl get pvc -l app.kubernetes.io/name=hdfs -l app.kubernetes.io/instance="$HDFS_CLUSTER_NAME" -l app.kubernetes.io/component=datanode -o name | sed -e 's#persistentvolumeclaim/##'); do
kubectl apply -f - << EOF
apiVersion: batch/v1
kind: Job
metadata:
  name: migrate-datanode-${pvc}
spec:
  template:
    spec:
      containers:
        - name: migrate
          image: docker.stackable.tech/stackable/hadoop:3.3.4-stackable23.1.0
          command: ["bash", "-c", "ls -la /stackable/data/data && if [ -d /stackable/data/data/data ]; then echo Removing might existing target dir && rm -rf /stackable/data/data/datanode && echo Renaming folder && mv /stackable/data/data/data /stackable/data/data/datanode; else echo Nothing to do; fi"]
          volumeMounts:
            - name: data
              mountPath: /stackable/data/data
      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: ${pvc}
      restartPolicy: Never
  backoffLimit: 1
EOF
done

The migration process for this now becomes:

  • Stop HDFS cluster by either removing the HdfsCluster definition object or scaling all roles to 0 replicas

  • Uninstall Stackable Operator for Apache Hadoop

  • Run migration script

  • Install newer version of Stackable Operator for Apache Hadoop

Stackable Operator for Apache Hive

These two changes mean that resources previously defined like this:

  s3:
    reference: minio
  metastore:
    roleGroups:
      default:
        replicas: 1
        config:
          database:
            connString: jdbc:postgresql://hive-postgresql:5432/hive
            user: hive
            password: hive
            dbType: postgres

will now be defined like this:

  clusterConfig:
    database:
      connString: jdbc:postgresql://hive-postgresql:5432/hive
      user: hive
      password: hive
      dbType: postgres
    s3:
      reference: minio
  metastore:
    roleGroups:
      default:
        replicas: 1

Stackable Operator for Apache Kafka

This means a stackable version >= 23.1 has to be used for the product image.

spec:
  ...
  zookeeperConfigMapName: simple-kafka-znode
  config:
    authentication:
      - authenticationClass: kafka-client-auth-tls
    tls:
      secretClass: tls
    clientAuthentication:
      authenticationClass: kafka-client-auth-tls
    internalTls:
      secretClass: kafka-internal-tls

Changes to:

spec:
  ...
  clusterConfig:
    authentication:
      - authenticationClass: kafka-client-auth-tls
    tls:
      internalSecretClass: kafka-internal-tls
      serverSecretClass: tls
    zookeeperConfigMapName: simple-kafka-znode

Stackable Operator for Apache Nifi

This means a stackable version >= 23.1 has to be used for the product image.

Stackable Operator for Trino

This means a stackable version >= 23.1 has to be used for the product image.

This changes the secret definition from:

stringData:
  LDAP_USER: cn=admin,dc=example,dc=org
  LDAP_PASSWORD: admin

to:

stringData:
  user: cn=admin,dc=example,dc=org
  password: admin

Stackable Operator for Apache Zookeeper

Similar to the Kafka example above, the configuration settings are consolidated under .spec i.e. from:

  config:
    tls:
      secretClass: tls
    clientAuthentication:
      authenticationClass: zk-client-tls
    quorumTlsSecretClass: tls

to:

  clusterConfig:
    authentication:
      - authenticationClass: zk-client-tls
    tls:
      serverSecretClass: tls
      quorumSecretClass: tls

Upgrade from 22.11

Using stackablectl

You can list the available releases as follows

$ stackablectl release list
RELEASE            RELEASE DATE   DESCRIPTION
23.1               2023-01-27     Fourth release focusing on image selection and logging
22.11              2022-11-08     Third release focusing on resource management
22.09              2022-09-09     Second release focusing on security and OpenShift support
22.06              2022-06-30     First official release of the Stackable Data Platform

To uninstall the 22.11 release run

$ stackablectl release uninstall 22.11
[INFO ] Uninstalling release 22.11
[INFO ] Uninstalling airflow operator
[INFO ] Uninstalling commons operator
# ...

Afterwards you will need to update the CustomResourceDefinitions (CRDs) installed by the Stackable Platform. The reason for this is that helm will uninstall the operators but not the CRDs.

$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/airflow-operator/23.1.0/deploy/helm/airflow-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/commons-operator/23.1.0/deploy/helm/commons-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/druid-operator/23.1.0/deploy/helm/druid-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/hbase-operator/23.1.0/deploy/helm/hbase-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/hdfs-operator/23.1.0/deploy/helm/hdfs-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/hive-operator/23.1.0/deploy/helm/hive-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/kafka-operator/23.1.0/deploy/helm/kafka-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/nifi-operator/23.1.0/deploy/helm/nifi-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/opa-operator/23.1.0/deploy/helm/opa-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/secret-operator/23.1.0/deploy/helm/secret-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/spark-k8s-operator/23.1.0/deploy/helm/spark-k8s-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/superset-operator/23.1.0/deploy/helm/superset-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/trino-operator/23.1.0/deploy/helm/trino-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/zookeeper-operator/23.1.0/deploy/helm/zookeeper-operator/crds/crds.yaml

To install the 23.1 release run

$ stackablectl release install 23.1
[INFO ] Installing release 23.1
[INFO ] Installing airflow operator in version 23.1.0
[INFO ] Installing commons operator in version 23.1.0
[INFO ] Installing druid operator in version 23.1.0
[INFO ] Installing hbase operator in version 23.1.0
[INFO ] Installing hdfs operator in version 23.1.0
[INFO ] Installing hive operator in version 23.1.0
[INFO ] Installing kafka operator in version 23.1.0
[INFO ] Installing listener operator in version 23.1.0
[INFO ] Installing nifi operator in version 23.1.0
[INFO ] Installing opa operator in version 23.1.0
[INFO ] Installing secret operator in version 23.1.0
[INFO ] Installing spark-k8s operator in version 23.1.0
[INFO ] Installing superset operator in version 23.1.0
[INFO ] Installing trino operator in version 23.1.0
[INFO ] Installing zookeeper operator in version 23.1.0
# ...

Using helm

Use helm list to list the currently installed operators.

You can use the following command to uninstall all operators that are part of the release 22.11:

$ helm uninstall airflow-operator commons-operator druid-operator hbase-operator hdfs-operator hive-operator kafka-operator nifi-operator opa-operator secret-operator spark-k8s-operator superset-operator trino-operator zookeeper-operator
release "airflow-operator" uninstalled
release "commons-operator" uninstalled
# ...

Afterwards you will need to update the CustomResourceDefinitions (CRDs) installed by the Stackable Platform. This is because helm will uninstall the operators but not the CRDs.

$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/airflow-operator/23.1.0/deploy/helm/airflow-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/commons-operator/23.1.0/deploy/helm/commons-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/druid-operator/23.1.0/deploy/helm/druid-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/hbase-operator/23.1.0/deploy/helm/hbase-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/hdfs-operator/23.1.0/deploy/helm/hdfs-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/hive-operator/23.1.0/deploy/helm/hive-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/kafka-operator/23.1.0/deploy/helm/kafka-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/nifi-operator/23.1.0/deploy/helm/nifi-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/opa-operator/23.1.0/deploy/helm/opa-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/secret-operator/23.1.0/deploy/helm/secret-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/spark-k8s-operator/23.1.0/deploy/helm/spark-k8s-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/superset-operator/23.1.0/deploy/helm/superset-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/trino-operator/23.1.0/deploy/helm/trino-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/zookeeper-operator/23.1.0/deploy/helm/zookeeper-operator/crds/crds.yaml

To install the release 23.1 run

$ helm repo add stackable-stable https://repo.stackable.tech/repository/helm-stable/
$ helm repo update stackable-stable
$ helm install --wait airflow-operator stackable-stable/airflow-operator --version 23.1.0
$ helm install --wait commons-operator stackable-stable/commons-operator --version 23.1.0
$ helm install --wait druid-operator stackable-stable/druid-operator --version 23.1.0
$ helm install --wait hbase-operator stackable-stable/hbase-operator --version 23.1.0
$ helm install --wait hdfs-operator stackable-stable/hdfs-operator --version 23.1.0
$ helm install --wait hive-operator stackable-stable/hive-operator --version 23.1.0
$ helm install --wait kafka-operator stackable-stable/kafka-operator --version 23.1.0
$ helm install --wait listener-operator stackable-stable/listener-operator --version 23.1.0
$ helm install --wait nifi-operator stackable-stable/nifi-operator --version 23.1.0
$ helm install --wait opa-operator stackable-stable/opa-operator --version 23.1.0
$ helm install --wait secret-operator stackable-stable/secret-operator --version 23.1.0
$ helm install --wait spark-k8s-operator stackable-stable/spark-k8s-operator --version 23.1.0
$ helm install --wait superset-operator stackable-stable/superset-operator --version 23.1.0
$ helm install --wait trino-operator stackable-stable/trino-operator --version 23.1.0
$ helm install --wait zookeeper-operator stackable-stable/zookeeper-operator --version 23.1.0

Release 22.11

This is the third release of the Stackable Data Platform, which this time focuses on resource management.

New platform features

The following new major platform features were added:

CPU and memory limits configurable

The operators now request resources from Kubernetes for the products and required CPU and memory can now also be configured for all products. If your product instances are less performant after the update, the new defaults might be set too low and we recommend to set custom requests for your cluster.

Orphaned Resources

The operators now properly clean up after scaling down products. This means for example deleting StatefulSets that were left over after scaling down.

New Versions

New product versions are supported.

Product features

Additionally there are some individual product features that are noteworthy

Supported Kubernetes versions

This release supports the following Kubernetes versions:

  • 1.25 (new)

  • 1.24

  • 1.23

  • 1.22

Upgrade from 22.09

Using stackablectl

You can list the available releases as follows

$ stackablectl release list
RELEASE            RELEASE DATE   DESCRIPTION
22.11              2022-11-08     Third release focusing on resource management
22.09              2022-09-09     Second release focusing on security and OpenShift support
22.06              2022-06-30     First official release of the Stackable Data Platform

To uninstall the 22.09 release run

$ stackablectl release uninstall 22.09
[INFO ] Uninstalling release 22.09
[INFO ] Uninstalling airflow operator
[INFO ] Uninstalling commons operator
# ...

Afterwards you will need to update the CustomResourceDefinitions (CRDs) installed by the Stackable Platform. The reason for this is that helm will uninstall the operators but not the CRDs.

$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/airflow-operator/0.6.0/deploy/helm/airflow-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/commons-operator/0.4.0/deploy/helm/commons-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/druid-operator/0.8.0/deploy/helm/druid-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/hbase-operator/0.5.0/deploy/helm/hbase-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/hdfs-operator/0.6.0/deploy/helm/hdfs-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/hive-operator/0.8.0/deploy/helm/hive-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/kafka-operator/0.8.0/deploy/helm/kafka-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/nifi-operator/0.8.0/deploy/helm/nifi-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/opa-operator/0.11.0/deploy/helm/opa-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/secret-operator/0.6.0/deploy/helm/secret-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/spark-k8s-operator/0.6.0/deploy/helm/spark-k8s-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/superset-operator/0.7.0/deploy/helm/superset-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/trino-operator/0.8.0/deploy/helm/trino-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/zookeeper-operator/0.12.0/deploy/helm/zookeeper-operator/crds/crds.yaml

To install the 22.11 release run

$ stackablectl release install 22.11
[INFO ] Installing release 22.11
[INFO ] Installing airflow operator in version 0.6.0
[INFO ] Installing commons operator in version 0.4.0
[INFO ] Installing druid operator in version 0.8.0
[INFO ] Installing hbase operator in version 0.5.0
[INFO ] Installing hdfs operator in version 0.6.0
[INFO ] Installing hive operator in version 0.8.0
[INFO ] Installing kafka operator in version 0.8.0
[INFO ] Installing nifi operator in version 0.8.0
[INFO ] Installing opa operator in version 0.11.0
[INFO ] Installing secret operator in version 0.6.0
[INFO ] Installing spark-k8s operator in version 0.6.0
[INFO ] Installing superset operator in version 0.7.0
[INFO ] Installing trino operator in version 0.7.0
[INFO ] Installing zookeeper operator in version 0.12.0
# ...

Using helm

Use helm list to list the currently installed operators.

You can use the following command to uninstall all of the operators that are part of the release 22.09:

$ helm uninstall airflow-operator commons-operator druid-operator hbase-operator hdfs-operator hive-operator kafka-operator nifi-operator opa-operator secret-operator spark-k8s-operator superset-operator trino-operator zookeeper-operator
release "airflow-operator" uninstalled
release "commons-operator" uninstalled
# ...

Afterwards you will need to update the CustomResourceDefinitions (CRDs) installed by the Stackable Platform. This is because helm will uninstall the operators but not the CRDs.

$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/airflow-operator/0.6.0/deploy/helm/airflow-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/commons-operator/0.4.0/deploy/helm/commons-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/druid-operator/0.8.0/deploy/helm/druid-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/hbase-operator/0.5.0/deploy/helm/hbase-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/hdfs-operator/0.6.0/deploy/helm/hdfs-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/hive-operator/0.8.0/deploy/helm/hive-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/kafka-operator/0.8.0/deploy/helm/kafka-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/nifi-operator/0.8.0/deploy/helm/nifi-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/opa-operator/0.11.0/deploy/helm/opa-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/secret-operator/0.6.0/deploy/helm/secret-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/spark-k8s-operator/0.6.0/deploy/helm/spark-k8s-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/superset-operator/0.7.0/deploy/helm/superset-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/trino-operator/0.8.0/deploy/helm/trino-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/zookeeper-operator/0.12.0/deploy/helm/zookeeper-operator/crds/crds.yaml

To install the release 22.11 run

$ helm repo add stackable-stable https://repo.stackable.tech/repository/helm-stable/
$ helm repo update stackable-stable
$ helm install --wait airflow-operator stackable-stable/airflow-operator --version 0.6.0
$ helm install --wait commons-operator stackable-stable/commons-operator --version 0.4.0
$ helm install --wait druid-operator stackable-stable/druid-operator --version 0.8.0
$ helm install --wait hbase-operator stackable-stable/hbase-operator --version 0.5.0
$ helm install --wait hdfs-operator stackable-stable/hdfs-operator --version 0.6.0
$ helm install --wait hive-operator stackable-stable/hive-operator --version 0.8.0
$ helm install --wait kafka-operator stackable-stable/kafka-operator --version 0.8.0
$ helm install --wait nifi-operator stackable-stable/nifi-operator --version 0.8.0
$ helm install --wait opa-operator stackable-stable/opa-operator --version 0.11.0
$ helm install --wait secret-operator stackable-stable/secret-operator --version 0.6.0
$ helm install --wait spark-k8s-operator stackable-stable/spark-k8s-operator --version 0.6.0
$ helm install --wait superset-operator stackable-stable/superset-operator --version 0.7.0
$ helm install --wait trino-operator stackable-stable/trino-operator --version 0.7.0
$ helm install --wait zookeeper-operator stackable-stable/zookeeper-operator --version 0.12.0

Breaking changes

You will need to adapt your existing CRDs due to the following breaking changes:

Stackable Operator for Apache Spark

The configuration of pod resource requests has been changed to be consistent with other operators that are part of the Stackable Data Platform (#174).

In the previous version, these were configured like this:

  driver:
    cores: 1
    coreLimit: "1200m"
    memory: "512m"`

From now on, Pod resources can be configured in two different ways. The first and recommended way is to add a resources section for each role as the following examples shows:

  driver:
    resources:
      cpu:
        min: "1"
        max: "1500m"
      memory:
        limit: "1Gi"

The second method is to use the sparkConf section and and set them individually as spark properties:

  sparkConf:
    spark.kubernetes.submission.waitAppCompletion: "false"
    spark.kubernetes.driver.pod.name: "resources-sparkconf-driver"
    spark.kubernetes.executor.podNamePrefix: "resources-sparkconf"
    spark.kubernetes.driver.request.cores: "2"
    spark.kubernetes.driver.limit.cores: "3"

When both methods are used, the settings in the sparkConf section override the resources configuration.

Note that none of the settings above have any influence over the parallelism used by Spark itself. The only supported way to affect this is as follows:

  sparkConf:
    spark.driver.cores: "3"
    spark.executor.cores: "3"

Release 22.09

This is the second release of the Stackable Data Platform. It contains lots of new features and bugfixes. The main features focus on OpenShift support and security.

New platform features

The following new major platform features were added:

OpenShift compatibility

We have made continued progress towards OpenShift compability, and the following operators can now be previewed on OpenShift. Further improvements are expected in future releases, but no stability or compatibility guarantees are currently made for OpenShift clusters.

Support for internal and external TLS

The following operators support operating the products at a maximal level of transport security by using TLS certificates to secure internal and external communication:

LDAP authentication

Use a central LDAP server to manage all of your user identities in a single place. The following operators added support for LDAP authentication:

stackablectl

stackablectl now supports deploying ready-to-use demos, which give an end-to-end demonstration of the usage of the Stackable Data Platform. The quickstart guide shows how to get started with stackablectl. Here you can see the available demos.

Supported Kubernetes versions

This release supports the following Kubernetes versions:

  • 1.24

  • 1.23

  • 1.22

Support for 1.21 was dropped.

Upgrade from 22.06

Using stackablectl

You can list the available releases as follows

$ stackablectl release list
RELEASE            RELEASE DATE   DESCRIPTION
22.11              2022-11-08     Third release candidate of 22.11
22.09              2022-09-09     Second release focusing on security and OpenShift support
22.06              2022-06-30     First official release of the Stackable Data Platform

To uninstall the 22.06 release run

$ stackablectl release uninstall 22.06
[INFO ] Uninstalling release 22.06
[INFO ] Uninstalling airflow operator
[INFO ] Uninstalling commons operator
# ...

Afterwards you will need to update the CustomResourceDefinitions (CRDs) installed by the Stackable Platform. The reason is, that helm will uninstall the operators but not the CRDs.

$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/airflow-operator/0.5.0/deploy/helm/airflow-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/commons-operator/0.3.0/deploy/helm/commons-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/druid-operator/0.7.0/deploy/helm/druid-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/hbase-operator/0.4.0/deploy/helm/hbase-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/hdfs-operator/0.5.0/deploy/helm/hdfs-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/hive-operator/0.7.0/deploy/helm/hive-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/kafka-operator/0.7.0/deploy/helm/kafka-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/nifi-operator/0.7.0/deploy/helm/nifi-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/opa-operator/0.10.0/deploy/helm/opa-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/secret-operator/0.5.0/deploy/helm/secret-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/spark-k8s-operator/0.5.0/deploy/helm/spark-k8s-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/superset-operator/0.6.0/deploy/helm/superset-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/trino-operator/0.6.0/deploy/helm/trino-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/zookeeper-operator/0.11.0/deploy/helm/zookeeper-operator/crds/crds.yaml

To install the 22.09 release run

$ stackablectl release install 22.09
[INFO ] Installing release 22.09
[INFO ] Installing airflow operator in version 0.5.0
[INFO ] Installing commons operator in version 0.3.0
[INFO ] Installing druid operator in version 0.7.0
[INFO ] Installing hbase operator in version 0.4.0
[INFO ] Installing hdfs operator in version 0.5.0
[INFO ] Installing hive operator in version 0.7.0
[INFO ] Installing kafka operator in version 0.7.0
[INFO ] Installing nifi operator in version 0.7.0
[INFO ] Installing opa operator in version 0.10.0
[INFO ] Installing secret operator in version 0.5.0
[INFO ] Installing spark-k8s operator in version 0.5.0
[INFO ] Installing superset operator in version 0.6.0
[INFO ] Installing trino operator in version 0.6.0
[INFO ] Installing zookeeper operator in version 0.11.0
# ...

Using helm

Use helm list to list the currently installed operators.

You can use the following command to uninstall all of the operators that are part of the release 22.06:

$ helm uninstall airflow-operator commons-operator druid-operator hbase-operator hdfs-operator hive-operator kafka-operator nifi-operator opa-operator secret-operator spark-k8s-operator superset-operator trino-operator zookeeper-operator
release "airflow-operator" uninstalled
release "commons-operator" uninstalled
# ...

Afterwards you will need to update the CustomResourceDefinitions (CRDs) installed by the Stackable Platform. The reason is, that helm will uninstall the operators but not the CRDs.

$ kubectl apply \
  -f https://raw.githubusercontent.com/stackabletech/airflow-operator/0.5.0/deploy/helm/airflow-operator/crds/crds.yaml \
  -f https://raw.githubusercontent.com/stackabletech/commons-operator/0.3.0/deploy/helm/commons-operator/crds/crds.yaml \
  -f https://raw.githubusercontent.com/stackabletech/druid-operator/0.7.0/deploy/helm/druid-operator/crds/crds.yaml \
  -f https://raw.githubusercontent.com/stackabletech/hbase-operator/0.4.0/deploy/helm/hbase-operator/crds/crds.yaml \
  -f https://raw.githubusercontent.com/stackabletech/hdfs-operator/0.5.0/deploy/helm/hdfs-operator/crds/crds.yaml \
  -f https://raw.githubusercontent.com/stackabletech/hive-operator/0.7.0/deploy/helm/hive-operator/crds/crds.yaml \
  -f https://raw.githubusercontent.com/stackabletech/kafka-operator/0.7.0/deploy/helm/kafka-operator/crds/crds.yaml \
  -f https://raw.githubusercontent.com/stackabletech/nifi-operator/0.7.0/deploy/helm/nifi-operator/crds/crds.yaml \
  -f https://raw.githubusercontent.com/stackabletech/opa-operator/0.10.0/deploy/helm/opa-operator/crds/crds.yaml \
  -f https://raw.githubusercontent.com/stackabletech/secret-operator/0.5.0/deploy/helm/secret-operator/crds/crds.yaml \
  -f https://raw.githubusercontent.com/stackabletech/spark-k8s-operator/0.5.0/deploy/helm/spark-k8s-operator/crds/crds.yaml \
  -f https://raw.githubusercontent.com/stackabletech/superset-operator/0.6.0/deploy/helm/superset-operator/crds/crds.yaml \
  -f https://raw.githubusercontent.com/stackabletech/trino-operator/0.6.0/deploy/helm/trino-operator/crds/crds.yaml \
  -f https://raw.githubusercontent.com/stackabletech/zookeeper-operator/0.11.0/deploy/helm/zookeeper-operator/crds/crds.yaml

To install the release 22.09 run

$ helm repo add stackable-stable https://repo.stackable.tech/repository/helm-stable/
$ helm repo update stackable-stable
$ helm install --wait airflow-operator stackable-stable/airflow-operator --version 0.5.0
$ helm install --wait commons-operator stackable-stable/commons-operator --version 0.3.0
$ helm install --wait druid-operator stackable-stable/druid-operator --version 0.7.0
$ helm install --wait hbase-operator stackable-stable/hbase-operator --version 0.4.0
$ helm install --wait hdfs-operator stackable-stable/hdfs-operator --version 0.5.0
$ helm install --wait hive-operator stackable-stable/hive-operator --version 0.7.0
$ helm install --wait kafka-operator stackable-stable/kafka-operator --version 0.7.0
$ helm install --wait nifi-operator stackable-stable/nifi-operator --version 0.7.0
$ helm install --wait opa-operator stackable-stable/opa-operator --version 0.10.0
$ helm install --wait secret-operator stackable-stable/secret-operator --version 0.5.0
$ helm install --wait spark-k8s-operator stackable-stable/spark-k8s-operator --version 0.5.0
$ helm install --wait superset-operator stackable-stable/superset-operator --version 0.6.0
$ helm install --wait trino-operator stackable-stable/trino-operator --version 0.6.0
$ helm install --wait zookeeper-operator stackable-stable/zookeeper-operator --version 0.11.0

Breaking changes

You will need to adapt your existing CRDs to the following breaking changes:

druid-operator

  1. HDFS deep storage is now configurable via the HDFS discovery config map instead of a url to a HDFS name node (#262). Instead of

  deepStorage:
    hdfs:
      storageDirectory: hdfs://druid-hdfs-namenode-default-0:8020/data

use

  deepStorage:
    hdfs:
      configMapName: druid-hdfs
      directory: /druid

kafka-operator

  1. Add TLS encryption and authentication support for internal and client communications. This is breaking for clients because the cluster is secured per default, which results in a client port change (#442). If you don’t want to use TLS to secure your Kafka cluster you can restore the old behavior by using the tls attribute as follows:

apiVersion: kafka.stackable.tech/v1alpha1
kind: KafkaCluster
# ...
spec:
  config:
    tls: null
  # ...

trino-operator

  1. TrinoCatalogs now have their own CRD object and get referenced by the TrinoCluster (#263). Instead of

apiVersion: trino.stackable.tech/v1alpha1
kind: TrinoCluster
# ...
spec:
  hiveConfigMapName: hive
  s3:
    inline:
      host: minio
      port: 9000
      accessStyle: Path
      credentials:
        secretClass: s3-credentials
  # ...

use

apiVersion: trino.stackable.tech/v1alpha1
kind: TrinoCluster
# ...
spec:
  catalogLabelSelector:
    trino: trino
  # ...
---
apiVersion: trino.stackable.tech/v1alpha1
kind: TrinoCatalog
metadata:
  name: hive
  labels:
    trino: trino
spec:
  connector:
    hive:
      metastore:
        configMap: hive
      s3:
        inline:
          host: minio
          port: 9000
          accessStyle: Path
          credentials:
              secretClass: s3-credentials

Release 22.06

This is our first release of the Stackable Data Platform, bringing Kubernetes operators for 12 products as well as stackablectl, the commandline tool to easily install data products in Kubernetes. Operators spin up production ready product applications. Also, there are some common features across all operators, such as monitoring, service discovery and configuration overrides. Find the Platform features, stackablectl features and Operators below.

Please report any issues you find in the specific operator repositories or in our dedicated github.com/stackabletech/issues/[issues] repository. You may also join us in our Slack community or contact us via our homepage.

While we are very proud of this release it is our first one and we’ll add new features and fix bugs all the time and will have regular releases from now on.

Platform features

Easily install production ready data applications

Using a familiar declarative approach, users can easily install data applications such as Apache Kafka or Trino across multiple cloud Kubernetes providers or on their own data centers. The installation process is fully automated while also providing the flexibility for the user to tune relevant aspects of each application.

Monitoring

All products have monitoring with prometheus enabled. Learn more

Service discovery

Products on the Stackable platform use service discovery to easily interconnect with each other. Learn more

Configuration overrides

All operators support configuration overrides, these are documented in the specific operator documentation pages.

Common S3 configuration

Many products support connecting to S3 to load and/or store data. There is a common resource for S3 connections and buckets across all operators that can be reused. Learn more

Roles and role groups

To support hybrid hardware clusters, the Stackable platform uses the concept of role groups. Services and applications can be configured to maximize hardware efficiency.

Standardized

Learn once reuse everywhere. We use the same conventions in all our operators. Configure your LDAP or S3 connections once and reuse them everywhere. All our operators reuse the same CRD structure as well.

stackablectl

stackablectl is used to install and interact with the operators, either individually or with multiple at once.

Operators

This is the list of all operators in this current release, with their versions for this release.

Products

Read up on the supported versions for each of these products.

Supporting operators

Supported Kubernetes versions

This release supports the following Kubernetes versions:

  • 1.23

  • 1.22

  • 1.21