Project overview

At the core of the Stackable Platform are the Kubernetes operators used to install and manage various data products, like the nifi-operator for example. You can find all of the operators if you search the organization repositories.

All the operators are written in Rust and the source code is found in the rust directory. tests contains the integration tests which use kuttl and our own test template beku.py. Documentation is written in Antora and found in the docs directory, see also Documentation further down the page. deploy and docker contain files used to package the operator into a Docker image and Helm chart. Some files in these repositories are actually templated:

The operator-templating repository contains a template for all operator repositories, where shared files are distributed from. You can read the README in that repository to find out more about the details. Whenever common files are changed, a GitHub action is used to distribute the changes to all operator repositories.

The operator-rs repository contains the common framework library for all operators. It is a Rust library that is used by all operators and contains shared structs and shared functionality.

The docker-images repository contains Dockerfiles for all the products that are supported by the SDP. The actual product artifacts are pulled from the product artifacs store and packaged into images. The images are pushed into an image registry.

The stackablectl commandline tool and the Stackable Cockpit UI are both found in the stackable-cockpit repository, and they both share some code. The structure of the repository is documented in its README.

The documentation is built with Antora and the playbook file to build it is located in the documentation repository, among some common platform documentation. The UI for the documentation is found in the documentation-ui repository; it is included as a submodule in the documentation repository. The documentation pulls in operator documentation files from the operator repositories. The documentation is found at https://docs.stackable.tech/.

There is also https://crds.stackable.tech/ where you can find generated documentation for all the CustomResourceDefinitions on the platform. The code to generate this page is found in the crddocs repository.

On top of the mentioned repositories, there are various smaller tools or product extensions that Stackable developed, they are all linked to from relevant places. Two examples are the image-tools used to build Docker images and the druid-opa-authorizer which is a Druid extension that enables OPA support for Druid.

T2 - Test & Troubleshoot Platform is used for integration testing across different versions and cloud providers, find more information in the README of the repository.

The issues repository exists solely for the purpose of tracking issues related to the Stackable Platform in general. Large topics that impact many or even all of the platform components are discussed here. There is no code in this repository.

A lot of artifacts are stored in the OCI registry. Currently, those artifacts can be browsed by calling the API.

The following command lists all the different projects the artifacts are distributed across:

sdp contains the product and operator Docker images. The Helm Charts for the operators are found under sdp-charts. Some artifacts like the product binaries are stored in the Nexus repo under packages.

List the Helm Charts in sdp-charts:

List all available versions of a specific Helm Chart, e.g. the zookeeper-operator:

Docker images are stored in https://oci.stackable.tech as mentioned above. To list all the available repositories in a project, for example in the sdp project, run this command:

Since the API is using pagination it might be necessary to increment the page in the command to see additional results.

It is also possible to list all the available image tags for a repository of choice. The following command shows the tags of the trino repository:

Similar to the previous command, the API call uses pagination again. So the page value in the command can be incremented to see more results. Here the page_size parameter was also used to increase the results per page.

Another possibility, instead of using curl, would be the crane tool, which can also be used to browse the tags when given the path to a repository.