Stackable Data Platform (SDP) Release 23.7 is now publicly available!
Highlights
This release introduces the specification of resource quotas, a Kubernetes-specific feature called “Pod-overrides” and many updates to the versions of products supported by SDP.
Resource Quotas
Explicit resources are now applied to all containers, both operators and products.
This allows users to run the Stackable Data Platform on Kubernetes clusters that have set a ResourceQuota or LimitRange. Where these are not specified directly, defaults will be used. This feature will enable an even more predictable and stable ops experience for the Stackable Data Platform – making it possible to limit the amount of resources for any given component and give Kubernetes more tools to distribute and troubleshoot the workloads.
Pod Overrides
It is now possible to add custom settings which specify elements of a pod template (service, stateteful set etc.) on roles or rolegroups, which the operator then merges with the objects it writes before actually applying them.
This provides the user with a possibility for specifying any property that can be set on a regular Kubernetes Pod, but which is not directly exposed via the Stackable custom resource definition. Have a look at the documentation for more details.
OpenShift Certification
OLM bundles – a pre-requisite for the OpenShift certification process – have been created for each operator. All 15 SDP operators in release 23.4.1 are now OpenShift-certified and deployable directly from within an OpenShift cluster.
OpenShift certification means that our components are built using certified images with the latest security updates. They also contain clear license and identification information, as required by the certification, and are regularly monitored for security issues. This will ensure, that our containers can be deployed consistently and securely across hybrid cloud and on-premise installations via Open Shift.
New Product Versions
The following new product versions are now supported:
- Apache Airflow: 2.6.1
- Apache Druid: 26.0.0
- Apache Kafka: 3.4.0
- Apache NiFi: 1.20.0, 1.21.0
- Opa: 0.51
- Apache Spark: 3.4.0
- Apache Superset: 1.4.2, 1.5.3, 2.0.1, 2.1.0
- Trino: 414
- Apache ZooKeeper: 3.8.1
Each product version brings along new features and capabilities to empower your data team and make sure that you are able to keep up with what’s recent in the world of big data processing.
New Product-Specific Features
We have also added new product-specific features such as:
- HDFS: support for enabling secure mode with Kerberos
- Spark-k8s: support for using custom certificates when accessing S3 with TLS
- Trino: support arbitrary connectors using the generic connector for e.g. access to PostgreSQL
We are really happy to ship Kerberos support! In case you have not worked with it before – Kerberos is the way to secure a HDFS installation. Helping our users make their data platform secure is a major goal of our work at Stackable.
Learning Stackable
Further details on our release and how to upgrade can be found in our release notes as well as in the change logs of the individual operators (e.g. for Airflow).