We are excited that the new Stackable Data Platform 24.11 release is now publicly available! It enhances the platform’s flexibility, performance, and security, ensuring that organizations can continue to scale their platform operations efficiently and securely.
With this new release, the Stackable Data Platform now also supports running on Kubernetes 1.31, 1.30 and 1.29, as well as RedHat Openshift in version 4.15 and 4.14.
New Platform Features
- Authentication:
- Apache Airflow and Apache NiFi with OIDC support.
- Apache Kafka introduces Kerberos support.
- Security:
- TLS certificate provisioning now supported with cert-manager.
- RSA key lengths can be customized for generated key pairs.
- Enhanced Secret Operator permissions for managing listener volumes.
- Listener Updates:
- Kafka connectivity now supports IP addresses and Fully Qualified Domain Names (FQDNs).
- Dependencies:
- Apache HBase now supports Azure Data Lake Storage Gen2 (ADLS).
- Operations:
- HDFS upgrade support with manual intervention.
- Miscellaneous:
- NiFi, Airflow, and Superset now support custom Python code in configuration files.
- Docker images updated for OpenShift security constraints and non-root container setups.
Platform Improvements
- Vulnerabilities: Fixed 142 CVEs, including 11 critical and 55 high-severity.
- Authorization:
- AD connect to look up user groups and use them in OPA Rego Rules.
- Improved HDFS OPA Authorizer performance; replaced User Info Fetcher API with a Rego library.
- Logging:
- Increased NiFi log volume size and reduced log check interval; added option to disable create-reporting-task Job.
- Monitoring:
- Fixed JMX Exporter performance issues for several products.
- Listener:
- Corrected propagation of service annotations in Kafka Listener.
- Miscellaneous:
- Reduced CRD size.
- Bug Fixes:
- Resolved issues with Spark job submissions, Airflow config overrides, Trino credentials, and HBase config and more bug fixes.
New Product Versions
The following new product versions are now supported (get the list of all supported product versions here):
| Product | New version/s | What’s new ? |
|---|---|---|
| Airflow | 2.9.3 (LTS) 2.10.2 (exp) | The experimental version 2.10.2 contains: – Datasets no longer trigger inactive DAGs #38891 – OpenTelemetry Traces for Apache Airflow #37948 – Using Multiple Executors Concurrently #40701 |
| Druid | 30.0.0 (LTS) | Moved from experimental to LTS and includes: – Removed ZooKeeper-based segment loading #15705 – Improved groupBy queries #15757 – Improved AND filter performance #15838 – RabbitMQ extension #14137 – Improved the Supervisors view #16318 – You can now use the Query view to search in tables and columns. #15990 |
| Hive | 4.0.0 (exp) | The experimental version 4.0.0 has various new features. Get the full list at: https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12343343&styleName=Text&projectId=12310843 |
| Kafka | 3.8.0 | Highlights include: – Kafka Tiered Storage V1 KAFKA-15420 – Add JBOD support in tiered storage KAFKA-16399 – KIP-768: Extend support for opaque (i.e. non-JWT) tokens in SASL/OAUTHBEARER KAFKA-15878 – Support setting zstd Compression Level KAFKA-7632 – Implement ConsumerInterceptor KAFKA-15942 |
| NiFi | 2.0.0 (exp) | Update from 2.0.0-M4 to 2.0.0 GA (still experimental) |
| OpenPolicyAgent | 0.67.1 | Bug fix version from 0.67.0 incl. a mix of features and performance improvements. Details at: https://github.com/open-policy-agent/opa/blob/main/CHANGELOG.md#0671 |
| Spark | 3.5.2 (LTS) | Bug fix version |
| Trino | 455 | Highlights include: – Add Exasol connector #16083 – Add support for the euclidean_distance(), dot_product(), and cosine_distance() functions #22397 – Improve performance of queries with selective joins by performing fine-grained filtering of rows using dynamic filters #22411 – Improve performance for column masking with Open Policy Agent access control #21359 – Add support for an event listener plugin that publishes events to Kafka #22426 – ⚠️ Breaking change: Improve query concurrency when memory is low by removing the query.low-memory-killer.delay configuration property #22936 – Add query starting time in QueryStatistics in all Event listeners #23113 – Add support S3 file system encryption with fault-tolerant execution mode #22529 – Fix memory tracking issue for aggregations that could cause worker crashes with out-of-memory errors #23098 |
More Info
Further details on our release and how to upgrade can be found in the release notes as well as in the change logs of the individual operators:
Airflow, Druid, HBase, HDFS, Kafka, NiFi, OpenPolicyAgent, Spark, Superset, Trino, ZooKeeper