Stackable Data Platform (SDP) Release 24.7 is now publicly available!
Security-first: Vulnerability Management
In this new 24.7 release we continue the security theme from our 24.3 release!
While we have new features as well (more on that below) by far the biggest focus was on vulnerability management. There are many good reasons for this:
- First and foremost we want our product to be as secure as possible, getting rid of known vulnerabilities is one big step in that direction
- Trust: Having robust vulnerability management procedures in places enhances our customers’ trust in our product, knowing that they have a partner in Stackable they can trust
- Regulatory Compliance: Many of our users and customers operate in regulated environments and we’re doing our part to help them.
- We are preparing ourselves for the upcoming Cyber Resilience Act (hear our co-founder talk about this at the upcoming Bitkom Forum Open Source in September) which requires robust vulnerability management
As of 24.7 we now build all our Java-based products straight from source which allows us to apply custom patches to bump dependencies, add features or fix bugs. We also updated our Docker base images from UBI8 to UBI9 which gives us a newer base image with more recent dependency versions.
Behind the scenes this work is informed by a workflow centered around a SecObserve instance which we will soon also be using to publish advisories (VEX) in the machine-readable CSAF format. Stay tuned!
Remember: As of our last release (24.3) we do publish SBOMs in the CycloneDX format.
Check out our last release notes or use the SBOM Browser to download the latest ones.
The result of all this is a reduction of vulnerabilities found by 75% compared between 24.3 and 24.7!
The more platforms the better
Stackable is now available on ARM64 architectures as well!
This was possible due to a collaboration between Stackable, Ampere and Hewlett Packard Enterprise. Thank you for your support in this. We received a HPE ProLiant RL300 Gen11 as a loaner to test our stack extensively on ARM.
We had previews of this functionality in past releases but they required manual work and overrides. This release should not require you to do anything special anymore. Even mixed-architecture clusters work!
As this is still brand new we still consider this experimental and will iterate on the workflows and testing for future versions but it is absolutely ready to give it a spin, so go ahead and do just that!
New Product-Specific Features
These are some new product-specific features added in 24.7. More details can be found in the release notes.
- Apache Airflow adds support for modularized DAGs
- Apache Druid now supports OIDC for authentication joining Apache Superset and Trino
- Apache HBase
- 2.6.0 is included in an experimental version including an equally experimental authorizer for Open Policy Agent meaning you can build your authorization rules for HBase in code
- We include the necessary JARs to access S3 and include a script to export snapshots to S3
- Secret Operator:
- As another precautionary security measure the default CA lifetime is reduced to one year
- The operator now logs when secrets are created, making debugging easier
- Trino adds support for row-level filtering and column masks in Rego rules
- Open Policy Agent now supports auditing by enabling logging of all decisions
Check out the release notes for all the nitty-gritty details. And our CRD documentation has also been updated with 24.7 goodness.
Product Version changes
This table lists the changes to the product versions. According to our lifecycle policy we’ll support LTS release lines for at least one year. But we’ll always provide a migration path.
Example: If you’re on Airflow 2.6.3 today you can upgrade SDP to 24.7 and then migrate to Airflow 2.9.2 in the coming weeks or months. The next version of SDP (probably 24.11) will then remove 2.6 support entirely.
| Product | New version/s | Removed | Deprecated | Notes |
|---|---|---|---|---|
| Airflow | 2.8.4 2.9.2 (new LTS) | 2.7.2 2.7.3 | 2.6.3 (old LTS) 2.8.1 2.8.4 | This version moves from the 2.6 line to 2.9 for LTS support. |
| Druid | 30.0.0 (experimental) | 27.0.0 | 28.0.1 | We see ingestion issues with 30.0.0 which is why we will not fully support it yet |
| HBase | 2.4.18 2.6.0 (experimental) | 2.4.17 | We plan to make 2.6 the new LTS release line in the next release | |
| HDFS | 3.4.0 (experimental) | We plan to make 3.4 the new LTS release line in the next release | ||
| Kafka | 3.6.2 3.7.1 (new LTS) | 3.4.1 (old LTS) 3.6.1 | ||
| NiFi | 1.27.0 (new LTS) 2.0.0-M4 | 1.23.2 | 1.21.0 (old LTS) | |
| OpenPolicyAgent | 0.66.0 | 0.57.0 | 0.61.0 | |
| Spark | 3.4.3 3.5.1 (new LTS) | 3.4.1 | 3.4.2 3.4.3 (old LTS) | |
| Superset | 3.1.3 4.0.2 (new LTS) | 2.1.1 3.0.1 3.0.3 | 2.1.3 3.1.0 3.1.3 | |
| Trino | 451 (new LTS) | 428 | 414 (old LTS) 442 | |
| ZooKeeper | 3.8.3 | 3.8.4 | 3.9 is the new LTS release line |
More Info
Further details on our release and how to upgrade can be found in our release notes as well as in the change logs of the individual operators:
Airflow, Druid, HBase, HDFS, Kafka, NiFi, OpenPolicyAgent, Spark, Superset, Trino, ZooKeeper