Apache Spark™ on the
Stackable Data Platform

Start by following the Getting Started guide in the operator documentation. Create a SparkApplication CRD that specifies the container image, main application file, executor and driver resources, and any necessary environment variables, then apply it with kubectl apply. The operator automatically provisions all required Kubernetes resources, schedules the job and manages its lifecycle, including log collection and event tracking.

The supported Apache Spark versions are listed in the Stackable documentation. Pick one that matches a compatible Spark image for your deployment, or bring your own image from a custom registry for patched versions or extra libraries. The operator runs multiple versions concurrently, so you can adopt new releases at your own pace.
Click here for the current list.

Yes, by specifying different container images for each SparkApplication. Each application runs independently, so multiple versions of Spark can coexist in the same Kubernetes cluster without conflicts. This is useful for testing new versions, migrating workloads gradually, or supporting teams with different dependency requirements.

Dependencies can be added in several ways: custom container images that include libraries or packages; mounted volumes containing JARs or configuration files; Maven coordinates for Java/Scala libraries (Iceberg, Hadoop connectors, etc.); and Python packages via pip for PySpark jobs. The operator documentation provides examples for each method, ensuring reproducibility and version control for dependencies.

Each driver and executor pod includes a Vector sidecar that collects logs in a structured, centralized format. Logs can also be written to S3 for long-term storage. The Spark History Server can read event logs from S3, letting you browse completed applications, view metrics and debug past runs easily – ensuring operational transparency and simplifying monitoring and troubleshooting.

Yes. You can configure S3 as a data source via the CRD, specifying credentials through Secrets and SecretClasses. Spark can read and write directly from these object stores, enabling integration with data lakes and cloud-native storage systems without additional ETL layers.

Yes. The operator documentation provides guidance for configuring Kerberos authentication for HDFS access. Stackable’s Secret Operator can generate keytabs and manage credentials securely. TLS and RBAC configurations ensure that workloads run securely within multi-tenant clusters.

Yes. Spark combines with other Stackable operators for end-to-end workflows: trigger Spark jobs from Airflow for orchestrated ETL pipelines, query Lakehouse tables with Trino after Spark transformations, and visualize aggregated results with Superset. Integration patterns are documented in the Stackable platform guides for reliable, scalable, maintainable analytics pipelines.

Yes. You can define resource requests and limits per role, such as driver or executor pods, and even per role-group. This allows you to optimize CPU, memory, and storage allocation for each workload type, ensuring high utilization without over-provisioning. Resource tuning also helps maintain cluster stability under mixed workloads and heavy usage.

The operator has been tested on major managed and self-hosted Kubernetes platforms: EKS, AKS, GKE, OpenShift, IONOS and K3s. This flexibility lets you deploy Spark reliably across cloud providers or on-premise infrastructure, while still benefiting from declarative management and operator automation.