Stackable Operator for Apache Kafka
Apache Kafka® on the
Stackable Data Platform
Run Kafka natively on Kubernetes
Automate and scale your event streaming
Events don’t wait. Payments, sensor readings, clicks, transactions – they arrive continuously, and something has to move them reliably between every system that cares. Apache Kafka is that backbone, and the Stackable Operator automates its deployment, scaling and management on Kubernetes – including OPA-based authorization, TLS, and unified monitoring through Prometheus and Grafana. Run it on-prem, air-gapped or in a sovereign cloud – your data stays yours.
Why Apache Kafka in the Stackable Data Platform (SDP)?
Apache Kafka is a distributed event streaming platform for high-performance, real-time data pipelines. Applications publish, store and consume streams of records reliably to and from Kafka. In the Stackable Data Platform, Kafka is the backbone of event-driven architectures – connecting producers and consumers across your infrastructure with scalability, durability and low latency.
High-throughput, low-latency
Millions of messages per second with minimal delay – ideal for real-time analytics, monitoring and ETL.
Scalable & resilient
Scale horizontally by simply adding brokers or partitions; replication across brokers provides fault tolerance and guaranteed delivery, keeping streams reliable even during failures.
Stream processing integration
Works directly with Spark and Trino to build real-time processing and enrichment pipelines.
Secure by design
End-to-end TLS encryption, SASL authentication and role-based access control provide a secure messaging backbone.
What does the Stackable Operator add for Apache Kafka?
Zero-config connections across the platform
The operator publishes a ConfigMap with all broker connection details and listener endpoints. Clients and other Stackable components (NiFi, Spark, Trino) connect without manual configuration, ready for end-to-end streaming pipelines.
Run both Kafka 3 and Kafka 4
Run legacy ZooKeeper-based Kafka 3 clusters and Kafka 4 clusters in KRaft mode with the same operator, adopting KRaft at your own pace.
Role-based broker topologies
Define broker roles and role groups to tailor configuration for different workloads within a single cluster.
What do all Stackable operators have in common?
One operator framework (Rust)
Every operator follows the same CRD pattern with Roles, RoleGroups and ConfigOverrides. Learn one operator, and you find your way around the next one instantly.
Infrastructure-as-Code
Every data app is a YAML CRD that lives in Git – reviewable, lintable and CI/CD-ready. The same definition works on Dev, Test and Prod.
Lifecycle management (Day-2 operations)
Deployment, restarts, certificate rotation and rolling upgrades are handled automatically – including pod restarts when configs or secrets change. And because you need to see what’s running, monitoring and logging come built in: Prometheus, Vector and OpenTelemetry feed ready-made Grafana dashboards – one observability setup for the whole platform.
Security by default
TLS, Kerberos for HDFS, OIDC login and OPA-based fine-grained authorization, all configurable per CRD. Daily vulnerability scans, SBOMs and cosign-signed images keep the whole platform patched – so you don’t track a dozen upstream projects yourself.
Kubernetes-native and modular
Runs on-prem, in any cloud or on a laptop, with no vendor lock-in – and explicit air-gapped support. Install only the operators you need, and add more later without touching the rest.
OpenShift-certified
All operators are Red Hat–certified and install directly from the Red Hat Certified Operators Catalog (OperatorHub) – compatible with Security Context Constraints and RBAC. Certified operation via a Stackable subscription.
Use Cases for Kafka
Event-driven architectures
Build modern applications on Kafka – from fraud detection to supply chain management – with reliable, low-latency communication between microservices. Stackable simplifies cluster operations and keeps things highly available.
Data streaming & integration
Kafka is the backbone for pipelines – collecting events from apps, sensors and services, then distributing them to Spark, HBase or OpenSearch. With Stackable, clusters scale easily to handle peak demand.
IoT & edge data ingestion
Telecom, automotive and energy firms rely on Kafka to ingest continuous streams from IoT devices. Stackable makes it easy to run resilient clusters that route data securely from the edge to central platforms.
FAQ - Frequently Asked Questions about Kafka and Stackable
The currently supported and tested Apache Kafka versions are listed on the Stackable documentation’s supported-versions page, kept up to date as new releases are validated and older ones retired. You select a version via the image in your KafkaCluster, with custom images supported where needed.
Click here for the current list.
The recommended approach is stackablectl: run `stackablectl operator install kafka` to install the latest released version of the Kafka operator, then deploy a KafkaCluster Custom Resource to create and manage your cluster declaratively. Helm is also supported if you prefer managing deployments through GitOps or Helm pipelines. On OpenShift, the operator can be installed via the Red Hat Certified Operators Catalog.
Kafka 3 deployments require ZooKeeper for cluster coordination, topic metadata and broker management; ZooKeeper can be deployed using the Stackable ZooKeeper Operator for easy lifecycle management. Kafka 4 deployments use the built-in KRaft cluster coordination algorithm and no longer require Apache ZooKeeper.
The operator automatically generates a ConfigMap that contains all broker connection details and listener endpoints. Client applications – producers, consumers, or other Stackable components (NiFi, Spark, Trino) – can reference this ConfigMap to connect seamlessly to the Kafka cluster without manual configuration.
You can integrate Open Policy Agent (OPA) directly into your Kafka cluster for fine-grained access control. In the KafkaCluster CRD, define an authorization.opa section and reference a policy ConfigMap containing your Rego rules and OPA package. This enables centralized, auditable authorization policies aligned with your organization’s security standards.
TLS can be enabled by defining a tls section in your KafkaCluster resource. The operator automatically provisions and distributes the required certificates across all brokers and clients, ensuring encrypted communication between nodes and external clients. Certificate renewal and rotation are fully automated through Stackable’s integrated certificate management.
Yes. The operator supports overriding Kafka configuration parameters directly through the KafkaCluster CRD. You can customize log levels, broker properties or JVM options (e.g. log4j settings) to match your environment’s operational and performance needs.
The operator has been tested on major managed and self-hosted Kubernetes platforms: EKS, AKS, GKE, OpenShift, IONOS and K3s. This flexibility lets you deploy Kafka reliably across cloud providers or on-premise infrastructure, while still benefiting from declarative management and operator automation.
Resources - Learn how to use the Stackable Operator for Apache Kafka
Getting Started Guide
Step-by-step instructions to deploy your first Kafka cluster with the Stackable Operator.
Technical Documentation
Full reference for configuration, CRDs, security setup and monitoring integration.
GitHub Repository
The official open-source repository for the Stackable Kafka Operator, containing source code, Helm charts and example configurations. Ideal for developers and operators who want to explore configurations, contribute or automate deployments.
Demo Pipelines
Try the Waterlevel or Earthquake demos with a single stackablectl command.
Subscribe to our Newsletter
With the Stackable newsletter, you’ll always stay up to date on the latest from Stackable!
Newsletter
Subscribe to the newsletter
With the Stackable newsletter you’ll always be up to date when it comes to updates around Stackable!