Apache ZooKeeper™ is a distributed coordination service that helps clustered applications manage shared configuration, naming, synchronization, and group membership. It provides a consistent, fault-tolerant way for distributed systems to agree on state, elect leaders, and detect failures, making it a foundational piece of many early Big Data architectures. While ZooKeeper remains in active use in several systems, its role is narrowing as modern alternatives mature. This article walks through how it works, where it still fits, and when you should think twice before including it in a new deployment. We look at how the Stackable Data Platform (SDP) approaches this specifically at the end.
How does Apache ZooKeeper actually work?
Apache ZooKeeper is a centralized service that stores small amounts of coordination data in a hierarchical namespace, similar to a file system. Client applications connect to a ZooKeeper ensemble, read and write nodes called znodes, and register watches to be notified when data changes. The ensemble uses a consensus protocol to keep all nodes consistent, so every client sees the same state regardless of which server it connects to.
The ensemble typically runs as an odd number of nodes, most commonly three or five, to ensure a quorum can always be reached. ZooKeeper uses the ZAB (ZooKeeper Atomic Broadcast) protocol to replicate every write to a majority of nodes before acknowledging it to the client. This guarantees linearizability for writes and provides sequential consistency for reads from the same client session.
Znodes can be persistent, ephemeral, or sequential. Ephemeral znodes are deleted automatically when the client session that created them expires, which makes them useful for leader election and failure detection. Applications watching a znode receive a notification the moment it changes, enabling event-driven coordination without polling.
In practice, ZooKeeper is optimized for reads. It is not designed for storing large amounts of data, and write throughput is intentionally limited because every write must go through the leader and be replicated to a quorum. This architecture trades raw throughput for strong consistency guarantees.
What role did ZooKeeper play in Apache Kafka?
Apache Kafka® relied on Apache ZooKeeper™ to manage its cluster metadata, broker registration, topic configuration, partition leadership, and consumer group coordination. Every Kafka broker registered itself in ZooKeeper on startup, and the controller broker used ZooKeeper watches to detect when brokers joined or left the cluster, triggering leader elections for affected partitions.
This dependency meant that operating Kafka required running and maintaining a separate ZooKeeper ensemble alongside the Kafka cluster. The two systems had different operational models, different configuration surfaces, and different failure modes. A ZooKeeper outage could prevent Kafka from completing leader elections, even if the brokers themselves were healthy. Scaling Kafka also meant reasoning about ZooKeeper capacity separately, since metadata operations flowed through the ensemble.
Consumer group offsets were historically stored in ZooKeeper as well, though Kafka moved those to an internal topic early in its development. By the time Kafka reached widespread enterprise adoption, ZooKeeper’s role had narrowed to cluster metadata and controller coordination, but that remaining dependency was still significant enough to affect how teams planned capacity, security boundaries, and upgrades.
What is KRaft mode and does it replace ZooKeeper in Kafka?
KRaft mode (short for Kafka Raft) is Kafka’s built-in metadata management system that replaces the external ZooKeeper dependency entirely. Introduced as a preview in Kafka 2.8 and declared production-ready in Kafka 3.3, KRaft moves cluster metadata into Kafka itself using a Raft-based consensus protocol. As of Kafka 3.5, ZooKeeper mode was deprecated, and Kafka 4.0, released in 2025, removed ZooKeeper support completely.
In KRaft mode, a set of controller nodes manages the metadata log using the Raft protocol. Brokers replicate metadata from the active controller rather than reading from an external system. This eliminates the operational overhead of running a separate ensemble, reduces the number of moving parts in a Kafka deployment, and allows the controller to propagate metadata changes to brokers faster than the previous ZooKeeper-based approach allowed.
For teams running Kafka today, the answer is straightforward: ZooKeeper is no longer part of a Kafka deployment. Any Kafka version still supported by its community requires KRaft, and migrating existing ZooKeeper-based clusters to KRaft is possible in the 3.x series using a built-in migration tool. Teams still running older Kafka versions with ZooKeeper should treat that as a technical debt item requiring attention.
Where is ZooKeeper still actively used in modern data platforms?
Despite Kafka’s move away from ZooKeeper, several widely deployed open-source systems still depend on it as of 2026. Apache HBase® uses ZooKeeper for region server registration, master election, and distributed locking. Apache Hadoop® YARN and HDFS NameNode high availability both rely on ZooKeeper for active/standby failover. Apache SolrCloud uses ZooKeeper to store its cluster configuration and shard routing information.
These systems have not moved as quickly as Kafka to internalize their coordination layer, partly because their architectures are more complex and partly because the ZooKeeper dependency is more deeply embedded in their design. In environments where HBase or HDFS HA is still in production, ZooKeeper remains a required component and needs to be operated accordingly.
Some organizations also use ZooKeeper directly as a coordination primitive in custom distributed applications, though this use case has become less common as higher-level alternatives have matured. If your data platform does not include HBase, HDFS HA, or SolrCloud, there is a reasonable chance you do not need ZooKeeper at all.
What are the main operational challenges of running ZooKeeper?
Running a ZooKeeper ensemble reliably requires careful attention to several operational concerns that are easy to underestimate. The most common issues involve heap sizing, disk I/O, and session management, all of which interact in ways that can cause cascading failures in dependent systems.
- Heap and GC pressure: ZooKeeper runs on the JVM, and garbage collection pauses can cause session timeouts in clients. Tuning heap size and GC settings is necessary but not always straightforward, especially under high watch counts.
- Disk latency sensitivity: ZooKeeper writes a transaction log to disk before acknowledging writes. High disk latency directly translates to write latency and can cause leader elections if the latency triggers session timeouts. Dedicated SSDs are strongly recommended.
- Watch storms: When a large number of clients watch the same znode and that znode changes, ZooKeeper sends notifications to all watchers simultaneously. In large clusters, this can cause a brief but significant spike in network and processing load.
- Ensemble sizing constraints: Adding nodes to a ZooKeeper ensemble requires a rolling restart and careful quorum management. You cannot scale it as freely as a stateless service.
- Monitoring complexity: ZooKeeper exposes metrics through the four-letter word commands and JMX, but integrating those into modern observability stacks requires additional tooling.
On Kubernetes, these challenges compound. ZooKeeper is stateful, requires stable network identities, and is sensitive to pod scheduling and storage performance. Running it well on Kubernetes means using StatefulSets with carefully configured persistent volumes and anti-affinity rules, which adds to the operational surface area of any platform that includes it.
What alternatives to ZooKeeper are data platforms adopting?
The most widely adopted alternative to ZooKeeper for distributed coordination is etcd, the key-value store that Kubernetes itself uses for cluster state. etcd uses the Raft consensus protocol, offers a clean gRPC API, and integrates naturally into Kubernetes-native environments. For applications that need a ZooKeeper-compatible API without running ZooKeeper, some teams have experimented with compatibility layers, though these are not widely standardized.
For service discovery and configuration management specifically, Consul is another common choice, offering health checking, key-value storage, and service mesh capabilities in a single system. Its operational model differs significantly from ZooKeeper, but it addresses many of the same coordination use cases.
Within the Apache ecosystem, individual projects are solving the problem in different ways. Kafka chose to build its own Raft-based metadata system. Other projects are evaluating similar approaches or considering etcd as an external dependency. The general direction across the ecosystem is toward reducing dependence on ZooKeeper rather than replacing it with a single universal alternative.
For teams building new data platform architectures, the practical question is less about which ZooKeeper alternative to choose and more about whether any external coordination service is needed at all, given that many modern systems have internalized this layer.
Should new data platform deployments still include ZooKeeper?
For most new data platform deployments in 2026, the answer is no. If you are not deploying a system that explicitly requires ZooKeeper, such as Apache HBase® or HDFS NameNode HA, there is no reason to introduce it. Kafka no longer needs it. Most modern data platform components either manage coordination internally or rely on Kubernetes-native primitives.
If your architecture does include HBase or HDFS HA, ZooKeeper remains a necessary dependency and should be treated as a first-class operational concern rather than an afterthought. In that context, running a dedicated, properly sized ensemble with appropriate monitoring is the right approach.
For greenfield deployments, the cleaner path is to choose components that do not require ZooKeeper. Kafka in KRaft mode, Trino, and Apache Spark™ all operate without it. This reduces the total operational surface area of the platform and eliminates a class of failure modes that have historically caused incidents in ZooKeeper-dependent systems.
The short version: include ZooKeeper only when a specific dependency forces you to. Otherwise, leave it out.
How Stackable helps with Apache ZooKeeper and distributed coordination
The SDP takes a practical stance on ZooKeeper: support it where necessary, and avoid it where modern alternatives exist. Here is what that looks like concretely:
- Kafka without ZooKeeper: The Stackable Operator for Apache Kafka® deploys Kafka in KRaft mode by default, eliminating the need to run or manage a ZooKeeper ensemble for Kafka workloads.
- ZooKeeper Operator where needed: For systems that still require ZooKeeper, such as Apache HBase®, the SDP includes a dedicated ZooKeeper Operator that handles StatefulSet configuration, persistent volume management, and ensemble lifecycle on Kubernetes, reducing the manual work involved in running it correctly.
- Integrated monitoring: ZooKeeper instances managed by the SDP expose metrics in a format compatible with standard Kubernetes observability stacks, so you do not need to build custom metric collection pipelines.
- Composable architecture: Because the SDP is modular, you only deploy the components your platform actually needs. If your use case does not require ZooKeeper, it simply is not there.
If you are evaluating how to structure a Kubernetes-native data platform that minimizes operational complexity, including around coordination services like ZooKeeper, take a look at the available platform demos or get in touch with the Stackable team to discuss your specific setup.