HDFS, or the Hadoop Distributed File System, is a distributed storage layer built to store very large files across clusters of commodity hardware. For most modern, Kubernetes-native data platforms, you simply don’t need it. Object storage systems and cloud-native storage abstractions have replaced HDFS in the majority of production architectures, offering better scalability, lower operational overhead, and a cleaner separation of compute and storage. That said, there are specific situations where HDFS still makes sense, and knowing when those apply is genuinely useful. This article covers how HDFS works, why it dominated for so long, what you should use instead today, and how Stackable approaches storage in the Stackable Data Platform (SDP).
How does HDFS actually store and manage data?
HDFS splits large files into fixed-size blocks, typically 128 MB or 256 MB, and distributes those blocks across multiple DataNodes in a cluster. A central NameNode tracks the metadata: which blocks belong to which file, and where each block lives. Replication, usually three copies per block, handles fault tolerance without requiring dedicated storage hardware.
When a client writes a file, HDFS streams the data in a pipeline across DataNodes. Reads are locality-aware, meaning the system tries to serve data from the node closest to the requesting process, which cuts down on network traffic in co-located compute-and-storage setups. The NameNode holds all filesystem metadata in memory, which makes it fast, but it also makes it a single point of architectural pressure, since the entire cluster’s metadata state lives in one place.
This design was deliberately built for sequential, batch-oriented reads and writes on very large files. It was never meant for random access, small files, or low-latency queries.
Why was HDFS the default choice for big data workloads?
HDFS became the default because it was purpose-built to handle data at a scale that existing filesystems simply couldn’t reach, and it shipped as the storage layer of Apache Hadoop®, which defined the early big data ecosystem. When MapReduce jobs needed to process terabytes or petabytes, HDFS provided a reliable, fault-tolerant foundation that ran on cheap commodity servers rather than expensive storage area networks.
The advantages at the time were real. HDFS handled hardware failure gracefully through replication. It scaled horizontally by adding more DataNodes. It was open source, which mattered to organizations trying to avoid proprietary storage vendor lock-in. And because Hadoop moved compute to the data rather than the other way around, it minimized the network bottlenecks that plagued earlier distributed processing approaches.
For most of the 2010s, if you ran Spark, Hive, or any Hadoop-ecosystem tool, HDFS was the assumed storage layer. The ecosystem was built around it, which created a self-reinforcing standard.
What are the main limitations of HDFS in modern infrastructure?
HDFS has several structural limitations that make it a poor fit for modern, cloud-native infrastructure. The NameNode bottleneck, tight coupling of compute and storage, poor support for small files, and high operational complexity are the most significant problems teams run into when running HDFS today.
- NameNode as a single point of failure: Even with High Availability configurations using standby NameNodes, the metadata architecture creates operational risk and limits scalability. All filesystem operations pass through the NameNode, which becomes a bottleneck at scale.
- Compute-storage coupling: HDFS assumes that compute and storage live on the same nodes. That made sense when network bandwidth was a real constraint, but modern data center networking and cloud infrastructure have changed that equation. Decoupling compute and storage is now standard practice, and HDFS resists it.
- Small file problem: HDFS handles large sequential files well, but storing millions of small files creates enormous pressure on the NameNode’s memory, since each file requires its own metadata entry regardless of size.
- Operational overhead: Managing HDFS clusters requires significant expertise. Rebalancing data after node additions, handling NameNode memory pressure, and managing block reports across large clusters are non-trivial tasks that add up quickly.
- Poor fit for containers and Kubernetes: HDFS was designed for long-lived, stateful clusters. Kubernetes is designed for ephemeral, reschedulable workloads. These two models conflict in ways that are genuinely difficult to reconcile.
What are the alternatives to HDFS for cloud-native data platforms?
The main alternatives are object storage systems, combined with open table formats that add transactional and schema management capabilities on top. Object storage, whether Amazon S3, Google Cloud Storage, Azure Blob Storage, or self-hosted solutions like MinIO, provides durable, scalable storage without the operational complexity of HDFS.
Object storage separates compute from storage cleanly. Your processing engines, whether Apache Spark™, Trino, or others, read and write data independently of where that data physically lives. This makes it straightforward to scale compute and storage independently, which is one of the core architectural advantages of modern data platforms.
On top of object storage, open table formats have become the standard way to manage structured data at scale:
- Apache Iceberg: Provides ACID transactions, schema evolution, time travel, and partition pruning on top of object storage. Widely adopted and supported across most modern query engines.
- Delta Lake: Originated in the Spark ecosystem, offers similar transactional guarantees and is well integrated with Spark-based workflows.
- Apache Hudi: Focuses on incremental data processing and upsert patterns, which are common in streaming and CDC use cases.
For on-premises deployments where a fully managed object store isn’t available, MinIO is a common choice. It provides an S3-compatible API, runs on Kubernetes natively, and integrates with the same tooling as cloud object storage.
Does HDFS work on Kubernetes?
HDFS can run on Kubernetes, but it works poorly in practice. HDFS requires stable, long-lived nodes with persistent local storage and predictable network identities, which conflicts directly with Kubernetes‘ model of ephemeral, reschedulable pods. The result is an architecture that requires significant workarounds just to function reliably.
Running HDFS on Kubernetes means managing StatefulSets with persistent volume claims, ensuring that DataNode pods aren’t rescheduled to nodes without their associated storage, and handling the NameNode’s sensitivity to pod restarts. These are solvable problems, but they take considerable effort and produce a system that’s more fragile than either a traditional HDFS cluster or a Kubernetes-native storage solution.
Most teams that have tried running HDFS on Kubernetes have either migrated away from HDFS entirely or kept their HDFS clusters outside Kubernetes on dedicated hardware. Running HDFS inside Kubernetes is rarely the right architectural decision in 2026.
When does it still make sense to use HDFS?
HDFS still makes sense in a narrow set of scenarios: mainly when you have an existing, large-scale HDFS deployment that’s working well, when your workloads are tightly optimized for data locality, or when you’re operating in air-gapped environments where object storage alternatives aren’t available or practical.
Specifically, consider keeping HDFS when:
- You have a mature, stable Hadoop cluster with significant data already stored in HDFS and a migration would require substantial effort without a clear payoff.
- Your workloads are dominated by large sequential batch jobs where data locality provides measurable performance advantages that outweigh the operational cost.
- You operate in a regulated or air-gapped environment where introducing new storage infrastructure requires lengthy approval processes, and your existing HDFS setup already meets compliance requirements.
- Your team has deep HDFS expertise and the operational cost of running it is already well understood and budgeted.
For greenfield deployments or teams actively modernizing their data infrastructure, HDFS is rarely the right starting point. The ecosystem has moved on, and the tooling built around object storage and open table formats is now more mature and better supported than it was when HDFS was the only viable option.
How do Kubernetes-native platforms handle storage without HDFS?
Kubernetes-native data platforms handle storage by combining object storage for data persistence with Kubernetes-native persistent volumes for stateful components, and open table formats for structured data management. This approach cleanly separates the concerns that HDFS tried to handle in a single system.
In a Kubernetes-native architecture, the storage layer typically looks like this:
- Object storage for data: Raw data, processed datasets, and table format metadata all live in object storage, accessed via S3-compatible APIs regardless of whether the storage is cloud-hosted or self-managed.
- Persistent volumes for stateful services: Components that need local disk, such as Apache Kafka® brokers or catalog services, use Kubernetes persistent volume claims backed by appropriate storage classes.
- Open table formats for structure: Apache Iceberg or similar formats add transactional semantics, schema management, and query optimization on top of object storage, replacing the role that Hive Metastore and HDFS played together in the Hadoop ecosystem.
- Storage-agnostic compute engines: Query engines like Trino and processing frameworks like Apache Spark™ connect to object storage directly, scaling compute independently of storage capacity.
This architecture is more flexible, more transparent in how data flows through the system, and significantly easier to operate on Kubernetes than any approach involving HDFS. You can explore working demos of these patterns to see how the components fit together in practice.
How Stackable helps with modern data storage on Kubernetes
The SDP is built around exactly this architecture: object storage as the data layer, open table formats for structure, and Kubernetes-native operators for every component in the stack. Rather than carrying HDFS forward as a legacy dependency, the SDP gives you a modular, composable platform where storage and compute are cleanly separated from the start.
Specifically, the SDP provides:
- Kubernetes-native operators for Apache Kafka®, Apache Spark™, Trino, and other data tools, each managing its own storage requirements through Kubernetes-native persistent volumes and storage classes.
- S3-compatible object storage integration that works with MinIO for on-premises deployments, or any cloud object storage provider, without changing your application configuration.
- Apache Iceberg support for transactional data lake workloads, giving you schema evolution, time travel, and ACID guarantees without HDFS.
- Infrastructure-as-code provisioning via
stackablectl, so your entire storage and compute topology is defined declaratively, reproducible, and auditable. - Full data sovereignty: because the SDP is 100% open source and cloud-agnostic, you run it in your own data center, in any cloud, or in a hybrid setup without depending on any vendor’s proprietary storage layer.
If you’re evaluating whether to migrate away from HDFS or building a new Kubernetes-native data platform from scratch, we’re happy to talk through the specifics of your setup. Get in touch with the Stackable team to discuss your architecture.
Apache Hadoop® is a registered trademark of the Apache Software Foundation. Apache Kafka® is a registered trademark of the Apache Software Foundation. Apache Spark™ is a trademark of the Apache Software Foundation. Apache Iceberg is a project of the Apache Software Foundation. Trino is a trademark of the Trino Software Foundation. All other trademarks are the property of their respective owners. This article does not constitute legal or compliance advice.