What is Apache Iceberg and how does it work?

Apache Iceberg™ is an open table format built for large-scale analytic datasets living in data lakes. It brings database-style reliability to distributed file systems by tracking every file, schema change, and partition in a structured metadata layer. The result is that query engines get a consistent, transactional view of data that plain file-based storage simply cannot offer. Below, we cover how Iceberg works, what problems it solves, and how it compares to other open table formats.

At the end, we look at how Stackable integrates Apache Iceberg into a Kubernetes-native data platform.

How does Apache Iceberg store and track data?

Apache Iceberg stores data as ordinary files, typically Parquet, ORC, or Avro, in object storage or a distributed file system. It tracks those files through a layered metadata structure. At the top sits a catalog pointer, which references a metadata file. That metadata file links to one or more snapshot manifests, which in turn list individual manifest files. Each manifest file records the exact data files belonging to a given snapshot, along with column-level statistics.

This three-tier structure, catalog, metadata files, manifest lists, manifest files, data files, is what makes Iceberg behave like a transactional table rather than a loose collection of files. Every write operation produces a new snapshot. Readers always see a consistent, immutable view of the table at a specific point in time, regardless of what writers are doing concurrently. Older snapshots are retained until explicitly expired, which is what makes time travel queries possible.

Because the metadata tracks column-level statistics like min and max values per file, query engines can skip entire files that cannot contain matching rows. This file-level pruning, combined with partition pruning, significantly cuts down the amount of data scanned for typical analytic queries.

What problems does Apache Iceberg solve for data lakes?

Apache Iceberg solves the core reliability and performance problems that make raw data lakes hard to use in production: no ACID guarantees, no consistent reads during writes, no reliable schema enforcement, and expensive full-table scans caused by poor metadata management.

Traditional data lakes store data as files without any coordination layer. This creates several well-known problems:

Inconsistent reads: A reader scanning a directory mid-write may see partial results, mixing old and new files.
No atomicity: A failed write can leave corrupted or incomplete data with no rollback mechanism.
Partition management overhead: Hive-style partitioning exposes partition columns to users, requiring queries to know the physical layout of data. Reorganizing partitions is destructive and expensive.
Schema drift: Adding or renaming columns can break downstream consumers if there is no formal schema registry or evolution protocol.
Slow metadata operations: Listing directories to discover files does not scale to tables with millions of partitions.

Iceberg addresses each of these by moving metadata management out of the file system and into a structured, versioned catalog. The result is a data lake that behaves much more like a relational database from the perspective of query engines and data consumers.

What are the key features of Apache Iceberg?

The key features of Apache Iceberg are ACID transactions, time travel and snapshot isolation, hidden partitioning, schema evolution, partition evolution, and broad multi-engine compatibility. Together, these make Iceberg a solid foundation for a production-grade open source data platform.

ACID transactions: Optimistic concurrency control ensures that concurrent reads and writes do not corrupt table state. Conflicting writes are detected and rejected rather than silently merged.
Time travel: Because every write creates a new snapshot, you can query any previous version of a table by referencing a snapshot ID or a timestamp.
Hidden partitioning: Iceberg computes partition values from column data automatically. Queries do not need to include partition predicates, and the physical layout can change without rewriting queries.
Schema evolution: Columns can be added, renamed, reordered, or dropped without rewriting existing data files. Iceberg tracks column identity by ID rather than by name, preventing silent data corruption when columns are renamed.
Incremental reads: Consumers can read only the files added or deleted between two snapshots, which makes streaming and incremental ETL pipelines more efficient.
Row-level deletes: Iceberg supports merge-on-read and copy-on-write delete modes, enabling UPDATE and DELETE operations on immutable file formats.

What is the difference between Apache Iceberg, Delta Lake, and Apache Hudi?

Apache Iceberg, Delta Lake, and Apache Hudi are all open table formats that add transactional semantics to data lake storage, but they differ in design philosophy, engine compatibility, and primary use cases. The clearest distinction is that Iceberg was designed from the start to be engine-agnostic, while the other two originated within specific ecosystem contexts.

Apache Iceberg

Iceberg was developed at Netflix and donated to the Apache Software Foundation. Its specification is open and engine-neutral, meaning any query engine can implement full read and write support without depending on a particular vendor’s runtime. This makes it a natural fit for multi-engine environments where Trino, Apache Spark™, Flink, and other tools need to share the same tables.

Delta Lake

Delta Lake was created by Databricks and later open-sourced under the Linux Foundation. It uses a transaction log stored alongside data files. Engine support beyond Spark has grown significantly, but many teams find that deep integration with the Databricks platform is where it performs best. The Delta Lake protocol has been opened and documented, and independent implementations now exist, though ecosystem breadth still trails Iceberg in some environments.

Apache Hudi

Apache Hudi (Hadoop Upserts Deletes and Incrementals) was developed at Uber and is optimized for record-level upsert and delete workloads with low-latency ingestion. It introduced concepts like merge-on-read and copy-on-write that Iceberg has since adopted. Hudi’s strength is near-real-time data ingestion pipelines, though its metadata model is more complex to operate than Iceberg’s for pure analytic workloads.

In practice, Iceberg is the most widely adopted choice for new data lakehouse architectures in 2026, particularly where engine flexibility and long-term format stability matter most.

Which query engines are compatible with Apache Iceberg?

Apache Iceberg works with a wide range of query engines, including Apache Spark™, Trino, Apache Flink, Presto, Dremio, StarRocks, ClickHouse, and DuckDB, among others. This broad compatibility is a direct result of Iceberg’s open, engine-neutral specification.

Engines integrate with Iceberg through a catalog interface. The catalog resolves table names to metadata locations, and from there each engine reads the manifest structure independently. Supported catalog backends include the Hive Metastore, AWS Glue, Nessie, REST catalogs, and JDBC-based catalogs.

For teams running Kubernetes-based data platforms, this means a single Iceberg table can serve a Trino cluster for interactive queries, a Spark job for batch transformations, and a Flink application for streaming writes, all without data format conversion or synchronization overhead. The table format acts as the shared contract between engines.

How does Apache Iceberg handle schema and partition evolution?

Apache Iceberg handles schema evolution by tracking columns by a unique integer ID rather than by name. Renaming, reordering, or dropping a column does not break existing data files or downstream readers. Partition evolution works similarly: the partitioning strategy for a table can change without rewriting historical data, because Iceberg records which partition spec applied to which snapshot.

Schema evolution in detail

When you add a column, Iceberg assigns it a new ID and records a default value (or null) for existing files that predate the column. When you rename a column, only the metadata changes, and data files are untouched. Readers that have not yet updated their schema continue to work against the old metadata version. This makes schema changes safe to perform without coordinating all consumers at once, which is a real operational advantage in large organizations.

Partition evolution in detail

Hive-style partitioning requires data to be physically reorganized when the partition strategy changes. Iceberg avoids this by associating each snapshot with the partition spec that was active when it was written. A query planner reads the relevant partition spec for each set of files and applies the correct pruning logic, even when different parts of the table were written under different partitioning schemes. For example, you can migrate a table from daily partitioning to hourly partitioning without touching historical files, as new writes use the new spec and old files retain the old spec.

When should you use Apache Iceberg for your data platform?

Apache Iceberg is worth using when you need ACID-compliant, multi-engine access to large analytic datasets stored in open file formats, particularly when your requirements include time travel, schema evolution, or the ability to switch query engines without migrating data.

Iceberg is a strong fit in these scenarios:

You are building a data lakehouse that needs to serve both batch and streaming workloads from the same storage layer.
Multiple teams or tools need to read and write the same tables concurrently without coordination overhead.
You need to perform UPDATE, DELETE, or MERGE operations on data stored in Parquet or ORC files.
Regulatory or audit requirements demand point-in-time query capability or a full history of data changes.
You want to avoid format lock-in and retain the flexibility to change query engines as your platform evolves.

Iceberg is less critical if your workload is purely append-only, your dataset is small enough to query with a single engine, or you do not need transactional guarantees. In those cases, the overhead of managing an Iceberg catalog may not be justified. For most production data platforms handling data at scale, though, the operational benefits of Iceberg’s metadata model outweigh the added complexity.

How Stackable supports Apache Iceberg workloads

The Stackable Data Platform (SDP) is a modular, Kubernetes-native data platform that provides the infrastructure layer for running Apache Iceberg-based data lakehouses in production. Rather than assembling and wiring together individual components manually, the SDP lets you deploy and operate the tools that work with Iceberg, including Apache Spark™ and Trino, through declarative Kubernetes operators.

Concretely, the SDP helps with Iceberg workloads in these ways:

Kubernetes-native operators: The Stackable Operator for Apache Spark™ and the Trino Operator manage the full lifecycle of these engines, including configuration, upgrades, and monitoring, so they can reliably read from and write to Iceberg tables.
Catalog integration: The SDP supports catalog backends compatible with Iceberg, giving you a consistent metadata layer across engines without manual wiring.
Data sovereignty: Because the SDP runs on-premises, in any cloud, or in hybrid environments, your Iceberg tables and the data within them stay under your control. No vendor intermediates access to your storage layer.
Open source, no lock-in: The SDP is 100% open source. The Iceberg format itself is open, and the engines the SDP operates are open source. The combination gives you a fully traceable, auditable software supply chain.
Infrastructure as code: All platform components are configured declaratively, making Iceberg-based pipelines reproducible and version-controlled from day one.

If you are evaluating how to run Apache Iceberg workloads on Kubernetes without proprietary dependencies, get in touch with the Stackable team to discuss your architecture.

Apache Iceberg, Apache Spark, Apache Kafka, Apache ZooKeeper, Apache Hive, Apache HBase, Apache NiFi, Apache Superset, Apache Hadoop, Apache Phoenix, and Apache Airflow are trademarks of the Apache Software Foundation. Trino is a trademark of the Trino Software Foundation. Use of these marks does not imply endorsement by the respective foundations.