What is an open table format and why does it matter for data lakehouses?

An open table format is a standardized, open specification that defines how tabular data is stored, organized, and tracked on a distributed file system or object storage. Instead of locking data inside a proprietary database engine, open table formats store data as ordinary files, typically Parquet or ORC, alongside metadata that tracks schema, partitions, snapshots, and transaction history. Any compatible query engine can read that data directly, no copying or converting required. The sections below answer the most common questions about how these formats work, how they compare, and where they fit in modern data architectures. At the end, we look at how the Stackable Data Platform (SDP) supports open table format workloads specifically.

How do open table formats actually work under the hood?

Open table formats work by separating the data files from a metadata layer that tracks every change to the table. The data itself sits in columnar files, most commonly Apache Parquet, on object storage or HDFS. Above that, a metadata layer records which files belong to which version of the table, what the schema looks like, and how partitions are arranged. Query engines read the metadata first, then access only the relevant data files.

This architecture enables several capabilities that a plain directory of Parquet files simply cannot provide on its own:

Snapshot isolation: Each write produces a new snapshot. Readers see a consistent view of the table regardless of concurrent writes.
ACID transactions: Changes are committed atomically. A failed write does not leave the table in a corrupt or partial state.
Partition evolution: The metadata tracks partitioning strategies independently of the data files, so you can change how a table is partitioned without rewriting all existing data.
Schema evolution: Columns can be added, renamed, or reordered, and the metadata records which schema version applies to which files.

The metadata itself is typically a tree of JSON or Avro files. Apache Iceberg, for example, maintains a catalog pointer to the current metadata file, which in turn references a manifest list, which references individual manifests describing each data file. That indirection is what makes time travel, incremental reads, and concurrent access possible, without a central database engine managing locks.

What problems do open table formats solve for data lakehouses?

Open table formats solve the core reliability and usability problems that made traditional data lakes difficult to operate at scale. A raw data lake is essentially a collection of files with no transactional guarantees, no schema enforcement, and no efficient way to update or delete individual records. Open table formats add a transaction log and metadata layer that transforms a file system into something that behaves like a database table, while keeping the data open and portable.

The specific problems they address include:

No ACID guarantees: Without a transaction log, concurrent reads and writes can produce inconsistent results. Open table formats enforce atomicity and isolation.
Inefficient full-table scans: Metadata-level partition pruning and file-level statistics let query engines skip irrelevant files, dramatically reducing the data scanned per query.
Difficult updates and deletes: Regulations like GDPR require the ability to delete specific records. Open table formats support row-level deletes and upserts without rewriting entire partitions.
Schema drift: As upstream data sources evolve, schemas change. Open table formats track schema versions so that old and new data can coexist and be queried correctly.
No time travel: Auditing and reproducibility require access to historical data states. Snapshot-based versioning makes it possible to query a table as it existed at any prior point in time.

Together, these capabilities define the data lakehouse architecture: the cost efficiency and openness of a data lake combined with the reliability and query performance of a data warehouse.

What’s the difference between Apache Iceberg, Delta Lake, and Apache Hudi?

Apache Iceberg™, Delta Lake, and Apache Hudi™ are the three dominant open table formats. They share the same fundamental goal but differ in their design priorities, metadata models, and ecosystem integrations. Apache Iceberg is a format specification maintained by the Apache Software Foundation with broad, engine-agnostic support. Delta Lake originated at Databricks and is now an open standard under the Linux Foundation. Apache Hudi was developed at Uber and focuses on incremental data processing and streaming ingestion use cases.

Apache Iceberg

Iceberg was designed from the start to be engine-agnostic and specification-first. Its metadata model is precise and well-documented, which has made it the default choice for organizations running multiple query engines, Trino, Apache Spark™, Apache Flink, DuckDB, and others, against the same tables. Iceberg supports hidden partitioning, meaning the partition logic is stored in metadata rather than encoded in directory names. This simplifies queries and allows partition strategies to evolve without data rewrites. Its catalog abstraction, REST, Hive, Glue, JDBC, makes it straightforward to integrate into existing infrastructure.

Delta Lake

Delta Lake uses a transaction log stored as a sequence of JSON files in the table directory. It has deep integration with Apache Spark and the Databricks ecosystem, and teams already running Spark pipelines often find it the path of least resistance. The Delta Kernel project and the Delta Sharing protocol extend its interoperability, and the format has matured significantly as an open standard. Teams not tied to the Databricks platform can use Delta Lake independently, though Spark remains its most natural query engine.

Apache Hudi

Hudi’s primary strength is incremental processing. It was built to handle high-frequency upserts and streaming writes efficiently, maintaining separate read-optimized and near-real-time views of the same table. This makes it a strong fit for CDC (change data capture) pipelines and event-driven architectures where data arrives continuously and must be queryable with minimal latency. Its ecosystem support has grown, but it remains most at home in Spark-heavy environments.

Which open table format should you choose for your lakehouse?

For most organizations building a new data lakehouse in 2026, Apache Iceberg is the pragmatic default. It has the broadest query engine support, a clear and stable specification, and strong momentum across the open source ecosystem. If your team is already deeply invested in Apache Spark and the Databricks ecosystem, Delta Lake is a natural fit. If your primary challenge is high-throughput incremental ingestion with CDC pipelines, Apache Hudi deserves serious evaluation.

The key criteria to evaluate are:

Query engine diversity: If you need Trino, Flink, Spark, and DuckDB to all read the same tables, Iceberg’s engine-agnostic design is the lowest-friction path.
Write patterns: Bulk batch writes favor Iceberg or Delta Lake. High-frequency upserts and CDC streams favor Hudi.
Catalog infrastructure: Iceberg’s REST catalog is becoming a de facto standard and integrates cleanly with many metadata services. Delta Lake relies more heavily on Spark session configuration.
Existing tooling: Migrating an existing Spark and Delta Lake environment introduces real risk and cost. Evaluate whether the benefits of switching justify that effort.
Vendor independence: If avoiding lock-in to a specific commercial platform matters to you, Iceberg’s foundation-neutral governance and multi-engine support make it the more open choice.

There is no universally correct answer, but engine diversity is usually the deciding factor for platform teams managing shared infrastructure across multiple teams.

How do open table formats support data governance and compliance?

Open table formats support data governance and compliance by providing the audit trail, schema control, and record-level data management that regulatory frameworks require. Because every change to a table is recorded in a versioned transaction log, organizations can reconstruct exactly what data existed at any point in time, which data was modified, and by which operation. This traceability is a foundational requirement for compliance with regulations such as GDPR, as well as internal data quality and lineage programs.

Specific governance capabilities include:

Time travel and audit history: Query any prior snapshot of a table to verify what data was present at a given moment, supporting audit requests and incident investigations.
Row-level deletes: GDPR and similar regulations require the ability to delete specific individuals’ data on request. Open table formats support targeted row deletion without full-table rewrites.
Schema versioning: Every schema change is recorded. Teams can trace when a column was added or changed and which pipeline version introduced it.
Partition and file-level metadata: Fine-grained metadata enables access control policies to be applied at the partition or column level, supporting data classification and access governance.

These capabilities do not replace a governance framework, but they provide the technical foundation that makes governance enforceable at the storage layer, not only at the application layer.

Are open table formats compatible with existing query engines and tools?

Yes, open table formats are broadly compatible with the most widely used query engines and data tools, though the depth of support varies by format and engine. Apache Iceberg has the widest multi-engine support, with native connectors available for Trino, Apache Spark™, Apache Flink, Presto, DuckDB, Snowflake, and BigQuery, among others. Delta Lake has strong Spark support and growing support in other engines via the Delta Kernel library. Apache Hudi supports Spark natively and has connectors for Flink and Trino.

Compatibility extends beyond query engines. Modern data catalog tools, orchestration platforms, and BI tools increasingly support reading Iceberg and Delta Lake metadata directly. The Iceberg REST Catalog specification is emerging as a common interface, meaning catalog implementations from different vendors can interoperate without format-specific client libraries.

In practice, adopting an open table format does not require replacing your existing query infrastructure. You can introduce Iceberg tables into an environment that already uses Trino or Spark, and both engines will read the same tables without data duplication. This composability is one of the strongest arguments for open table formats over proprietary storage formats tied to a specific database engine. You can explore working demos of these integrations to see how the components fit together in practice.

What is the role of open table formats in a data mesh architecture?

In a data mesh architecture, open table formats serve as the shared storage contract that allows domain teams to own and publish their data products independently while remaining interoperable across the organization. Data mesh distributes data ownership to domain teams, each of which manages its own data pipelines and storage. Open table formats provide the technical foundation for this model by ensuring that data products stored in one domain’s storage can be read by any query engine used by any other domain, without central coordination of storage formats.

The connection between open table formats and data mesh is particularly strong in three areas:

Federated access without data movement: A domain team can publish an Iceberg table to shared object storage and grant other domains read access. Consumers query the data in place using their preferred engine, eliminating the need for centralized ETL pipelines to move data between domains.
Schema contracts: Open table format schemas act as the interface contract between producers and consumers. Schema evolution rules, additive changes allowed, breaking changes versioned, enforce the kind of backward compatibility that data products require.
Governance at the product boundary: Access control, retention policies, and lineage tracking can be applied at the table level, aligning with the data product ownership model where each domain is accountable for its own data quality and compliance.

Open table formats are not a data mesh implementation on their own, but they are a practical prerequisite for making federated data ownership work at scale without creating a fragmented, incompatible storage environment.

How Stackable supports open table format workloads

The SDP is a modular, Kubernetes-native open source data platform that provides the query engines, orchestration, and infrastructure tooling needed to run open table format workloads in production, whether on-premises, in the cloud, or in a hybrid environment. Because the SDP is 100% open source and cloud-agnostic, you retain full data sovereignty: your tables stay on your storage, readable by your engines, without dependency on a proprietary platform.

Concretely, the SDP supports open table format architectures through:

Trino integration: The SDP includes a Stackable Operator for Trino, enabling Trino deployments that query Apache Iceberg tables directly on S3-compatible or HDFS storage.
Apache Spark™ support: The Stackable Operator for Apache Spark™ handles Spark workloads that read and write Delta Lake or Iceberg tables, including batch and streaming jobs.
Infrastructure as code: All components are configured declaratively via Kubernetes custom resources, making deployments reproducible and auditable, which aligns directly with the governance requirements that open table formats enable at the data layer.
Composable architecture: You add only the components you need. If your use case requires Iceberg with Trino and Spark but nothing else, you deploy exactly that, without mandatory bundled services.

If you are evaluating open table formats for a new or existing data platform and want to understand how the SDP fits your architecture, get in touch with the Stackable team to discuss your specific setup.