How do open table formats keep your data lakehouse portable across engines?

Open table formats keep your data lakehouse portable across engines by separating how data is stored from how it is queried. Instead of tying your data to a proprietary engine, formats like Apache Iceberg, Delta Lake, and Apache Hudi store data as open files, typically Parquet or ORC, alongside a standardized metadata layer that any compatible engine can read and write. This means you can query the same dataset with Trino, Apache Spark™, or Flink without migrating, copying, or converting anything.

That portability guarantee holds as long as your catalog layer and tooling respect the format’s open specification. Below, we get into how each piece of that works, where real interoperability happens, and where the limits are.

What makes a table format ‚open‘ in a data lakehouse?

A table format is open when its specification is publicly documented, not controlled by a single vendor, and implementable by any engine without licensing restrictions. In a data lakehouse, an open table format defines how data files are organized on object storage, how schema and partitioning are tracked, and how changes are recorded, all through a metadata layer that any conforming engine can interpret.

The practical effect is that you own your data files outright. Because the format specification is public, a new engine, an upgraded version of an existing engine, or even a completely different vendor’s tooling can read and write your tables without requiring you to export or transform the data first. This is the foundational property that makes open source data platforms viable as a long-term architecture choice.

Three properties distinguish a genuinely open table format:

Open specification: The format’s metadata structure is documented publicly and not gated behind proprietary APIs.
Vendor-neutral governance: No single company controls the specification’s evolution unilaterally.
File-level openness: The underlying data files use open columnar formats like Parquet or ORC, not proprietary binary formats.

How do open table formats decouple storage from compute?

Open table formats decouple storage from compute by placing a metadata layer between the raw data files on object storage and the query engine that processes them. The engine reads the metadata to locate the relevant data files, applies its own execution logic, and writes results back, without ever owning or locking the underlying files.

In a traditional database or proprietary data warehouse, the storage format and the compute engine are tightly coupled. The engine controls how data is physically laid out, and reading that data outside the engine is either impossible or requires a full export. Open table formats break this coupling by making the metadata layer the shared contract. The data files live on any object-compatible storage, S3, GCS, Azure Blob Storage, or on-premises MinIO, and the metadata describes their location, schema, partitioning, and change history in a standardized way.

This architecture means you can swap, add, or run multiple compute engines against the same data simultaneously. A batch job running on Apache Spark™ and an interactive query from Trino can both operate on the same Iceberg table without coordination beyond the catalog. Neither engine needs to know anything about the other’s internal format.

What’s the difference between Apache Iceberg, Delta Lake, and Apache Hudi?

Apache Iceberg, Delta Lake, and Apache Hudi all add ACID transactions, schema evolution, and time travel to data stored on object storage, but they differ in governance model, engine ecosystem, and design emphasis. Apache Iceberg is governed by the Apache Software Foundation. Delta Lake is a Linux Foundation project with historically strong ties to Spark-based ecosystems. Apache Hudi was designed with record-level upserts and streaming ingestion as a primary concern.

Apache Iceberg

Apache Iceberg uses a tree-structured metadata model with snapshot isolation, making it well suited for large-scale batch analytics and concurrent multi-engine access. Its specification is engine-agnostic by design, and it has broad support across Trino, Apache Spark™, Flink, Hive, and Dremio. Schema evolution and hidden partitioning are first-class features, meaning you can change table structure without rewriting data or breaking existing queries.

Delta Lake and Apache Hudi

Delta Lake uses a transaction log, the Delta Log, stored alongside the data files. It integrates tightly with Spark but has expanded its engine support significantly. Apache Hudi organizes tables around record keys and supports both copy-on-write and merge-on-read storage types, which makes it a common choice for CDC pipelines and near-real-time ingestion workloads where individual record updates are frequent. All three formats are actively developed and support the core portability primitives, but their community governance and engine support breadth differ enough to matter for long-term architecture decisions.

Why does engine interoperability depend on the catalog layer?

Engine interoperability depends on the catalog layer because the catalog is what tells each engine where a table’s metadata lives. Without a shared, engine-neutral catalog, two engines may each be able to read an open table format but have no way to discover the same table, coordinate on the current snapshot, or see each other’s writes.

The catalog translates a table name into a metadata location. When multiple engines share a single catalog, such as Apache Hive Metastore, AWS Glue, or the open REST Catalog specification defined by Apache Iceberg, they all resolve the same table name to the same metadata pointer. This is what makes concurrent access safe and consistent.

Without a shared catalog, you end up with engine-specific table registrations that diverge over time. One engine’s writes become invisible to another, and the portability promise of the open format breaks down at the organizational layer even if the file format itself is fully open. Choosing a catalog that multiple engines support natively is therefore just as important as choosing the right table format.

What is time travel and how does it work across engines?

Time travel in a data lakehouse is the ability to query a table as it existed at a specific point in the past, a previous snapshot, a specific timestamp, or a prior schema version. It works because open table formats record every change to a table as an immutable snapshot in the metadata layer, preserving the file references for each historical state without immediately deleting old data files.

When you run a time travel query, the engine reads the metadata to find the snapshot corresponding to your requested point in time, then reads only the data files referenced by that snapshot. The current data files are untouched. Because the snapshot history is part of the open format’s specification, any engine that implements the format correctly can execute time travel queries against the same table using the same syntax or API.

In practice, syntax varies slightly across engines. Trino and Apache Spark™ both support Iceberg time travel but use different SQL clauses. The underlying mechanism is identical, both resolve to the same snapshot in the same metadata, so the results are consistent. Time travel is also what makes reproducible data pipelines practical: you can pin a pipeline to a specific snapshot and guarantee it processes the same data regardless of when it runs.

How do open table formats prevent vendor lock-in?

Open table formats prevent vendor lock-in by ensuring your data is always readable without the vendor’s proprietary tooling. Because the format specification is public and the data files are stored in open columnar formats, you can change your query engine, your cloud provider, or your data platform without needing to export, migrate, or convert your data.

The contrast with proprietary formats is straightforward. When data is stored in a format that only one vendor’s engine understands natively, switching engines requires a full data migration, expensive, time-consuming, and often technically risky at scale. Open table formats remove that exit cost. Your data stays where it is; only the tooling changes.

A few specific mechanisms reinforce this:

Portable file storage: Data lives in Parquet or ORC on any object-compatible storage, not in a vendor-managed proprietary store.
Open metadata specification: Any engine can parse the metadata without a proprietary SDK or license.
Catalog portability: REST Catalog and other open catalog APIs allow metadata to be accessed by any conforming client.
Multi-engine write support: Multiple engines can write to the same table, so no single engine becomes a mandatory bottleneck.

What are the limitations of open table formats for portability?

Open table formats improve portability significantly, but they don’t eliminate all friction. The main limitations are catalog compatibility gaps, feature divergence between engine implementations, operational complexity, and the pace at which newer format features gain broad engine support.

Not every engine implements every feature of a format’s specification at the same time. A feature introduced in Apache Iceberg’s v2 specification, such as row-level deletes, may be fully supported in one engine and partially supported or unsupported in another. Queries that rely on those features may behave differently or fail when run against a different engine, even though the table format itself is open.

Catalog compatibility is a related gap. If two engines use different catalog implementations that don’t share state, they can’t safely coordinate concurrent writes, even to an open format table. Adopting a shared catalog resolves this, but introduces its own operational dependency.

There’s also the question of operational complexity. Running multiple engines against a shared open table format requires careful management of compaction, snapshot expiry, and schema evolution to avoid performance degradation and storage bloat. Open formats give you the freedom to run any engine, but that freedom comes with the responsibility of managing the table lifecycle yourself, or choosing a platform that handles it for you.

How Stackable supports open table format portability

The Stackable Data Platform (SDP) is built around the same principles that make open table formats valuable: open specifications, no proprietary lock-in, and the freedom to compose your stack from the tools that fit your workload. SDP provides a Kubernetes-native environment where you can run the engines and services that work with Apache Iceberg and other open formats without being tied to a single cloud provider or managed service.

Concretely, SDP addresses the portability challenges described in this article in several ways:

Multi-engine deployment: SDP operators for Apache Spark™, Trino, and Apache Kafka® can be deployed and managed on the same cluster, sharing access to the same object storage and catalog configuration.
Catalog integration: SDP supports integration with open catalog services, so multiple engines can resolve the same table metadata without manual coordination.
Infrastructure as code: Table format configuration, storage connections, and engine settings are declared in Kubernetes-native manifests, making your entire data stack reproducible and auditable.
Data sovereignty: Because SDP runs on your own infrastructure, on-premises, in any cloud, or hybrid, your data files and metadata never leave your control.

If you want to see how this works in practice before committing to a full deployment, the SDP demo environment includes working examples with open table formats and multiple query engines. To discuss your specific architecture, get in touch with the Stackable team directly.

Apache Kafka® is a registered trademark of The Apache Software Foundation. Apache Spark™, Apache Iceberg, Apache Hive™, and Apache Hudi are trademarks of The Apache Software Foundation. Trino is a trademark of the Trino Software Foundation. All other trademarks are the property of their respective owners.