How do open table formats prevent vendor lock-in in a data lakehouse?

Open table formats prevent vendor lock-in in a data lakehouse by separating how data is stored from how it is queried. Formats like Apache Iceberg, Delta Lake, and Apache Hudi store data as open files on standard object storage, so any compatible query engine can read and write that data without needing a proprietary runtime or service. Here’s how this works in practice, where the differences lie, and what to consider when choosing a format for your organization.

What makes a data lakehouse vulnerable to vendor lock-in?

A data lakehouse becomes vulnerable to vendor lock-in when its storage format, metadata layer, or query engine is tightly coupled to a single vendor’s implementation. If you can only read or write your data using one company’s tools, migrating away means either re-engineering your entire data pipeline or accepting that your data is effectively held hostage by the platform you chose.

The most common sources of lock-in in a lakehouse architecture are:

Proprietary table formats: Some platforms use internal formats that third-party engines can’t read without a conversion step, which adds cost and risk.
Managed metadata services: When the catalog tracking table structure, partitions, and schema evolution is a closed, vendor-operated service, switching query engines or cloud providers means migrating that catalog too.
Compute-storage bundling: Platforms that bundle their storage layer with their own query engine make it hard to bring in a different compute tool, even when the underlying files are stored in a standard format like Parquet.
Licensing restrictions: Some table format implementations are open source in name but governed by licenses or contributor agreements that give one company disproportionate control over the roadmap.

The result is that architectural decisions made early in a project, often under time pressure, can define your infrastructure dependencies for years. Switching costs grow with data volume, pipeline complexity, and how familiar your team has become with the incumbent tooling.

What are open table formats and how do they work?

Open table formats are standardized specifications that define how tabular data is organized, tracked, and queried on top of file-based storage. They sit between raw files (typically Parquet or ORC) and the query engines that process them, providing a metadata layer that supports transactions, schema evolution, and time travel without requiring a proprietary database engine.

At their core, open table formats work by maintaining a transaction log or metadata tree alongside the actual data files. When a query engine reads a table, it first reads the metadata to understand which files are relevant, what the schema looks like, and whether any in-progress transactions need to be accounted for. When data is written, the format appends a new entry to the log rather than overwriting files in place. That’s what enables features like snapshot isolation and rollback.

This design has several practical consequences:

Data files stay in open formats (Parquet, ORC, Avro) on standard object storage like S3, GCS, or Azure Blob Storage.
Any query engine that implements the format’s specification can read and write the same tables.
Schema changes, partition updates, and data deletions are tracked atomically, so concurrent readers and writers don’t corrupt each other’s results.
You can query historical versions of a table without maintaining separate copies of the data.

Because the specification is open, multiple vendors and open-source projects can implement support independently. That’s exactly what breaks the coupling between your data and any single platform.

What’s the difference between Apache Iceberg, Delta Lake, and Apache Hudi?

Apache Iceberg, Delta Lake, and Apache Hudi are the three dominant open table formats. They solve the same core problem, but differ in design philosophy, metadata structure, and ecosystem alignment. The key distinction: Iceberg prioritizes broad, engine-agnostic interoperability, Delta Lake originated within the Databricks ecosystem and has strong Spark integration, and Apache Hudi was designed with streaming and incremental data processing in mind.

Apache Iceberg

Apache Iceberg was originally developed at Netflix and donated to the Apache Software Foundation. Its metadata model is based on a tree of manifest files that track data file locations, partition information, and statistics. Iceberg’s specification is deliberately engine-agnostic, which has led to broad adoption across query engines including Trino, Apache Spark™, Apache Flink, Presto, and DuckDB. Its hidden partitioning feature allows partition strategies to change over time without rewriting data or breaking existing queries, which is a real operational advantage for long-lived tables.

Delta Lake

Delta Lake was created by Databricks and later open-sourced under the Linux Foundation’s Delta Lake project. It uses a transaction log stored as JSON files in the table directory. Delta Lake has deep, first-class integration with Apache Spark™ and is widely used in organizations already running Spark-heavy workloads. Support from other engines has grown substantially, though historically its ecosystem was more Spark-centric than Iceberg’s.

Apache Hudi

Apache Hudi (Hadoop Upserts Deletes and Incrementals) was built at Uber and focuses on efficient upserts and incremental processing. It supports two table types: Copy-on-Write, which rewrites files on every update, and Merge-on-Read, which appends delta logs and merges on read. Hudi is well-suited to use cases where records are frequently updated, such as CDC (change data capture) pipelines from operational databases. Its ecosystem integrations are solid but narrower than Iceberg’s in terms of engine variety.

How do open table formats decouple storage from compute?

Open table formats decouple storage from compute by making the data layer independently addressable. Data files live in standard object storage and the table metadata is a portable specification rather than a service, so any compute engine that implements the format can attach to the same tables without coordinating with the storage system or with other engines.

In a traditional data warehouse, the storage and compute layers are managed by the same system. The warehouse controls how data is laid out on disk, how indexes are maintained, and which query planner is used. You can’t swap out the query engine without also migrating the data.

With an open table format, the separation works like this: data files sit in object storage, table metadata (manifests, transaction logs, schema definitions) sits alongside them in the same storage location, and the query engine reads that metadata at query time to plan its execution. The engine doesn’t own the data. It reads the specification, locates the relevant files, and processes them. A second engine can do exactly the same thing against the same files simultaneously.

This means you can run Trino for interactive analytics, Apache Spark™ for batch transformations, and Apache Flink for streaming ingestion, all against the same Iceberg or Hudi tables, without any of those engines needing to know about each other. You can also replace one engine with another without touching the underlying data. That’s the practical definition of compute-storage decoupling.

Which query engines and tools are compatible with open table formats?

Compatibility with open table formats is now broad across the major open-source query engines. Apache Iceberg has the widest support, with native integrations in Trino, Apache Spark™, Apache Flink, Presto, DuckDB, StarRocks, and Snowflake (via external tables). Delta Lake is natively supported in Apache Spark™ and has connectors for Trino, Apache Flink, and several cloud-native services. Apache Hudi integrates with Apache Spark™, Apache Flink, Trino, and Presto.

Beyond query engines, the catalog layer matters just as much for interoperability. The Hive Metastore has historically been the default catalog for lakehouse architectures, but the Apache Iceberg REST Catalog specification has emerged as a more portable alternative, supported by tools like Project Nessie (which adds Git-like branching for data) and several managed catalog services. A well-chosen catalog lets you swap query engines without losing table registration and access control metadata.

On the ingestion and transformation side, tools like Apache Kafka® (for streaming data), dbt (for SQL-based transformations), and various ETL frameworks have added open table format support, so the compatibility story extends well beyond query time. If you are evaluating a data platform solution, checking format support across your full pipeline, not just your query engine, is the right starting point.

When should an organization switch to an open table format?

An organization should switch to an open table format when it faces growing friction from format lock-in, needs ACID transaction support on its data lake, or wants to run multiple query engines against the same data without duplication. The decision becomes most urgent when migration costs or licensing constraints are already influencing architecture choices.

Specific signals that the time is right include:

You are maintaining multiple copies of the same data to serve different tools, which adds storage cost and creates consistency risk.
Schema changes are painful because your current format requires full table rewrites or manual coordination across pipelines.
You need reliable deletes or updates at the row level for GDPR compliance or CDC use cases, which raw Parquet on object storage cannot support without a table format layer.
You are evaluating a new query engine and want the flexibility to adopt it without migrating your data.
Your cloud or platform costs are rising and you want the option to move workloads between providers without a full data migration.

Migration from an existing format to an open table format is not trivial, particularly at scale, but the ecosystem tooling has matured considerably. Most major formats provide utilities for in-place migration of Parquet-based tables, and a phased approach, starting with new tables and migrating existing ones incrementally, reduces risk. You can explore working demos of lakehouse architectures to get a concrete sense of what a migration target looks like before committing.

How does data sovereignty factor into choosing an open table format?

Data sovereignty is directly relevant to open table format selection because the format determines whether your data can be read, moved, and governed independently of any vendor’s platform. An open table format stored on infrastructure you control means you retain full authority over where your data lives, who can access it, and which tools process it, without requiring permission or continued service from a third party.

For organizations in regulated industries, particularly financial services, healthcare, and the public sector, data sovereignty is not an abstract preference. It has legal and compliance dimensions: data residency requirements, audit obligations, and the need to demonstrate control over data processing. A proprietary format or a vendor-managed catalog introduces a dependency that can conflict with these requirements, because the vendor’s service availability and terms of service become part of your compliance posture.

Open table formats address this in two ways. First, the data itself remains in open, readable files on storage you control. Second, because the format specification is open and independently implemented, you are not dependent on any single vendor to read or process your data in the future. If a vendor changes its licensing model, discontinues a product, or raises prices, your data remains accessible.

The choice between Iceberg, Delta Lake, and Hudi also carries sovereignty implications. Apache Iceberg and Apache Hudi are governed by the Apache Software Foundation under the Apache License 2.0, which provides strong community governance and no single-vendor control. Delta Lake is governed by the Linux Foundation’s Delta Lake project. Understanding the governance model of the format you adopt is part of a complete data sovereignty assessment, not just the technical specification.

How Stackable helps with open table formats and data sovereignty

The Stackable Data Platform (SDP) is built to support exactly the kind of open, modular lakehouse architecture described in this article. Rather than locking you into a proprietary runtime, the SDP gives you Kubernetes-native operators for the open-source tools that make open table formats practical at scale:

Trino for interactive SQL queries against Iceberg, Delta Lake, and Hudi tables stored on your own object storage.
Apache Spark™ for batch and streaming transformations against the same tables, managed via the Stackable Operator for Apache Spark™.
Apache Kafka® for streaming ingestion into open table format tables, managed via the Stackable Operator for Apache Kafka®.
Open Hive Metastore and Nessie catalog support to keep your table metadata portable and under your control.

Because the SDP is 100% open source and cloud-agnostic, you can deploy it on-premises, in any cloud, or in a hybrid environment, and your data stays where you put it. There is no proprietary layer between your data and your tools. This is what „your data, your platform“ means in practice: the infrastructure serves you, not the other way around.

If you want to see how this fits your specific setup, talk to the Stackable team about your architecture and requirements.