What is schema evolution and how do open table formats handle it?

Schema evolution is what happens when you need to change the structure of a dataset over time, adding columns, renaming fields, changing data types, without breaking everything that reads or writes that data. Traditional pipelines struggle with this because they treat the schema as something fixed, baked into code or storage formats. Open table formats like Apache Iceberg, Delta Lake, and Apache Hudi solve this by tracking schema changes as versioned metadata, so readers and writers stay compatible even as things change. Here’s how each format handles it, and what that means for your data engineering decisions.

Why does schema evolution break traditional data pipelines?

Traditional pipelines break during schema changes because they treat the schema as a static contract. When a source system adds a column, renames a field, or changes a data type, downstream readers expecting the old structure either fail outright or, worse, silently produce wrong results. The schema is often encoded in application code, serialization formats, or pipeline configuration, with no built-in way to detect or adapt to drift.

In file-based data lakes using raw Parquet or CSV, there’s no central schema registry. Each file may carry its own structure, and readers have to reconcile differences manually. This creates a category of bugs that are notoriously hard to diagnose: a job runs successfully but produces wrong aggregations because a renamed column was silently dropped. The cost scales with pipeline complexity. When dozens of downstream jobs depend on a single upstream table, one unannounced schema change can trigger cascading failures across the entire platform.

The deeper problem is that traditional formats lack a versioned metadata layer. Without it, there’s no way to ask „what did this table look like three weeks ago?“ or „which version of the schema does this batch of files belong to?“ Open table formats address exactly this gap by treating the schema as a first-class, versioned artifact stored alongside the data.

What types of schema changes are considered safe versus breaking?

Schema changes fall into two broad categories: safe changes, which preserve compatibility with existing readers and writers, and breaking changes, which don’t. Safe changes include adding a new nullable column, adding a new field to a nested struct, and widening a numeric type, for example, promoting an integer to a long. Breaking changes include deleting a column, renaming a column without aliasing, narrowing a data type, and reordering fields in formats that are position-dependent.

The distinction matters because it determines whether a schema change can be deployed without coordinating all upstream producers and downstream consumers at once. Safe changes allow a rolling deployment: the new schema is applied, and existing readers simply ignore the new column or use a default value. Breaking changes require either a coordinated cutover, a schema migration job, or compatibility modes that translate between old and new versions.

Worth noting: „safe“ is context-dependent. Adding a non-nullable column with no default value is safe for writers that always populate it, but breaks readers that construct records without that field. Understanding the full read/write surface of a table is a prerequisite for classifying any change accurately.

How does Apache Iceberg handle schema evolution?

Apache Iceberg handles schema evolution by assigning a unique integer ID to every column at creation time and tracking all schema changes in its metadata layer. Because Iceberg uses column IDs rather than column names or positions to map data, renaming a column, reordering fields, or adding new columns doesn’t break existing readers. The metadata records the full history of schema versions, and each snapshot of the table references the schema that was active when it was written.

Iceberg explicitly supports the following schema change operations without requiring a full table rewrite: adding columns (including nested fields), dropping columns, renaming columns, reordering columns, and widening types, for example, int to long, float to double, or decimal precision increases. Type narrowing and changing a column from optional to required are not supported as safe operations.

One practical consequence of the column-ID approach is that time-travel queries work correctly even when the schema has changed. A query against a snapshot from six months ago will use the schema that was active at that time, with column IDs mapping correctly to the physical files. This makes Apache Iceberg particularly well suited for audit trails, reproducible analytics, and long-running analytical workloads where schema stability can’t be guaranteed across the full data lifecycle. If you’re evaluating open-source data platform solutions, Iceberg’s metadata model is one of the most mature available.

How does Delta Lake manage schema changes and enforcement?

Delta Lake manages schema changes through two mechanisms: schema enforcement and schema evolution. Schema enforcement is on by default and rejects any write that doesn’t match the current table schema, protecting the table from accidental corruption. Schema evolution is opt-in and must be explicitly enabled, either by setting a configuration option or by using a mergeSchema option at write time.

Schema enforcement

When schema enforcement is active, Delta Lake validates every incoming DataFrame against the table’s registered schema before committing the write. If the incoming data contains new columns, mismatched types, or missing required fields, the write fails with a clear error. This is useful in production pipelines where schema drift from upstream sources should be caught immediately rather than silently absorbed.

Schema evolution

When schema evolution is enabled, Delta Lake automatically merges the incoming schema with the existing table schema. New columns are added, and the existing data is treated as having null values for those columns. Delta Lake supports adding new columns and promoting certain types, for example, byte to short to int to long. It doesn’t automatically handle column renames, which are treated as a drop and an add, or type narrowing. For more complex migrations, Delta Lake provides a CHANGE COLUMN command and supports generated columns and column mapping in newer versions, which allow renaming without physical data rewrites.

How does Apache Hudi approach schema compatibility?

Apache Hudi approaches schema compatibility through a configurable schema evolution framework that integrates with a schema registry. Hudi supports both backward and forward compatibility modes and can enforce them at write time by comparing the incoming schema against the registered schema for the table. The level of enforcement depends on the configured compatibility mode, which can range from fully backward compatible to no compatibility checking at all.

Hudi’s schema evolution support covers adding new nullable columns, deleting columns in certain configurations, and type promotions. In practice, Hudi’s schema handling is closely tied to the file format in use: tables written as Parquet have different constraints than those written as ORC. Hudi also provides a schema-on-read capability through its Hive sync and Spark integrations, meaning the schema presented to query engines can differ from the physical schema stored in files.

One area where Hudi stands out is its native support for record-level upserts and deletes, which interact with schema evolution in non-trivial ways. When a record is updated under a new schema, Hudi must reconcile the old physical record with the new schema definition. Its merge-on-read table type handles this by deferring reconciliation to query time, which reduces write amplification but adds complexity to the read path.

What’s the difference between backward and forward schema compatibility?

Backward compatibility means a new schema can read data written with an old schema. Forward compatibility means an old schema can read data written with a new schema. The two are distinct and often need to be considered independently when planning schema changes in production systems.

A concrete example makes this clearer. Suppose you add a new optional column to a table. A reader using the new schema can read old files without that column by substituting a null or default value, and that’s backward compatibility. A reader using the old schema can read new files and simply ignore the new column, and that’s forward compatibility. Adding an optional column is therefore both backward and forward compatible, which is why it’s the most commonly recommended safe change.

Deleting a column breaks forward compatibility: an old reader expecting that column won’t find it in new files. Renaming a column breaks both directions unless the format supports aliasing. Full compatibility in both directions simultaneously is called full compatibility, and it’s the most restrictive mode, limiting changes essentially to adding optional fields. Data teams working with streaming and event-driven architectures need to be especially deliberate about compatibility direction, because producers and consumers often deploy independently.

Which open table format handles schema evolution best?

Apache Iceberg handles schema evolution most completely among the three major open table formats, primarily because of its column-ID-based metadata model. This design means that renaming, reordering, and adding columns are all non-breaking operations at the storage level, with no data rewrite required. The schema history is versioned alongside snapshot history, making time-travel queries schema-aware by default.

Delta Lake is a strong choice when the priority is schema enforcement and controlled evolution within a Spark-centric environment. Its opt-in evolution model and clear enforcement errors make it predictable in production pipelines, though column renames still require more care than in Iceberg.

Apache Hudi is worth evaluating when the workload is heavily upsert-driven and schema evolution is a secondary concern. Its compatibility framework is configurable and integrates with schema registries, but the interaction between schema changes and its merge strategies adds operational complexity that the other formats avoid.

For most data lakehouse use cases where schema stability can’t be guaranteed and the team needs to evolve tables without coordinating all consumers simultaneously, Iceberg’s approach is the most transparent and least error-prone. That said, the best format is always the one your query engine and storage layer support well, and ecosystem fit matters as much as feature comparison.

How can data teams automate schema evolution in production?

Data teams can automate schema evolution in production by combining a schema registry, compatibility checks in the CI/CD pipeline, and format-native evolution capabilities. The goal is to catch incompatible changes before they reach production and apply compatible changes automatically without manual intervention.

A practical automation approach involves several layers:

Schema registry integration: Register every table schema in a central registry (such as Confluent Schema Registry or AWS Glue Schema Registry) and configure producers to validate against it before writing. This catches breaking changes at the source.
Compatibility checks in CI/CD: Add a schema diff step to your data pipeline CI/CD process that compares the proposed schema against the current registered version and fails the build if the change violates the configured compatibility mode.
Format-native evolution flags: Use the built-in evolution options of your open table format, such as Iceberg’s ALTER TABLE commands or Delta Lake’s mergeSchema option, to apply safe changes automatically when they pass compatibility checks.
Automated migration jobs: For breaking changes that can’t be avoided, automate the migration job as part of the deployment process, including validation of row counts and sample data before and after the migration.
Monitoring and alerting: Track schema version metrics alongside pipeline health metrics. Alert when a table’s schema version changes unexpectedly or when a downstream job encounters a schema mismatch error.

The key principle is to make schema changes explicit and traceable rather than implicit and reactive. When a schema change is treated with the same discipline as a code change, including review, testing, and versioning, the risk of silent data corruption drops significantly. Data engineering teams that invest in this infrastructure early avoid the category of bugs that are hardest to debug: the ones where the pipeline ran successfully but the numbers were wrong.

How Stackable helps with schema evolution

The Stackable Data Platform (SDP) gives data engineering teams a modular, Kubernetes-native foundation for running the open table formats and query engines that make schema evolution tractable in practice. Rather than managing each component in isolation, the SDP provides a composable stack where the tools that schema evolution depends on work together out of the box:

Apache Iceberg support via Trino and Apache Spark™: The SDP includes operators for both Trino and Apache Spark™, both of which support Iceberg as a first-class table format. Teams can run schema evolution operations through standard SQL or Spark APIs without custom infrastructure work.
Infrastructure as code: All platform components are configured declaratively. Schema-related configurations, catalog settings, and format options are version-controlled alongside application code, making schema management reproducible and auditable.
Open and cloud-agnostic: Because the SDP is 100% open source and runs on any Kubernetes cluster, on-premises or in the cloud, you’re not tied to a specific cloud provider’s schema registry or catalog service. Data sovereignty is preserved by design.
Integrated stack, no lock-in: The platform bundles the components data teams need for a modern data lakehouse without proprietary dependencies, so schema evolution tooling you build today remains portable tomorrow.

If you want to see how the SDP handles real data lakehouse workloads, the demo environment is a good starting point. For questions about your specific setup, reach out to the team directly.

Apache Kafka® is a registered trademark of the Apache Software Foundation. Apache Spark™, Apache Iceberg, Apache Hive™, and Apache Hadoop® are trademarks of the Apache Software Foundation. Trino is a trademark of the Trino Software Foundation. All other trademarks are the property of their respective owners.