Lambda vs. Kappa architecture: which fits a modern data lakehouse?

For most modern data lakehouse deployments, kappa architecture is the better fit. It treats all data as a stream, which cuts out the dual-pipeline complexity that lambda requires. That said, the right choice depends on your team’s tolerance for reprocessing costs, how mature your streaming infrastructure actually is, and whether your historical batch workloads are genuinely irreplaceable. We look specifically at how the Stackable Data Platform (SDP) supports both approaches at the end.

What are the core differences between lambda and kappa architecture?

Lambda architecture splits data processing into two separate layers: a batch layer that reprocesses all historical data on a schedule, and a speed layer that handles real-time streaming. Results from both layers are merged in a serving layer. Kappa removes the batch layer entirely, using a single streaming pipeline to handle both real-time and historical data reprocessing.

The practical consequence of this split is significant. Lambda teams maintain two codebases that must produce identical results, which creates ongoing synchronization overhead. Any change to business logic has to be applied in both pipelines, tested independently, and reconciled at the serving layer. This is manageable when the batch layer handles genuinely complex transformations that streaming engines can’t yet express, but it becomes a real maintenance burden when the two layers start drifting apart.

Kappa architecture, proposed by Jay Kreps in 2014 as a simplification of lambda, bets that a single well-designed streaming system can handle everything. Historical reprocessing is achieved by replaying events from a durable log, such as Apache Kafka®, rather than running a separate batch job. The tradeoff is that your streaming infrastructure must handle sustained high-throughput replay, and your event log must retain data long enough to support whatever reprocessing window you need.

What is a data lakehouse and how does it change the architecture decision?

A data lakehouse combines the low-cost, flexible storage of a data lake with the structured query capabilities and ACID transaction support of a data warehouse. Open table formats such as Apache Iceberg provide schema evolution, time travel, and transactional guarantees directly on object storage, which fundamentally changes how both lambda and kappa architectures can operate.

In a traditional architecture, the batch and speed layers of lambda often used entirely different storage systems, a distributed file system for batch, an in-memory store for real-time results. A lakehouse collapses that distinction. Because Iceberg supports incremental writes, compaction, and snapshot isolation, a streaming engine can write directly to lakehouse storage in small, frequent commits without hurting query performance for analytical workloads.

This shift tilts the decision toward kappa. When your storage layer already supports the properties that batch processing was historically needed to provide, consistency, time travel, efficient large-scale reads, the argument for maintaining a separate batch pipeline weakens considerably. The lakehouse effectively absorbs much of the work the lambda batch layer used to do.

Why does lambda architecture struggle with modern lakehouse environments?

Lambda struggles in lakehouse environments primarily because the lakehouse’s transactional table formats reduce the need for a dedicated batch layer, while the dual-pipeline model adds complexity that the lakehouse’s unified storage was designed to eliminate. Maintaining two separate processing paths against a single Iceberg or Delta table introduces coordination challenges that negate many of the lakehouse’s benefits.

The problems surface in a few recurring ways. Merging batch and streaming results into a single lakehouse table requires careful management of write conflicts and snapshot ordering. Schema evolution, one of the key advantages of open table formats, becomes harder to apply consistently when two independent pipelines must agree on schema changes at the same time. And testing that both pipelines produce identical outputs is an ongoing engineering cost that grows as business logic evolves.

There’s also an operational consideration worth mentioning. Lakehouse environments are typically managed through infrastructure-as-code tooling, where declarative configuration governs how data flows through the system. A dual-pipeline lambda setup introduces more moving parts, more failure modes, and more configuration surface area to manage. For teams already investing in a modern data platform, that added complexity is rarely worth it.

How does kappa architecture work with streaming-first data platforms?

In a kappa architecture on a streaming-first platform, all data enters through a durable event log, Apache Kafka® being the most common choice. A stream processing engine, such as Apache Flink or Apache Spark™ Structured Streaming, consumes events continuously and writes results to the serving layer. Historical reprocessing is triggered by replaying the event log from an earlier offset, using the same processing logic as the live pipeline.

This approach has real implications for how you design your platform. Your event retention policy becomes a first-class architectural decision: the longer you retain events in Kafka, the further back you can reprocess without needing a separate historical store. Many teams configure tiered storage for Kafka topics to extend retention cost-effectively, keeping recent data on fast local storage and older data on object storage.

The stream processor itself must be stateful and fault-tolerant. Flink’s checkpointing mechanism, for example, allows a job to resume from a consistent state after failure, essential for long-running reprocessing jobs. When writing to a lakehouse, the processor should use exactly-once semantics where the table format supports it, ensuring that replayed events don’t produce duplicate records in the output table.

When should teams still choose lambda over kappa?

Lambda architecture remains a reasonable choice in specific situations: when your batch processing logic is genuinely too complex or computationally expensive to express as a continuous stream, when your streaming infrastructure can’t sustain the throughput required for full historical replay, or when regulatory requirements mandate a separate, auditable batch reconciliation process that runs independently of real-time ingestion.

A concrete example is large-scale machine learning feature engineering that runs weekly across years of historical data. If the computation requires full dataset scans with complex joins that a streaming engine handles inefficiently, a dedicated batch layer using Apache Spark™ may be more practical than forcing that logic into a streaming job. The key question is whether the operational cost of maintaining two pipelines is lower than the engineering cost of rewriting the batch logic for streaming.

Teams migrating from legacy systems also sometimes adopt lambda as a transitional architecture. Keeping an existing batch pipeline running while gradually building out streaming capabilities reduces migration risk. In this context, lambda is a stepping stone rather than a long-term target, and the goal should be to converge toward kappa once the streaming layer is proven and stable.

What tools and frameworks support each architecture in a lakehouse stack?

Tools commonly used in lambda architectures

Lambda stacks typically pair a batch engine with a separate streaming engine. Apache Spark™ is the most widely used batch layer, capable of processing large historical datasets efficiently. For the speed layer, Apache Kafka® combined with Flink or Spark Structured Streaming handles real-time ingestion. The serving layer often uses a query engine such as Trino to federate queries across both batch and streaming outputs stored in Iceberg tables.

Tools commonly used in kappa architectures

Kappa stacks center on Apache Kafka® as the durable event log and a stateful stream processor, most commonly Apache Flink, for both real-time and reprocessing workloads. Apache Iceberg provides the lakehouse storage layer, supporting incremental writes from the streaming engine and efficient analytical reads. Trino serves as the query engine for ad hoc analysis on top of Iceberg tables, providing a consistent interface regardless of whether data arrived via live streaming or historical replay. You can explore working demos of these tools in combination to see how they interact in practice.

Which architecture is better suited for data mesh deployments?

Kappa architecture aligns more naturally with data mesh principles. Data mesh distributes ownership of data products to domain teams, each responsible for producing and serving its own data. A single, well-defined streaming pipeline per domain is easier to own, test, and evolve independently than a dual-pipeline lambda setup where batch and streaming logic must be kept in sync across team boundaries.

In a data mesh, each domain exposes its data as a product through a standardized interface, typically a lakehouse table with a published schema and SLA. Kappa’s unified pipeline model makes it straightforward for a domain team to define exactly one processing path from raw events to the published product, reducing the coordination overhead that lambda would introduce. Schema changes, versioning, and lineage tracking are all simpler when there’s a single pipeline to reason about.

Lambda can work in a data mesh context, but it tends to push complexity upward. Either the platform team must provide tooling to manage dual pipelines consistently across domains, or each domain team must independently solve the synchronization problem. Neither outcome fits well with a model that’s supposed to reduce central platform dependencies and increase domain autonomy.

How Stackable supports streaming and lakehouse architectures

The Stackable Data Platform (SDP) is a modular, Kubernetes-native data platform that provides the operators and tooling needed to run both kappa and lambda architectures on your own infrastructure, without cloud provider lock-in. Whether you’re building a streaming-first lakehouse or managing a hybrid batch-and-streaming setup, the SDP gives you the composable building blocks to do it on your terms.

Specifically, the SDP provides:

The Stackable Operator for Apache Kafka® for deploying and managing Kafka clusters declaratively on Kubernetes, including configuration for tiered storage and topic retention policies relevant to kappa reprocessing.
Operators for Apache Spark™ and Trino, enabling batch processing and federated querying across Iceberg tables in a lambda or hybrid stack.
Native support for Apache Iceberg as a lakehouse table format, with consistent integration across streaming and batch engines.
Infrastructure-as-code configuration for all components, making it straightforward to version, audit, and reproduce your entire data architecture.
Data sovereignty by design: all components run in your environment, with no data leaving your control.

If you want to discuss which architecture fits your current setup and how the SDP can support it, get in touch with the Stackable team directly.