How does a data lakehouse support machine learning workloads?

A data lakehouse supports machine learning workloads by combining the scalable, flexible storage of a data lake with the structured query capabilities and governance of a data warehouse, all within a single architecture. This unified layer gives ML teams direct access to raw and curated data without the pipeline complexity of moving data between systems. The sections below cover each layer of that story, from storage organization to real-time inference.

We built the Stackable Data Platform specifically for teams running these kinds of workloads on Kubernetes, and we cover how it fits into this picture at the end.

What makes a data lakehouse different from a data lake or warehouse for ML?

A data lake gives you flexibility but poor governance. A warehouse gives you structure but struggles with unstructured data at scale. The lakehouse sits in between, offering structured metadata, ACID transaction support, and query optimization directly on top of open file storage. That combination is exactly what ML workloads need.

ML pipelines typically require access to large volumes of raw data, images, logs, text, sensor readings, alongside structured feature tables and labeled datasets. A traditional data warehouse can’t handle unstructured formats and large binary files well. A data lake handles the volume fine but lacks the schema enforcement and versioning that reproducible training pipelines depend on.

The lakehouse solves this by adding a metadata and transaction layer on top of object storage, most commonly through open table formats like Apache Iceberg, Delta Lake, or Apache Hudi. This means you can run SQL queries, travel back to earlier dataset versions, and enforce schema changes without moving data into a separate system. For ML teams, that means reproducible experiments, consistent feature access, and training runs you can actually audit.

How does a data lakehouse store and organize ML training data?

Training data is stored as columnar files, typically Parquet or ORC, in object storage, organized through open table formats that handle partitioning, versioning, and schema management. This keeps data accessible to both SQL engines and ML frameworks without duplication or format conversion.

In practice, training data is organized in zones: raw ingestion, cleaned and validated, and feature-ready. Each zone can be queried independently, so data engineers and ML engineers can work on different stages without getting in each other’s way.

Apache Iceberg is particularly useful here. It supports partition evolution, hidden partitioning, and snapshot isolation, so a training job can read a consistent snapshot of a dataset even while new data is being written. That eliminates a common source of non-determinism in ML pipelines. Time-travel queries also let you reconstruct exactly which data was used to train a specific model version, which matters for debugging and compliance.

What role does the lakehouse play in feature engineering and feature stores?

The lakehouse acts as the offline storage layer for feature stores, holding the historical data needed to compute, backfill, and serve features for model training. Feature engineering pipelines read from the lakehouse, compute derived features, and write the results back as structured tables that training jobs consume directly.

Many teams use the lakehouse as a lightweight feature store in its own right, especially early on. Feature tables are stored as versioned Iceberg or Delta tables, partitioned by entity ID and timestamp. This makes point-in-time correct joins possible, which is essential for avoiding training-serving skew.

More mature setups pair the lakehouse with a dedicated feature store like Feast or Hopsworks. The lakehouse handles offline, batch-oriented feature computation while the feature store manages online serving with low-latency lookups. Both systems share the same underlying data, which keeps offline and online feature definitions consistent.

How do ML frameworks like Apache Spark connect to a data lakehouse?

Apache Spark™ connects to a data lakehouse through catalog integrations and native table format readers. It reads and writes Iceberg, Delta, and Hudi tables directly using built-in or pluggable connectors, making the lakehouse a first-class data source for distributed training and feature computation.

The typical connection path uses a catalog service, a Hive Metastore or a REST catalog compatible with Apache Iceberg, to resolve table locations and schemas. Spark jobs then read data in parallel across partitions, which scales naturally with the size of the training dataset.

But Spark isn’t the only option. Ray, Dask, and PyTorch DataLoader can also access lakehouse data through the same Parquet files the table format manages. A preprocessing pipeline might use Spark for large-scale feature computation, while a training job uses PyTorch directly on the resulting Parquet files, all pointing at the same lakehouse tables. No single compute engine required.

How does a data lakehouse support model governance and experiment tracking?

The lakehouse supports model governance by providing traceable, versioned datasets that can be linked to specific training runs. When combined with an experiment tracking tool like MLflow, it becomes the foundation for reproducible, governed ML workflows.

Governance in ML has two dimensions: data lineage and model lineage. The lakehouse handles data lineage through snapshot versioning and schema history in the table format. Every training job can record which table snapshot it consumed, and that reference can be stored alongside the model artifact in an experiment tracker.

Model lineage, hyperparameters, metrics, artifact locations, is typically managed by a tool like MLflow, which stores model metadata and points back to the lakehouse table version used for training. This creates a complete audit trail: from raw data ingestion through feature computation to model registration. For regulated industries, that kind of traceability isn’t a nice-to-have. It’s a compliance requirement.

Can a data lakehouse handle real-time data for ML inference and streaming?

Yes, but it needs a streaming layer alongside it. The lakehouse itself is primarily a batch and micro-batch system. When paired with Apache Kafka®, streaming pipelines can continuously write new data into lakehouse tables, which inference pipelines or online feature stores then consume with low latency.

This architecture is sometimes called a streaming lakehouse. Kafka handles the real-time event stream. A stream processing layer, Apache Flink or Apache Spark™ Structured Streaming, reads from Kafka, applies transformations, and writes results into Iceberg tables with small commit intervals measured in seconds to minutes. Inference pipelines can then query the latest table state or subscribe to new commits.

For very low-latency inference at the sub-second level, the lakehouse alone isn’t the right serving layer. In those cases, the lakehouse stores historical and training data, while a separate online store, Redis, a key-value database, or a dedicated feature store, handles real-time lookups. Both systems stay synchronized through the same streaming pipeline that writes to the lakehouse.

What are the key challenges of running ML workloads on a data lakehouse?

A few challenges come up consistently: small file accumulation, point-in-time correctness for training data, compute resource isolation between data engineering and ML jobs, and schema evolution across long-lived feature tables.

Small file accumulation is a persistent operational headache. Streaming writes produce many small Parquet files, which degrade query performance over time. Regular compaction jobs are necessary, and without automated maintenance, this becomes a manual burden that slows down training pipelines.

Point-in-time correctness is conceptually straightforward but operationally tricky. If a training job doesn’t explicitly pin a snapshot version, it may read different data on reruns, making experiments non-reproducible. Teams need either discipline or tooling to enforce snapshot references in training code.

Compute isolation is another practical concern. A large Spark training job and a real-time feature computation pipeline competing for the same cluster resources will cause latency spikes and pipeline failures. Kubernetes-native deployments help here because resource quotas and namespace isolation give each workload its own compute boundaries.

Schema evolution in long-lived feature tables also requires careful coordination. Adding a column is usually safe with Iceberg’s schema evolution support, but removing or renaming columns can break downstream training jobs that depend on specific field names. A clear schema change process, with versioned table contracts, is worth establishing early.

How Stackable helps with data lakehouse machine learning workloads

The Stackable Data Platform (SDP) is a modular, Kubernetes-native open-source data platform that provides the infrastructure layer for running data lakehouse and ML workloads without vendor lock-in. Rather than assembling and operating each component separately, teams get a curated, pre-integrated set of open-source tools that work together from day one.

Specifically, the SDP helps with:

Integrated streaming and storage: The Stackable Operator for Apache Kafka® manages Kafka deployments on Kubernetes, so the real-time ingestion layer for your lakehouse is handled alongside the rest of your platform.
Apache Spark™ on Kubernetes: The SDP includes a Spark operator that runs distributed training and feature computation jobs natively on Kubernetes, with resource isolation between workloads.
Open table format support: The platform works with Apache Iceberg, giving ML teams versioned, time-travel-capable datasets without proprietary format dependencies.
Data sovereignty: Because the SDP is 100% open source and cloud-agnostic, your training data and model artifacts stay in your own infrastructure, whether on-premises, in the cloud, or in a hybrid setup.
Infrastructure as code: Automated provisioning and lifecycle management means your lakehouse components, Kafka, Spark, Trino, and the rest, are configured reproducibly and updated without manual intervention.

If you want to see how this fits together in practice, the SDP demos show working lakehouse and streaming setups you can run yourself. Or if you have a specific ML architecture in mind, talk to us directly and we can walk through what a deployment would look like for your setup.