How do you run a data lakehouse natively on Kubernetes?

Running a data lakehouse on Kubernetes means deploying open-source table format engines, query processors, and object storage connectors as containerized workloads managed by Kubernetes operators. The combination of Apache Iceberg™ for the table format, Apache Spark™ or Trino for compute, and S3-compatible object storage gives you a fully functional lakehouse that runs on any infrastructure Kubernetes supports. Here’s how each layer works and how the pieces connect.

What components make up a data lakehouse architecture?

A data lakehouse combines the low-cost, schema-flexible storage of a data lake with the structured query capabilities and ACID transaction guarantees of a data warehouse. The core components are a table format layer, a compute engine, a metadata catalog, and an object storage backend. Together, they let you run SQL analytics, machine learning workloads, and streaming ingestion against the same data store.

Breaking those layers down:

Table format: Apache Iceberg™ (or Apache Hive™ metastore-compatible formats) defines how data files are organized, versioned, and tracked. Iceberg adds snapshot isolation, schema evolution, and partition pruning on top of raw Parquet or ORC files.
Compute engine: Apache Spark™ handles batch processing and large-scale transformations. Trino handles interactive, low-latency SQL queries across multiple sources. Both can read and write Iceberg tables simultaneously.
Metadata catalog: A Hive metastore or a REST-compatible Iceberg catalog stores table definitions and snapshot history. This is the shared contract between compute engines.
Object storage: S3-compatible storage, whether cloud-hosted or on-premises via MinIO or Ceph, holds the actual data files. Keeping compute and storage separate is what makes the lakehouse model elastic.

The lakehouse pattern removes the need to maintain separate systems for exploration (the lake) and reporting (the warehouse). One consistent data layer serves both purposes.

Why is Kubernetes a natural fit for data lakehouse deployments?

Kubernetes solves the hardest operational problems that distributed data systems create: resource scheduling, workload isolation, configuration management, and lifecycle automation. A lakehouse isn’t a single application. It’s a collection of stateful and stateless services that need to co-exist, scale independently, and recover from failure without someone manually stepping in.

Several Kubernetes properties map directly to lakehouse requirements:

Declarative configuration: You describe the desired state of each component in YAML manifests. Kubernetes continuously reconciles the actual state toward that desired state, which makes deployments reproducible and auditable.
Independent scaling: Spark executor pools, Trino worker nodes, and catalog services each scale on their own schedules. Kubernetes handles bin-packing and resource allocation across nodes without you writing custom orchestration logic.
Portability: The same manifests that run on a bare-metal cluster in a private data center run on managed Kubernetes services in any cloud. This is the foundation of a cloud-agnostic data lakehouse strategy.
Operator pattern: Kubernetes operators encode domain-specific operational knowledge, how to upgrade Spark, how to reconfigure Trino, how to handle rolling restarts, into controllers that run continuously. This replaces manual runbooks.

The operator pattern is particularly significant for a lakehouse. Each major component (Spark, Trino, the catalog) carries enough operational complexity that automating it through a dedicated controller pays off quickly at production scale.

What open-source tools power a Kubernetes-native data lakehouse?

A Kubernetes-native data lakehouse is built on a set of well-established open-source projects: Apache Spark™ for batch compute, Trino for interactive SQL, Apache Iceberg™ as the table format, Apache Hive™ metastore or an Iceberg REST catalog for metadata, and S3-compatible object storage for persistence. These tools are all cloud-agnostic and have active, independent communities.

Compute engines: Spark and Trino

Apache Spark™ is the standard for large-scale data transformation, machine learning pipelines, and streaming ingestion via Spark Structured Streaming. On Kubernetes, Spark runs in client or cluster mode, with executor pods spun up dynamically and terminated when the job completes. This avoids the overhead of a persistent YARN cluster.

Trino is optimized for federated, interactive SQL. It can query Iceberg tables on object storage, join them with data in relational databases or other sources, and return results in seconds rather than minutes. Trino’s coordinator-worker architecture maps cleanly onto Kubernetes deployments, with workers scaling horizontally based on query load.

Table format and catalog: Iceberg and Hive Metastore

Apache Iceberg™ gives the lakehouse its warehouse-like properties: ACID transactions, time travel, schema evolution without rewrites, and hidden partitioning. The Hive metastore (or a modern Iceberg REST catalog) stores the pointer from a table name to the current snapshot manifest. Both Spark and Trino support Iceberg natively, so they can share tables without format conversion.

How does storage work in a Kubernetes-based lakehouse?

Storage follows a decoupled model: compute pods are ephemeral and stateless, while data persists in S3-compatible object storage that exists outside the Kubernetes cluster. Kubernetes persistent volumes are used only for operational state, catalog databases, small configuration stores, not for the data files themselves.

This separation has practical consequences for how you design the system:

Spark and Trino pods read and write data files directly over the S3 API. When a pod is terminated, no data is lost because the files live in object storage, not on the pod’s local disk.
On-premises deployments typically use MinIO or Ceph to provide an S3-compatible endpoint. This means the same application code runs unchanged whether the backend is AWS S3, Google Cloud Storage, Azure Blob Storage, or a self-hosted cluster.
Iceberg’s snapshot model means writes are atomic at the object storage level. A new snapshot is committed only after all data files are written, so partial writes never corrupt the table’s readable state.
Kubernetes PersistentVolumeClaims are appropriate for the Hive metastore’s relational database (typically PostgreSQL) and for Trino’s coordinator state. These are small, low-throughput workloads compared to the data plane.

The key design principle: treat object storage as the system of record and Kubernetes as the compute layer that accesses it. This keeps the architecture stateless at the Kubernetes layer, which simplifies upgrades, node replacements, and multi-region deployments.

How do you deploy a data lakehouse on Kubernetes step by step?

Deployment follows a layered sequence: provision the cluster, deploy object storage, install the metadata catalog, deploy the compute engines via operators or Helm charts, and validate end-to-end connectivity. Each layer depends on the one below it, so the order matters.

Provision a Kubernetes cluster: This can be a managed cluster (EKS, GKE, AKS) or a self-hosted cluster on bare metal or VMs. Make sure nodes have sufficient CPU and memory headroom for Spark executors and Trino workers running concurrently.
Deploy S3-compatible object storage: For on-premises setups, deploy MinIO using its Kubernetes operator. Configure a bucket for lakehouse data and a separate bucket for Iceberg metadata if your catalog requires it.
Deploy the Hive Metastore or Iceberg REST catalog: The metastore needs a PostgreSQL backend. Deploy PostgreSQL first (using a Kubernetes operator), then deploy the metastore service pointed at that database. Expose it as a Kubernetes service so compute engines can reach it by DNS name.
Deploy Trino: Use the official Trino Helm chart or a Kubernetes operator. Configure the Iceberg connector with the metastore URI and the S3 endpoint and credentials. Set resource limits on coordinator and worker pods.
Deploy Apache Spark™: Use the Spark operator (maintained under the Kubeflow project) to submit SparkApplications as custom resources. Configure the Iceberg Spark runtime JAR and the same catalog settings used by Trino so both engines share table definitions.
Validate connectivity: Run a Spark job that creates an Iceberg table and writes sample data. Then run a Trino query against the same table. If both succeed and return consistent results, the integration is working correctly.

Configuration management is the most common source of errors at this stage. Keeping catalog URIs, S3 endpoints, and credentials consistent across Spark and Trino configurations, ideally via Kubernetes ConfigMaps and Secrets, prevents the subtle mismatches that cause jobs to fail silently. The available platform demos can give you a working reference point before you write your own manifests.

What are the main challenges of running a lakehouse on Kubernetes?

The main challenges are resource contention between compute engines, catalog consistency under concurrent writes, networking complexity for data-intensive workloads, and the operational overhead of managing multiple interdependent components. None of these are blockers, but each requires deliberate design choices.

Resource contention and scheduling

Spark and Trino compete for the same node resources. Without proper resource quotas and priority classes, a large Spark job can starve Trino workers and degrade interactive query performance. Kubernetes namespace-level resource quotas and pod priority classes give you the controls to separate batch and interactive workloads, but you need to configure them explicitly. Kubernetes doesn’t apply sensible defaults for mixed data workloads.

Catalog consistency and concurrent writes

Apache Iceberg™ handles concurrent writes through optimistic concurrency control: if two writers attempt to commit conflicting snapshots, one will fail and must retry. This is correct behavior, but it requires your ingestion pipelines to handle retry logic. The Hive metastore itself can become a bottleneck under high commit rates because it uses row-level locks in its relational database. For high-throughput ingestion, consider a dedicated Iceberg REST catalog that offloads this coordination.

Networking and data locality

Kubernetes networking is designed for service-to-service communication, not high-throughput data transfer. When Spark executors pull large volumes of data from object storage, the bandwidth available through the cluster’s overlay network becomes a constraint. Running object storage close to the Kubernetes cluster, ideally in the same data center or availability zone, reduces latency and avoids egress costs.

How does a Kubernetes lakehouse support data sovereignty and avoid vendor lock-in?

A Kubernetes-native data lakehouse supports data sovereignty by keeping your data in infrastructure you control and using open formats and protocols that no single vendor owns. The combination of S3-compatible storage, Apache Iceberg™ as an open table format, and Kubernetes as a portable orchestration layer means you can move the entire stack between environments without changing your data or your application logic.

The specific mechanisms that prevent lock-in:

Open table format: Iceberg is not a proprietary format. Any engine that implements the Iceberg spec, Spark, Trino, Flink, DuckDB, and others, can read your tables. You’re not tied to a specific query engine.
S3 API compatibility: The S3 API is a de facto standard implemented by every major cloud provider and many on-premises solutions. Your data access code doesn’t change when you switch storage backends.
Kubernetes portability: Kubernetes runs on every major cloud and on bare metal. Your deployment manifests are portable. Migrating from one environment to another is an infrastructure change, not an application rewrite.
No proprietary connectors: The open-source components in this stack communicate over standard protocols (Thrift for the Hive metastore, HTTP for the Iceberg REST catalog, JDBC for query results). No proprietary SDKs required.

Data sovereignty goes beyond avoiding lock-in. It means knowing exactly where your data is, who can access it, and under what conditions. A self-hosted Kubernetes lakehouse gives you that control in a way that a fully managed cloud service, by design, does not. For organizations in regulated industries, this distinction matters for compliance as much as for cost.

How Stackable helps with running a data lakehouse on Kubernetes

The Stackable Data Platform (SDP) is built specifically for this kind of deployment. Rather than assembling Helm charts, operators, and configuration glue yourself, the SDP provides a curated, tested set of Kubernetes operators for the components that make up a Kubernetes-native data lakehouse:

Stackable Operator for Apache Spark™ manages the full lifecycle of Spark applications as Kubernetes-native resources, including configuration, upgrades, and monitoring integration.
Trino operator handles coordinator and worker deployment, catalog configuration, and rolling upgrades without manual intervention.
Hive Metastore operator manages the catalog service and its PostgreSQL dependency as a single, declarative unit.
Open-source and cloud-agnostic by design: The SDP runs on any Kubernetes cluster, on-premises, in any cloud, at the edge, or in a hybrid setup. No proprietary extensions, no cloud-provider dependencies.
Infrastructure as code from day one: All components are configured declaratively, making deployments reproducible, traceable, and auditable.

If you want to see a working data lakehouse on Kubernetes before committing to a full deployment, the SDP demo catalog includes end-to-end examples. To discuss your specific architecture or compliance requirements, reach out to the Stackable team directly.