Stackable Docs Hub

Stackable

Stackable

How to run Apache Kafka, Trino and Spark in your own data centre – a reference architecture for data sovereignty

Running a self-hosted big data stack is not the path of least resistance. Managed cloud services are faster to provision, and someone else handles the operational burden. But for organisations where data sovereignty is a genuine requirement, not a preference, the calculus changes. Regulatory frameworks such as the Digital Operational Resilience Act (DORA) and the NIS-2 Directive are pushing financial institutions and critical infrastructure operators to demonstrate control over their data and the systems that process it. Running Apache Kafka®, Apache Spark™, and Trino in your own data centre is one concrete way to meet that bar.

This post walks through a reference architecture for doing exactly that: a modular, Kubernetes-native open-source data stack that you can deploy on-premises without sacrificing the operational maturity you would expect from a cloud-native setup. We look at how Stackable approaches this problem at the end.

The case for on-premises big data in a cloud-first world

The default assumption in most infrastructure conversations is that cloud is the right answer unless proven otherwise. That assumption is reasonable for many workloads, but it breaks down when data residency, regulatory compliance, or total cost of ownership become primary constraints rather than secondary considerations.

European enterprises in particular are navigating a tightening regulatory environment. DORA requires financial entities to maintain operational resilience and demonstrate control over third-party ICT dependencies. The NIS-2 Directive extends similar obligations to a broader set of critical sectors. In both cases, the ability to audit, control, and if necessary isolate your data processing infrastructure is not optional. Hyperscaler-native solutions can satisfy some of these requirements, but they introduce dependencies on external providers that are difficult to fully audit or override.

There is also a practical cost argument. At scale, egress fees, storage costs, and compute pricing on public cloud platforms can grow significantly. Organisations with predictable, high-volume workloads often find that owning the hardware and operating the stack themselves produces a lower total cost of ownership over a three-to-five-year horizon. The trade-off is operational complexity, and that is exactly what a well-designed reference architecture is meant to address.

Core components of the reference architecture

A production-grade on-premises data platform built around open-source components needs to cover three functional layers: ingestion and streaming, processing and transformation, and query and analytics. Apache Kafka®, Apache Spark™, and Trino map cleanly onto these three layers respectively.

Apache Kafka for event streaming

Apache Kafka® handles the ingestion layer. It provides a distributed, fault-tolerant log that decouples producers from consumers and supports high-throughput event streaming. In an on-premises context, Kafka’s ability to run across multiple physical nodes with configurable replication makes it well suited to data centre deployments where hardware failure is a real operational concern rather than an abstraction.

Apache Spark for distributed processing

Apache Spark™ sits in the processing layer. It handles both batch and streaming workloads, supports machine learning pipelines via MLlib, and integrates with a wide range of storage backends including Apache Iceberg-based data lakehouses. Spark’s ability to run natively on Kubernetes means it fits naturally into a containerised data centre environment without requiring a separate resource manager such as YARN.

Trino for federated query

Trino provides the analytics and query layer. It is a distributed SQL query engine designed to query data where it lives, across object storage, relational databases, and other sources, without requiring data movement. In a self-hosted architecture, Trino’s connector model allows teams to expose a unified SQL interface over heterogeneous data sources while keeping the data itself in place. You can explore a working example of this pattern in the data lakehouse demo with Iceberg, Trino, and Spark.

How Kubernetes orchestrates Kafka, Spark, and Trino at scale

Kubernetes has become the standard substrate for running distributed workloads, and for good reason. It provides declarative configuration, self-healing, horizontal scaling, and a consistent operational model across environments. Running Kafka, Spark, and Trino on Kubernetes means you get these properties for your data stack without building custom orchestration logic.

Each component has specific requirements that Kubernetes handles well. Kafka brokers are stateful and need stable network identities and persistent storage, which Kubernetes StatefulSets and PersistentVolumeClaims provide. Spark executors are ephemeral and benefit from Kubernetes‘ ability to schedule and terminate pods dynamically based on job demand. Trino workers are stateless and scale horizontally, making them straightforward to manage as a Kubernetes Deployment.

Operator pattern for data applications

The Kubernetes Operator pattern extends the platform’s control loop to application-specific logic. Rather than managing Kafka or Trino through generic Helm charts and manual configuration, an operator encodes operational knowledge, such as how to perform a rolling upgrade, how to handle broker failure, or how to reconfigure a cluster without downtime, into a controller that runs continuously inside the cluster. This is the approach that makes Kubernetes genuinely practical for stateful data workloads rather than just theoretically possible.

Operators also enable infrastructure-as-code at the application level. You define the desired state of your Kafka cluster or Trino deployment in a Kubernetes custom resource, commit that to version control, and the operator reconciles the actual state to match. This makes your data platform configuration reproducible, auditable, and portable across environments.

Security and access control across the stack

Security in a self-hosted data stack is not a single configuration step. It spans authentication, authorisation, encryption in transit, encryption at rest, and audit logging, and it needs to be consistent across all three components.

For authentication, integrating with an existing identity provider via Kerberos or OpenID Connect (OIDC) is the standard approach in enterprise environments. Kafka supports both SASL/GSSAPI (Kerberos) and SASL/OAUTHBEARER (OIDC), Trino supports LDAP and OIDC, and Spark can inherit credentials from the Kubernetes service account or integrate with external secret management systems.

Authorisation and policy enforcement

Authorisation is where the stack gets more complex. Kafka uses ACLs to control which clients can produce or consume from which topics. Trino has a built-in access control framework that supports file-based rules and integrates with Open Policy Agent (OPA) for more expressive policy definitions. Spark jobs run under a Kubernetes service account, and RBAC policies control what those accounts can access within the cluster.

The key principle is to enforce authorisation as close to the data as possible and to make policies declarative and version-controlled. Policies as Code, where access rules are defined in files stored in a repository and applied automatically, gives you the auditability that regulatory frameworks require and reduces the risk of configuration drift over time.

Encryption and certificate management

TLS should be enabled for all inter-component communication: between Kafka brokers, between the Trino coordinator and workers, and between the Spark driver and executors. Certificate management at scale is non-trivial, and integrating with a certificate authority such as cert-manager running inside the cluster is a practical way to automate certificate issuance and rotation without manual intervention.

Integrating the stack into an existing corporate data centre

Most organisations deploying an on-premises data platform are not starting from a blank slate. There is existing network infrastructure, storage systems, identity management, and monitoring tooling that the new stack needs to integrate with rather than replace.

Network integration is typically the first challenge. Kubernetes clusters in a corporate data centre need to coexist with existing VLANs, firewall rules, and DNS configurations. Kafka’s listener configuration needs to expose brokers on addresses that external producers and consumers can reach, while keeping internal cluster communication on a separate network path. Trino’s coordinator needs to be reachable from BI tools and SQL clients, which often means exposing it through an internal load balancer or ingress controller.

Storage integration

For the data layer, the reference architecture assumes an S3-compatible object store running on-premises, such as MinIO, as the primary storage backend for Spark and Trino. This gives you the same API surface as cloud object storage while keeping data within your own infrastructure. Existing NFS or SAN storage can be used for Kafka’s persistent volumes if the performance characteristics are appropriate, though local NVMe storage on dedicated nodes typically gives better throughput for high-volume Kafka deployments.

Identity and secrets management

Integrating with an existing LDAP or Active Directory instance for authentication avoids creating a parallel identity system. For secrets, tools such as HashiCorp Vault or Kubernetes-native secret management with appropriate RBAC controls keep credentials out of configuration files and version control. The integrated data security demo illustrates how these layers can be composed in a working environment.

Monitoring, observability, and day-two operations

Getting the stack running is the first milestone. Keeping it running reliably over months and years is the harder problem, and it requires investing in observability from the start rather than retrofitting it later.

All three components expose metrics in Prometheus format. Kafka exposes broker, topic, and consumer group metrics. Trino exposes query execution metrics, worker resource usage, and connector-level statistics. Spark exposes executor metrics and job-level performance data. Scraping these with Prometheus and visualising them in Grafana gives you a unified view of the stack’s health without requiring proprietary monitoring agents.

Log aggregation and tracing

Structured logging from all components should be aggregated centrally. In a Kubernetes environment, this typically means shipping container logs to an OpenSearch or similar log management system via a log collector running as a DaemonSet. Correlating logs across Kafka, Spark, and Trino for a single data pipeline requires consistent use of trace IDs or correlation identifiers, which needs to be designed into the pipeline from the beginning.

Upgrade and maintenance cycles

Day-two operations include version upgrades, configuration changes, and capacity adjustments. Rolling upgrades for Kafka require careful sequencing to avoid data loss or consumer disruption. Trino upgrades are simpler because workers are stateless, but coordinator upgrades require draining active queries first. Spark job definitions are typically versioned independently of the cluster, so upgrading the Spark runtime is a cluster-level operation that does not necessarily affect running jobs if done with appropriate scheduling.

From reference architecture to production deployment

A reference architecture is a starting point, not a finished product. Moving from a documented design to a production deployment involves a series of decisions that depend on your specific hardware, network topology, team capabilities, and compliance requirements.

The most common gap between reference architecture and production is operational readiness. Teams that are new to running Kubernetes-native data workloads often underestimate the effort required to build runbooks, train on-call engineers, and establish change management processes for the stack. Starting with a smaller scope, perhaps Kafka and Trino before adding Spark, allows teams to build operational confidence incrementally.

Capacity planning is another area where the reference architecture needs to be grounded in real workload data. Kafka throughput requirements drive broker count and disk sizing. Trino query concurrency and data volume drive worker count and memory allocation. Spark job parallelism drives node count and CPU allocation. These numbers should come from profiling actual workloads, not from generic sizing guides.

Finally, the architecture should be validated against your specific compliance requirements before going to production. This means reviewing the security configuration against your organisation’s security baseline, confirming that audit logging covers the events your compliance team needs, and verifying that data residency controls are in place and testable. The JupyterHub and Keycloak demo shows one approach to identity-aware access control that can inform this validation process.

How Stackable supports on-premises data sovereignty

The Stackable Data Platform (SDP) is built specifically for organisations that want to run this kind of stack on their own infrastructure, without depending on a managed cloud service or a proprietary distribution. Here is what that looks like concretely:

  • Kubernetes-native operators for each component: The Stackable Operator for Apache Kafka®, the Stackable Operator for Apache Spark™, and the Stackable Operator for Trino encode operational knowledge for each component, handling upgrades, configuration, and failure recovery through the Kubernetes control loop.
  • Declarative, reproducible configuration: The entire stack is defined as Kubernetes custom resources, making your data platform configuration version-controlled, auditable, and portable across environments.
  • Integrated security: The SDP includes built-in support for TLS, OIDC, Kerberos, and OPA-based authorisation across all supported components, so security is not an afterthought.
  • 100% open source and cloud-agnostic: The SDP runs on-premises, in any cloud, or in a hybrid environment. There is no proprietary lock-in and no dependency on a specific cloud provider’s services.
  • Commercial support available: For teams that want operational backing, Stackable offers self-managed and managed service subscriptions alongside expert consulting and training.

If you want to see the architecture in action before committing to a deployment, the SDP demos provide working, deployable examples. To discuss your specific requirements, get in touch with the Stackable team.

Ähnliche Artikel

Comments are closed.