Apache Airflow® on the
Stackable Data Platform

The Stackable Operator supports the CeleryExecutor, for distributed task execution across a pool of worker pods connected via Redis, and the KubernetesExecutor, which dynamically launches a new pod per task for elastic scaling and workload isolation. Because both are integrated with the operator, you can switch execution modes declaratively and scale resources without re-architecting your deployment.

DAGs can be mounted declaratively in the AirflowCluster CRD: ConfigMaps are ideal for static, version-controlled DAGs, while git-sync sidecars continuously fetch DAGs from a Git repository for GitOps-style workflows. Stackable’s declarative approach keeps DAG distribution reproducible and auditable, fully integrated into your Kubernetes workflows.

After deploying Airflow with the Stackable Operator, follow the Getting Started guide to run the example DAGs packaged with the operator. These are already configured to demonstrate Kubernetes-native scheduling and can be triggered directly from the Airflow UI or API – a reproducible entry point into orchestrating jobs on the Stackable platform.

Absolutely. With the Stackable Operator, Airflow becomes the orchestration layer for the entire Stackable Data Platform. From a DAG you can launch a SparkApplication CRD, trigger data ingestion into NiFi, or query datasets via Trino. This tight integration makes Airflow a central control plane across distributed data pipelines, beyond just scheduling Python tasks.

The operator uses ListenerClasses, Stackable’s abstraction for managing ingress and service exposure. Whether you need a NodePort, a LoadBalancer or integration with an Ingress controller, you can configure external access declaratively in your CRD. This keeps networking setup consistent across the entire Stackable platform and aligns with enterprise-grade security policies.

The operator provides default resource requests and limits for Airflow components, so even small clusters start with sensible defaults. For production, the docs include minimal HA examples and tuning guidance for schedulers and workers. With Kubernetes-native scaling and Pod Disruption Budgets managed by the operator, Airflow clusters grow elastically while staying stable during upgrades and node failures.

Yes. The Stackable Operator integrates Kubernetes-native best practices like preStop hooks and termination grace periods, coordinating with the operator’s lifecycle management so schedulers and workers complete active tasks before being terminated. This makes rolling upgrades and maintenance windows safer and more predictable, avoiding job loss and keeping workflows consistent.

Each Airflow pod runs with a Vector sidecar, Stackable’s standard logging solution. Logs can be forwarded to Elasticsearch, OpenSearch, S3 or other backends for unified observability across your data platform. Airflow logs are collected, enriched and shipped the same way as Kafka, Spark, NiFi and other Stackable components, simplifying monitoring and troubleshooting.

Yes. Each Airflow component is deployed with a metrics sidecar, integrating seamlessly with the Prometheus and Grafana dashboards provided by Stackable. You get out-of-the-box observability for scheduler performance, DAG runtime metrics, worker load and cluster health – without extra setup.

Yes. The Stackable Operator supports the KubernetesExecutor while adding persistent logging options. Task logs can be stored in S3, MinIO or PVCs using Stackable’s built-in logging integration, so even when task pods are short-lived, logs remain accessible for debugging, auditing and compliance.

The full specification of the AirflowCluster Custom Resource is available in the Stackable CRD Reference. It documents every configurable field – from executors and dependency management (PostgreSQL, Redis) to logging, TLS and monitoring – so you can declaratively define production-ready clusters with the same consistency as any other Kubernetes resource.
Click here for CRD reference

Supported Apache Airflow versions are listed on the Stackable documentation’s supported-versions page, refreshed with every release as new versions arrive and older ones are deprecated. You choose a version through the image in your AirflowCluster, with custom registries and custom images supported as well.
Click here for the current list

Built-in user and role management, for quick setup.
LDAP or OIDC integration, for enterprise SSO and centralized identity management.
By reusing the same identity provider integrations available across the Stackable ecosystem, you get unified authentication and authorization across all platform components.

The operator has been tested on major managed and self-hosted Kubernetes platforms: EKS, AKS, GKE, OpenShift, IONOS and K3s. This flexibility lets you deploy Airflow reliably across cloud providers or on-premise infrastructure, while still benefiting from declarative management and operator automation.