What is a catalog in an Iceberg lakehouse, and how do you choose one?

An Apache Iceberg catalog tracks where your table metadata lives. Without one, query engines can’t find Iceberg tables, resolve schema versions, or safely coordinate concurrent writes. Think of it as the address book of your lakehouse. Which catalog makes sense for you depends on your query engines, how you prefer to operate things, and whether you need features like Git-style branching or multi-table transactions. Let’s work through each of those decisions.

At the end, we look at how Stackable supports Iceberg-based lakehouses on Kubernetes.

What does an Iceberg catalog actually do?

At its core, an Iceberg catalog is a registry that maps table names to their current metadata file location. When a query engine like Trino or Apache Spark™ opens an Iceberg table, it asks the catalog one question: „Where is the metadata pointer for this table?“ The catalog returns a path, and from there the engine follows the chain of metadata files, manifests, and data files that Iceberg maintains.

But it does two other things that matter a lot in practice. First, it handles atomic commits: when a write finishes, the catalog updates the metadata pointer in a single atomic operation. That’s what stops readers from ever seeing a partial write. Second, it enforces optimistic concurrency control by comparing the expected metadata version against the current one before accepting a commit. If two writers race, one wins and the other retries. The table stays consistent without any locking.

The catalog doesn’t store actual data or metadata files. Those live in object storage or HDFS. The catalog only tracks the current root pointer, which is why you can swap catalogs without touching your data files.

What are the main types of Iceberg catalogs?

There are four catalog types you’ll commonly encounter: Hive Metastore, REST catalog, Nessie, and JDBC-based catalogs. Each stores and exposes table metadata differently, and each comes with its own trade-offs around compatibility, operational overhead, and features.

Hive Metastore (HMS): stores Iceberg metadata pointers in a relational database alongside traditional Hive table definitions. Widely supported and already present in many existing data platforms.
REST catalog: exposes a standardized HTTP API defined by the Iceberg REST Catalog specification. Any engine that implements the spec can connect, regardless of what’s running behind it.
Nessie: an open-source transactional catalog that layers Git-style branching and tagging on top of Iceberg table management. Built around a versioned commit model.
JDBC catalog: stores metadata pointers directly in any JDBC-compatible relational database. Simple to set up, but it lacks the richer capabilities of REST or Nessie.

Some managed cloud services offer proprietary catalog implementations too, but these tend to lock you into a specific platform and limit how portable your setup actually is.

What’s the difference between Hive Metastore and a REST catalog?

The key difference is the interface and the assumptions built into it. Hive Metastore uses a Thrift RPC protocol and a schema designed around Hive’s own table model. A REST catalog exposes a clean, versioned HTTP API that’s engine-agnostic and requires no Thrift client library. In practice, REST catalogs are easier to integrate with newer engines and easier to sit behind standard infrastructure like load balancers and API gateways.

Hive Metastore has a real advantage in installed base. Many organizations already run it as part of an existing Hadoop-era stack, and most major query engines, including Trino and Spark, have mature HMS connectors. If your infrastructure already includes HMS and you’re adding Iceberg tables incrementally, using it as your Iceberg catalog is often the path of least resistance.

A REST catalog is a better foundation for new deployments. Because the Iceberg REST Catalog specification is engine-neutral, you can swap the backend implementation without touching client configuration. Several open-source and managed implementations exist, including Apache Polaris (incubating), and the ecosystem is growing. REST catalogs also handle multi-engine environments more cleanly because every client speaks the same HTTP protocol rather than sharing a Thrift dependency.

One thing worth knowing: Hive Metastore stores Iceberg metadata pointers in its own database schema, which can create subtle conflicts when the same table is accessed by both Hive and Iceberg-native engines. REST catalogs avoid this entirely by being Iceberg-native from the start.

What makes Nessie different from other Iceberg catalogs?

Nessie adds a Git-inspired versioning model on top of Iceberg table management. Where other catalogs only track the current state of a table, Nessie tracks a full commit history across multiple tables at once. You can create named branches and tags against that history, which means you can test schema changes or data transformations on a branch without touching production, then merge when you’re ready.

This branching model has genuine operational value in specific situations. Data engineers can validate pipeline changes against a development branch before promoting to production. Data scientists can pin an experiment to a specific tag and reproduce results exactly, even if the underlying tables have changed since. Teams running regulatory reporting can tag the exact state of all relevant tables at a reporting date and re-query that snapshot months later.

That said, Nessie isn’t the right fit for every team. The branching model adds conceptual overhead, and if your workflows are straightforward append-and-query pipelines, the extra capability just goes unused. Nessie also requires its own service, which is another operational component to manage. For teams that genuinely need cross-table transactions or reproducible data snapshots, though, it offers something no other open-source Iceberg catalog currently matches.

How does catalog choice affect query engine compatibility?

Your catalog choice directly determines which query engines can read and write your Iceberg tables without extra work. Trino, Apache Spark™, Apache Flink, and Dremio all support HMS and REST catalogs. Nessie has connectors for Spark and Dremio, with Trino support available through a Nessie-backed REST endpoint. JDBC catalogs are supported by Spark but have more limited support elsewhere.

In a multi-engine environment, where Spark handles batch ingestion and Trino handles interactive queries, for example, the REST catalog is currently the safest choice for broad compatibility. The Iceberg REST Catalog spec was designed precisely to decouple the engine from the catalog backend, so adding a new engine later doesn’t require reconfiguring your catalog layer.

One subtlety worth knowing: even when two engines both support the same catalog type, they may behave differently around catalog caching, metadata refresh intervals, and conflict resolution. Testing your specific engine combination against your catalog before going to production is always worth doing, regardless of which catalog you choose.

Should you run your own catalog or use a managed one?

Running your own catalog gives you full control over data locality, access policies, and upgrade timing. It also means you’re responsible for availability, backups, and day-to-day maintenance. A managed catalog offloads that operational burden but introduces a dependency on an external service, which may conflict with data sovereignty requirements or regulatory constraints.

For organizations with strict data sovereignty requirements, a self-hosted catalog is often the only realistic path. If your data can’t leave a specific jurisdiction or must stay within your own infrastructure perimeter, a managed catalog hosted by a third party may simply not be acceptable, regardless of how good it is technically. This comes up regularly in financial services, healthcare, and the public sector.

For teams without those constraints, a managed catalog can cut time to production significantly. The trade-off is accepting the provider’s release cadence, pricing model, and availability guarantees. Before committing, it’s worth checking whether the managed catalog supports the Iceberg REST spec, because that determines whether you can migrate to a self-hosted alternative later without rewriting your engine configurations.

What should you consider when choosing an Iceberg catalog?

The most important factors are engine compatibility, operational maturity, data sovereignty requirements, and the features your workflows actually need. No single catalog is right for every team, but working through these criteria will narrow the field quickly.

Engine compatibility: List the query engines you run today and plan to run in the next two years. Check which catalog types each engine supports natively and where community support is strongest.
Existing infrastructure: If you already operate a Hive Metastore, using it for Iceberg tables has a low adoption cost. If you’re building from scratch, a REST catalog is a cleaner foundation.
Data sovereignty and compliance: If your data must stay within a specific infrastructure perimeter, eliminate managed catalog options that can’t meet that requirement before evaluating anything else.
Versioning and branching needs: If your workflows involve data experimentation, regulatory snapshots, or multi-table transactions, look at Nessie specifically. If not, the added complexity isn’t worth it.
Operational capacity: Every self-hosted catalog is a service you own. Factor in the cost of running, monitoring, and upgrading it alongside your other infrastructure.
Migration path: Prefer catalogs that implement or expose the Iceberg REST spec. This keeps your options open: if your requirements change, you can swap the backend without reconfiguring every connected engine.

A practical starting point for most new deployments is a self-hosted REST catalog backed by a simple relational store, with Nessie as a considered upgrade if branching semantics become relevant later. That approach keeps your options open without over-engineering the catalog layer before you understand your actual usage patterns.

How Stackable supports Iceberg lakehouses

The Stackable Data Platform (SDP) is a modular, Kubernetes-native data platform that makes it straightforward to run the components of an Apache Iceberg lakehouse on your own infrastructure, whether on-premises, in the cloud, or in a hybrid setup. Because SDP is 100% open source and cloud-agnostic, you keep full control over your catalog, your metadata, and your data files without depending on a managed service.

Concretely, SDP supports the query engines and processing frameworks that integrate with Iceberg catalogs:

Trino for interactive SQL queries against Iceberg tables, with HMS and REST catalog support
Apache Spark™ for batch processing and data ingestion pipelines
Apache Kafka® for event streaming that feeds into lakehouse ingestion patterns
Automated provisioning and lifecycle management via Kubernetes Operators, so catalog services and query engines are configured consistently and repeatably

The infrastructure-as-code approach means your entire lakehouse stack, including the catalog layer, is defined declaratively and version-controlled. That supports reproducible deployments and makes it easier to meet the auditability requirements that regulated industries often face.

If you want to see how a lakehouse stack looks in practice on SDP, the platform demos are a good starting point. To discuss your specific architecture and catalog requirements, get in touch with the team directly.