This started as a talk at the Iceberg Meetup in Münster (6 October 2026) and here’s the written version for everyone else who is interested in the topic.
I’ll be honest: I rarely use Apache Iceberg myself. Our Stackable customers do – and our job is to keep their setups running as tool versions change and features appear and disappear underneath them. So this isn’t a greenfield success story. It’s a collection of the pitfalls we’ve watched real Iceberg lakehouses hit in production on Kubernetes, the error messages that come with them, and the patches we’ve had to write. Everything here – including the patches – is open source; if I say we fixed something, you’ll find it on our GitHub.
TL;DR
- A table that commits every few minutes and is never compacted doesn’t just get slow – it can take your Trino coordinator down with out-of-memory errors. The problem is rarely data volume; it’s commit count.
- Compacting a partition that’s still being written to loses a race. Iceberg uses optimistic concurrency, so the first writer to commit wins and the other fails. Only compact cold partitions.
- Don’t keep history by copying tables daily – that’s up to ~2,555× amplification over seven years (one 100 MB table becomes ~256 GB). Iceberg tags give you the same point-in-time reads for a fraction of the storage.
- The recurring tax is the catalog-to-engine interface. Across recent releases, Hive Metastore and NiFi changes broke Trino statistics, Spark’s Iceberg integration, and NiFi’s Iceberg support outright – each needed a patch. It’s a big part of why we see the Iceberg REST catalog as the way forward.
What does the lakehouse actually look like?
Before the pitfalls, let’s take a look at the cast. The setup I’ll refer to is from our data lakehouse demo: a handful of open-source engines that all read and write the same Iceberg tables, with a catalog tracking table metadata and object storage holding the data.
Apache NiFi pulls real-time sample data off the internet and writes it into Apache Kafka®. Apache Spark™ is the workhorse – the most complicated tool in the stack, and the one doing the heavy lifting. It handles batch writes and streaming micro-batches: every five minutes it reads the last five minutes off Kafka and appends to Iceberg. The tables live in an object store (MinIO in the demo, but any S3-compatible store works).
On the query side, Trino is the SQL engine – you point SQL at it and it behaves like a database over your lakehouse. Authorization runs through Open Policy Agent (OPA), which is where row-level and column-level security live. Apache Superset sits on top for dashboards that are, under the hood, just SQL queries against Trino. In this demo the catalog is an Apache Hive™ Metastore.
One honest note about reality: Iceberg-only is a lovely goal, and not every customer gets there. There’s always that one tool that only emits Parquet, or CSVs that arrive by email. So alongside the Iceberg lakehouse zone we keep a staging zone – where a folder in object storage is treated as a table – for the data that isn’t ready to be a proper Iceberg table yet. Plan for it; because pretending it won’t happen doesn’t make it go away.
Pitfalls we see in production
The following examples are customer setups, but in fairness they’re traps any of us could walk into. None of them is exotic; all of them are avoidable once you know they exist.
Why does a “small” Iceberg table get slow – and take Trino down?
That’s because Iceberg writes multiple new files per commit. Insert rows one at a time and you can end up with six data files for six rows; every commit also needs at least three metadata files. Now add a streaming job that writes every five minutes and never cleans up – work out how many five-minute windows there are in a day, multiply by the metadata overhead, and the file count runs away from you. A simple SELECT … GROUP BY then has to wade through all of those data and metadata files.
The first complaint is usually “everything is so slow” – followed by “but we don’t even have much data.” And that’s the tell: it wasn’t the data, it was the commits. It only became urgent when the Trino coordinator, which has to pull all that metadata into memory to plan the query, ate more and more heap until it fell over.
The fix is just unglamorous, scheduled maintenance. If you already run Apache Airflow, an Airflow job is the natural home; if you don’t, a plain Kubernetes CronJob is plenty – no need to bring out the big guns. A small job that enumerates your schemas and tables and, for each table, does two things:
- Compaction: rewrite many small files into fewer, larger ones.
- Expire snapshots: drop the small-file clutter you no longer need for time travel.
One nuance that trips people up: expiring snapshots does not delete your data. A year of data stays a year of data. It trims the churn older than your retention window and limits time travel to that window – so after expiry you can travel back seven days, say, but not fourteen. You do eventually need those old files actually removed from object storage, which is the other half of the job.
One more knob worth understanding: target file size, typically around 256 MB. That number is a Hadoop inheritance (the old 128 MB block size), and the engines are tuned around it – too many tiny files hurt read performance and, on S3, rack up a lot of expensive GET requests; too-large files force the query coordinator to crunch a giant chunk in its heap and, again, fall over. Aim for the middle. And if you genuinely only bulk-load once a night and already write ~256 MB files, you have nothing to compact.
Why does my compaction job keep failing with commit conflicts?
Because you’re compacting data that’s still being written. Compaction isn’t instant – reading, reshuffling and rewriting a partition takes minutes. Run it hourly against a table that a streaming job is continuously appending to, and most of the time it’s fine; occasionally it collides, retries (four times by default), exhausts the retries, and one side – the compaction or the insert – gives up with an error like:
Base metadata location '<old>' is not same as the
current table metadata location '<new>' for db.tableTo see why, it helps to know how Iceberg handles concurrency. A classic database locks the table, makes the change, and releases the lock. Iceberg doesn’t: it uses optimistic concurrency. Everyone writes at once, and only at commit time does Iceberg check whether someone else has committed since you started. The first writer to commit wins; the second is told, in effect, “you were too slow, try again.” With the Hive Metastore, the mismatch above is what that collision looks like.
The fix is to only compact cold partitions – the ones no longer being written to. What counts as “cold” depends on your jobs; for a table partitioned by day, compact yesterday-and-older with a couple of hours of safety margin:
ALTER TABLE iceberg.myschema.events
EXECUTE optimize
WHERE event_ts < date_trunc('day', current_timestamp - INTERVAL '2' HOUR);A Trino limitation to note: the WHERE clause has to match whole partitions, not arbitrary rows. Which is a good reason to partition your tables in the first place – it pays off for exactly this kind of maintenance.
An open question I’ll admit I haven’t fully answered: the Hive Metastore is fairly dumb here – it just bumps a version number, so any concurrent commit loses. A REST catalog could in principle be smarter: if two commits only append rows, they don’t actually conflict, and both could be applied server-side. Whether that cleanly solves the hot-partition case, I genuinely don’t know yet. It’s on my list to test.
How do I keep years of history without exploding storage?
The naive approach is to copy every table daily – orders becomes orders_2026_10_06 – so you can look back later. One customer did this across ~30–50 tables, kept for seven years. That duplicates the oldest records roughly 2,555 times; a 100 MB table balloons to about 256 GB. (These weren’t even Iceberg tables – the backup was plain CSV/Parquet copies.)
It could have been worse. The backup job had quietly died about six months earlier – it broke, came back, broke again – which is the only reason the storage bill never fully detonated. A side note from the same setup: around 250,000 tables in a single Hive Metastore schema gets slow, and you start raising timeouts to cope.
The better answer is Iceberg’s own time-travel features. Everything in Iceberg is already a snapshot, so give the snapshots you care about a tag – a named, retained reference. Expire-snapshots keeps tagged snapshots for as long as you tell it, so you can hold on to end-of-year or end-of-month points without keeping all the daily churn:
-- Keep an end-of-year snapshot effectively forever
ALTER TABLE prod.db.table CREATE TAG `EOY-2023` AS OF VERSION 365 RETAIN 2555 DAYS;
-- Keep the first end-of-month snapshot for six months
ALTER TABLE prod.db.table CREATE TAG `EOM-01` AS OF VERSION 30 RETAIN 180 DAYS;Even conservatively that’s around a 100× storage saving, and with Iceberg’s more granular branching and tagging you can save considerably more.
Two honest disclaimers. First, check your engine support. In our experience Spark is the most mature Iceberg client in the big-data space and is currently the only one that can create tags; Trino can read and time-travel to them, but not create them. (The example above is Spark SQL rather than Trino SQL for exactly that reason.) Second, and more importantly: if you point an older engine at the table – one that predates the tag concept – and run expire-snapshots, it may happily delete the snapshot a tag was protecting. Use up-to-date tools and verify tag support before you rely on it.
The part that actually hurts: catalogs and engine versions
The pitfalls above are mostly self-inflicted and fixable with discipline. The harder, recurring problem is keeping the same setup working as the versions move underneath you – and the riskiest piece is the catalog, because every engine talks to it.
Why is the catalog the riskiest part of the stack?
Our SDP support matrix for the Hive Metastore goes back to 2022-06. Until 3.1.x it was straightforward; then for about two years nothing moved; then 4.0.0 landed and, to put it mildly, it got complicated. It felt like every version broke something else.
To be fair to the Hive maintainers: the Metastore itself didn’t really break – the tools around it did. HMS removed a method from its interface, which it is entirely entitled to do. There’s no specification saying that method must exist. Hold that thought; because it’s the point of this whole section.
| Version | Released | What we saw |
|---|---|---|
| 3.1.3 | 2022-04 | The last state where everything just worked, about as well as it ever has. But the 3.1 branch dates to 2018 and is long end-of-life – not a place to stay. |
| 4.0.0 | 2024-03 | Broke Trino’s timestamp statistics (Trino #26214). Trino fixed it upstream (in Trino 480); we reported it rather than patching it ourselves. |
| 4.0.1 | 2024-10 | Removed the get_table method (HIVE-26537). Spark’s Iceberg integration still calls it through an old Hive client library, so you hit Invalid method name: 'get_table' – and Spark + Iceberg on Hive has been unusable since, still unfixed upstream as of writing (Iceberg #12878). |
| 4.1.0 | 2025-06 | First REST catalog support – genuinely modern and welcome – but broken on startup. |
| 4.2.0 | 2025-11 | REST catalog less broken, now with OAuth2. It passes our integration tests, though a user on Discord reported possible issues re-creating schemas. |
Our fix, so customers could move off the ancient 3.x line, was to patch the removed get_table method back into our Hive images for 4.2 (docker-images #1636), so Spark + Iceberg works against our Hive 4.2.0 and 4.2.1. It works – but it isn’t pretty, and it’s exactly the kind of thing you shouldn’t have to do.
Which is the lesson. You want a stable, documented, specified interface between the engine and the catalog – one where a method can’t simply vanish in a patch release. That is the case for the Iceberg REST catalog: because it’s specified, this class of breakage shouldn’t recur. It’s why, when customers ask, we point them at the REST catalog as the direction of travel.
What do you do when upstream removes a feature you depend on?
Short answer: you maintain it yourself until upstream brings it back – and sometimes patch it again afterwards. The clearest example is NiFi’s on-again, off-again Iceberg support.
First, a quick note: NiFi can only append data to Iceberg tables. It adds rows to an existing table – no create, no delete, no query, no updates. That sounds thin, but it still beats writing Parquet with NiFi and then running a separate Trino or DuckDB job to load it into Iceberg; it saves a hop. “Every tool can do Iceberg” is different from “Every tool can do all of Iceberg”.
| Version | What happened |
|---|---|
| 1.19.0 (2023) | Iceberg support added (NIFI-10442). Comprehensive for a first pass: S3, HDFS, Hive Metastore, Parquet/ORC/Avro, Kerberos. No REST catalog yet. |
| 1.28.1 | End of the 1.x line, which we held longer as an LTS branch. |
| 2.0.0-M4 | Looked great in the milestone builds. We ran our testing on it and were looking forward to the final release. |
| 2.0.0 | Iceberg support removed (NIFI-13938) – slipped in between the last milestone and the final release. We found out when a customer couldn’t find the Iceberg processor. The motivation (cutting CVEs from old Hive/Hadoop libraries) was understandable; the timing was unlucky for us. |
| 2.0.0–2.6.0 | We took the removed code, packaged it as a bundle, published it as stackabletech/nifi-iceberg-bundle, and shipped it in our images – same code, same features – and maintained it across these releases. |
| 2.7.0 | Iceberg back upstream, rebuilt from the ground up: S3, REST catalog, Parquet. Dropped: HDFS (and with it Kerberos), ORC, and the CVEs. The rewrite also shipped a few bugs and gaps we had to patch: configurable S3 endpoint / path-style access (NIFI-15524); fields mapped by ordinal position instead of column name (NIFI-16067); ClassCastException on complex types – arrays, maps, nested records (NIFI-16069). |
| 2.12.0 | All three bugs/issues fixed. If you’re on any 2.x, update to 2.12 – then these are resolved and you can use the REST catalog. |
We tried to get our old 2.x bundle working against 2.7 for the HDFS and Hive Metastore users, and couldn’t, due to classpath issues – so the happy ending holds unless you need one of those. For everyone else on a 2.x line, updating to 2.12 clears the bugs and opens up the REST catalog.
Further reading: Apache NiFi 2 – key updates.
“Aren’t you just shipping the old CVEs, then?”
That question came up after the talk, and it’s a fair one. When we re-added NiFi’s Iceberg support from the removed code, we were carrying largely the same Hadoop and Hive libraries – CVEs included. The honest framing is that the customer gets to decide: keep the job running, or strip the functionality to clear the CVE list.
But a CVE being present is not the same as it being exploitable. For many of these, in the way the library is actually used here, there’s no path for an attacker to reach the vulnerable code – and if someone is already deep enough inside the pod to trigger it, you’ve lost at a very different door. In practice a large share of a scary-looking CVE list turns out not to be exploitable in context. Compliance teams would rather hear “we removed it,” and functionally, present-but-unreachable and removed can come to the same thing. What matters is assessing each one on whether it can actually be exploited – not just making the list shorter.
So who keeps all this running?
None of this is exotic – it’s the normal life of a multi-engine Iceberg lakehouse. Someone has to own the maintenance jobs, track which engine version broke which integration, carry patches until upstream catches up, and make a defensible call on each CVE. You can absolutely do that yourself; it’s all open source, our patches included. Or it can be a standing job someone else does.
For us, that someone-else is the Stackable Data Platform (SDP) – a modular, Kubernetes-native data platform built from established open-source components (Spark, Trino, NiFi, Iceberg, the Hive Metastore, OPA, Superset), wired together with Kubernetes operators and tested together every week. Those weekly interoperability tests exist precisely to catch the kind of cross-engine breakage in this article before it reaches a customer. When upstream removes something we need, we patch it and push the fix back upstream – and because we run CVE processes with SBOMs and VEX, “are the CVEs still in there?” gets answered by assessing exploitability, not by reshuffling the list.
Two ways to get exactly the lakehouse described here:
- Run it yourself – the data lakehouse demo mentioned above stands up NiFi → Kafka → Spark → Iceberg/Trino → Superset, with OPA, on your own Kubernetes.
- Have it run for you – the Managed Stackable Data Lakehouse bundles Trino, Iceberg, Spark, Airflow, Superset, NiFi and OPA as a managed offering – no Kubernetes or ops team required.
Iceberg really does deliver on letting many engines safely share the same tables. It just doesn’t deliver the maintenance for free. Decide who owns that before the tables get slow – ideally before the coordinator falls over.