Part 2 of 2. Part 1 covered my data2day talk and what changes when access control starts asking the data catalog. This one is about what happened after the talk.
The short version
In my data2day demo, access control reads classifications from the data catalog, and a policy decides what they mean. Simon Harrer forked it and wired access to a different catalog instead – same Stackable platform underneath, different governance on top: there, the catalog makes the decision and the policy enforces it. The question both versions raise: who decides who gets to see your data – a rule that reads the catalog, or the catalog itself? Keep reading to find out more.
Two hours on the road
After my talk at data2day in Cologne, I met Simon Harrer. Simon is CEO and co-founder of Entropy Data, and we were both heading to the same place next: the BARC Data & Analytics Retreat, so he decided to hitch a ride with me.
The drive took about two hours, including one stop to top up the oil. Somewhere along the way, Simon forked my demo, replaced OpenMetadata with Entropy Data and gave it access control that asks Entropy Data. Not a bad use of a passenger seat. Over the following days the fork grew into a complete demo of its own, with data products, data contracts and a dashboard as the consumer. And the best thing: it’s public: github.com/entropy-data/entropydata-dbt-demo.
The fork is a good excuse to pick up a question I only brushed against in the talk: once the catalog takes part in access decisions, where do the rules live?
What the fork changes
Simon started from the public version of my demo: OpenMetadata as the catalog, and OPA rules that decide every query from what the catalog knows about the table – tags, owners and so on – combined with the groups the user belongs to.
The Stackable side underneath stays the same: the Stackable Data Platform (SDP) runs, among other things, Trino, Apache Airflow, Apache Superset, Apache Hive, Apache Hadoop HDFS and the Open Policy Agent (OPA) on Kubernetes, with Keycloak as the identity provider, all deployed via GitOps. dbt builds the tables, Airflow runs dbt.
On top of that, Entropy Data takes over from OpenMetadata, in the role of a data product marketplace. (I’ll call it the catalog in this post, because that’s the role it plays in the access story.)
The diagram below shows what Entropy Data adds to the demo.
Diagram: Entropy Data
Concretely, Entropy Data brings three things:
- Every data product is described in the Open Data Product Standard (ODPS), with a data contract in the Open Data Contract Standard (ODCS) that spells out what the data is and what it means.
- Lineage comes in via OpenLineage.
- Consumers request access to a data product in the marketplace, and the owner approves.
Simon also built a new data product on top of the existing ones, a „Nation Scorecard“, contract-first. The data contract is written first, then a coding agent implements the dbt project against it, using the marketplace as its map of what already exists.
Diagram: Entropy Data
There’s a video of that if you want to watch it happen, and there is also a blog post by Simon to accompany this post, telling his side of the story in more depth. Here, though, I want to look at one addition that’s easy to miss in the diagram: how a Trino query gets allowed or denied.
Witness or assistant referee?
In both demos, Trino asks OPA before every query, and in both, OPA is the referee: the one component that allows or denies it. And in both, OPA talks to the catalog before it makes the call. What differs is the role the catalog plays in that conversation. In my demo it is a witness: it states what it knows about the table, and the rule book decides what that means. In Simon’s fork it is an assistant referee: it makes the call, and OPA enforces it – the way an assistant flags offside and the referee stops play.
In my demo (part 1), OPA asks the catalog what a table is. The Resource Info Fetcher looks the table up in OpenMetadata and comes back with facts: owners, tags, the tags on every column. The rules that turn those facts into a decision are written in Rego and live in Git: „no owner, no access“, „a column tagged PII is masked unless you’re in the PII group“.
In Simon’s fork, OPA asks the catalog whether the user may run the query. The Rego policy maps the table to a data product and its output port, then sends Entropy Data exactly one question: may this user query this output port? Entropy Data says yes or no, based on the access agreements in the marketplace. Who may use a port is decided in Entropy Data, and OPA just enforces whatever answer comes back.
Simon’s demo flow makes this easy to see: a user from the „Market Analytics“ team opens the Nation Scorecard in Superset and gets „Access Denied“. They request access for their team in Entropy Data. The output port is set up to approve requests automatically, so the agreement is active straight away, and the same query now goes through. For the access request itself, nobody touches a policy file.
In my demo, the same scene would end differently. The denial would come from a rule – the table has no owner, say, or a column carries a tag that none of the user’s groups may see. To get in, either the data product owner changes the facts in OpenMetadata, by setting an owner or adjusting a tag, or the user is added to the right group in Keycloak. Here too, nobody touches a policy file. A pull request against the Rego policy is only needed when the rule itself should change – because a new classification needs a new rule, for example.
Put simply: in the first model the catalog supplies facts and the policy decides. In the second, the catalog decides and the policy enforces. A rule like „columns tagged PII are masked unless you’re in the PII group“ lives in Rego in my demo. In Simon’s fork, that rule would have to live in Entropy Data, because OPA’s rule book has shrunk to a single entry: whatever Entropy Data says. The table below goes through what follows from that.
Two models, side by side
| Rules in the policy (part 1) | Rules in the catalog (fork) | |
|---|---|---|
| OPA asks the catalog | What is this table? (owner, tags) | May this user query this output port? (OPA maps the table to the port first) |
| Where the rules live | Rego, in Git | Access agreements in Entropy Data (the table-to-port mapping in Rego) |
| Who decides about a specific request | The policy, from the facts the data product owner maintains in OpenMetadata (owner, tags) and the user’s groups in Keycloak | The data product owner, by approving or rejecting the request in Entropy Data |
| Who changes the rules | Platform team, via pull request to the Rego policy in Git | Platform team, via pull request – but the policy only holds the table-to-port mapping and one rule: ask Entropy Data |
| How access is granted | Group membership plus metadata-based rules | A request-and-approve workflow per data product |
| Granularity in the demo | Per table and column (masks, row filters possible) | Per output port |
| Audit trail | OPA decision log, including the catalog facts used | Access agreements in the marketplace (an OPA decision log can be switched on, as in my demo) |
| If the catalog can’t be reached | A fallback rule in Rego decides. Whether to fail open or fail closed is a business decision. | Same: the Rego policy decides what to do without an answer from Entropy Data. |
A word on the granularity row: it describes the two demos, not the limits of either approach. Simon’s demo grants access per output port because that’s how data products are consumed there. Mine masks columns because that was the point of the talk.
No right or wrong
I don’t think one of these models is the right one.
Rules in the policy work well when access follows general principles that apply to lots of objects: „classified data is masked“, „nothing without an owner“. They get reviewed like code and versioned with the platform, and the same rule covers every table, including the ones nobody has looked at yet.
Rules in the catalog work well when access is a business decision about one specific data product: who uses it, for what purpose, and who signed off. The data product owner makes that call in the tool where the data product lives. In the policy model, the owner can only steer the outcome indirectly, by changing tags and owners; a new kind of decision needs a new rule, and that means a pull request to the platform team.
Most organisations will need both, and the two fit together nicely. The catalog can decide whether you get to use a data product at all, while a general rule in the policy still masks the PII columns inside it. OPA is a good place to combine them, because it remains the single point in the platform that enforces access, whichever system holds the rules.
What both have in common
Put the fork next to my version and the differences are all in the governance layer. The Stackable side didn’t change, and that was the part I was most curious about:
- Same operators, same Kubernetes resources, same GitOps deployment. Simon wired Trino to OPA the same way I did in mine.
- In the demo, swapping the catalog took a car ride. (To be fair: a demo with sample data, not ten years of metadata.) It works because both sides talk through open interfaces: OPA, OIDC, and open standards for contracts, products and lineage.
- The governance layer can be swapped without touching the platform layer – the catalog, the policies or the rules model – and you keep everything else.
For me, that matters more than which of the two models you pick, and it’s what I took away from those two hours on the road: an open platform lets you make that choice, and change your mind later.
So, who decides? You do – and, more importantly, you’re not locked into that decision. That’s the part an open platform gives you that a closed one can’t.
The demo is on GitHub – Stackable Data Platform with Entropy Data: github.com/entropy-data/entropydata-dbt-demo
Talk to us: Sönke Liebau, CPO and co-founder of Stackable; Simon Harrer, CEO and co-founder of Entropy Data.