Stackable

Demos

Demos

Stackable Demos

Possible uses of the Stackable Data Platform

The following application examples demonstrate how easily and flexibly the Stackable Data Platform can be used for very different use cases. All demos deliberately focus on one topic at a time and:

  • use stackablectl to install 
  • can be set up 1:1 on your system
  • demonstrate contexts step by step 
  • are available as open source code in Github

Demo: NIFI-kafka-druid-water-level-data

Real-time display of water levels

Low water or danger of flooding – the water levels of our rivers have moved into the public interest in times of climate change.

Our Stackable Data Platform demo shows the water levels of rivers in near real-time for Germany based on data from Pegel Online.

Several components of the Stackable Data Platform play together without requiring much configuration effort:

Apache Nifi and Kafka are used to fetch water level measurements from gauging stations distributed across Germany via an API from Pegel Online and store them in Apache Druid.

Druid is a scalable real-time database that can be queried using SQL. This method is used in the demo to query gauge levels via Apache Superset and visualize them in the dashboard. For permanent storage, Druid requires a so-called “deep storage”, which is implemented in our example via MinIO as an S3-compatible object store, as it is available in most public and private cloud environments.

 

within K8s-Cluster:
stackablectl demo install nifi-kafka-druid-water-level-data

Demo: trino-taxi-data

Analysis with a data lake

This Stackable Data Platform demo shows data stored in S3 for analysis up to display in the dashboard. Our Stackable operators are used to configure and roll out various components. In particular, this example shows how role-based data access can be implemented using the Open Policy Agent:

  • MinIO, an S3-compatible object store, persistently stores the data for this demo.
  • Hive-Metastore stores the metadata necessary to make the sample data accessible via SQL and is used by Trino in our example.
  • Trino is our extremely fast, distributed SQL query engine for Big Data analytics that can be used to explore data spaces and that we use in the demo to provide SQL access to the data.
  • Finally, Apache Superset we use to retrieve data from Trino via SQL queries and build dashboards on that data.
  • Open Policy Agent (OPA): an open source, universal policy engine that unifies policy enforcement across the stack. In this demo, OPA authorizes which user can query which data.

 

within K8s-Cluster:
stackablectl demo install trino-taxi-data

Demo: NIFI-kafka-druid-earthquake-data

Event streaming of earthquake data

This Stackable Data Platform demo shows streamed earthquake data end-to-end up to the dashboard. It includes the following operators:

  • Superset: a modern platform for data exploration and visualization. This demo uses Superset to retrieve data from Druid via SQL queries and build dashboards on that data.
  • Kafka: A distributed event streaming platform for high-performance data pipelines, streaming analytics, and data integration. In this demo, Kafka is used as an event streaming platform to stream data in near real-time.
  • Nifi: An easy-to-use, powerful system to process and distribute data. This demos uses it to fetch earthquake-data from the internet and ingest it into Kafka.

  • Druid: A real-time database to support modern analytics applications. This demo uses Druid to ingest and store data in near real-time from Kafka and provide access to the data via SQL.
  • MinIO: An S3-compatible object store. In this demo, it is used as persistent storage for Druid to store all streamed data.

 

within K8s-Cluster:
stackablectl demo install nifi-kafka-druid-earthquake-data

Newsletter

Subscribe to the newsletter

With the Stackable newsletter you’ll always be up to date when it comes to updates around Stackable!

Copy link
Powered by Social Snap