Ingest CDC data using PostgreSQL, Kafka, and Debezium

This recipe downloads Postgres change data capture (CDC) about dvdrentals and connects Kafka to Debezium. CDC data is written to Kafka, and then consumed by Apache Pinot.

To understand how this recipe processes data, examine the commands executed in the Makefile.

Pinot Version	1.0.0
Code	startreedata/pinot-recipes/debezium-cdc

Prerequisites

To follow the code examples in this guide, you must install Docker (opens in a new tab) locally and download recipes.

Navigate to recipe

If you haven't already, download recipes.
In the terminal, navigate to this recipe's directory:

cd pinot-recipes/recipes/debezium-cdc

Spin up a Pinot cluster using the Makefile, which uses Docker compose:

make recipe

Navigate to localhost:9000/#/query (opens in a new tab) to see the data in Apache Pinot.

Clean up

make clean

Troubleshooting

To clean up old Docker installations that may be interfering with your testing of this recipe, run the following command:

docker system prune

Importing CSV files with columns containing spaces Ingesting DynamoDB data into Pinot using Kinesis and DynamoDB decoder