Create Pinot Cluster
In this guide you will learn how to create a Pinot Cluster in your local machine. You should have completed the instructions in the Install Pinot guide.
This guide will teach you how to create a Pinot Cluster in your local machine.
Apache Pinot is a distributed system made of different components, each performing a unique task while in operation. In the Quick Start example, we saw all these components deployed inside a single JVM (Java Virtual Machine). But, in reality, Pinot components are deployed as separate runtimes to enable high scalability and fault tolerance.
A typical Pinot cluster consists of the following components:
- Zookeeper (not strictly a Pinot component, but Pinot depends on it)
- Pinot Controller
- Pinot Broker
- Pinot Server
When creating a cluster, these components must be configured and started separately.
Let’s start with Docker first.
With Docker Compose
With Docker, we will spin up each component as a container. Rather than doing it individually, let’s create a Docker Compose project to bundle everything together.
Before getting started, make sure that your Docker installation meets the minimum requirements. We would recommend that you allocate 8 CPUs, 16GB of RAM, 4GB of swap, and 50 GB, but you can adjust this accordingly.
Create a file called
docker-compose.yml and add the following content to it.
command: "StartController -zkAddress zookeeper:2181"
command: "StartBroker -zkAddress zookeeper:2181"
command: "StartServer -zkAddress zookeeper:2181"
Here, we are using the
apachepinot/pinot:0.12.0 Docker image.
Start the stack by typing:
That will bring up a collection of containers including Zookeeper, Kafka, Pinot Controller, Pinot Broker, and Pinot Server.
Check their status by running:
With Launcher Scripts
We can set up a Pinot cluster using the launcher scripts that come with the Pinot distribution. Here, each Pinot component will be started as a separate JVM.
First, download the Pinot distribution, unpack it, and navigate to the
bin directory, as described in the Installation Developer Guide.
Next we'll start up each of the components.
You'll need to run each of the commands below in a separate terminal window.
Start Pinot Controller
bin/pinot-admin.sh StartController \
Start Pinot Broker
bin/pinot-admin.sh StartBroker \
Start Pinot Server
bin/pinot-admin.sh StartServer \
Now that you've got Pinot running locally, it's time to see what it can do by ingesting some data.
- To learn how to ingest batch data, see Ingesting from a batch data source.
- To learn how to ingest streaming data, see Ingesting from a streaming data source.