Skip to main content

Create Pinot Cluster

info

In this guide you will learn how to create a Pinot Cluster in your local machine. You should have completed the instructions in the Install Pinot guide.

This guide will teach you how to create a Pinot Cluster in your local machine.

Apache Pinot is a distributed system made of different components, each performing a unique task while in operation. In the Quick Start example, we saw all these components deployed inside a single JVM (Java Virtual Machine). But, in reality, Pinot components are deployed as separate runtimes to enable high scalability and fault tolerance.

A typical Pinot cluster consists of the following components:

  • Zookeeper (not strictly a Pinot component, but Pinot depends on it)
  • Pinot Controller
  • Pinot Broker
  • Pinot Server

When creating a cluster, these components must be configured and started separately.

Let’s start with Docker first.

With Docker Compose

With Docker, we will spin up each component as a container. Rather than doing it individually, let’s create a Docker Compose project to bundle everything together.

tip

Before getting started, make sure that your Docker installation meets the minimum requirements. We would recommend that you allocate 8 CPUs, 16GB of RAM, 4GB of swap, and 50 GB, but you can adjust this accordingly.

Create a file called docker-compose.yml and add the following content to it.

docker-compose.yml
version: '3.7'
services:
zookeeper:
image: zookeeper:3.5.6
hostname: zookeeper
container_name: zookeeper
ports:
- "2181:2181"
environment:
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_TICK_TIME: 2000
pinot-controller:
image: apachepinot/pinot:0.9.3
command: "StartController -zkAddress zookeeper:2181"
container_name: "pinot-controller"
restart: unless-stopped
ports:
- "9000:9000"
depends_on:
- zookeeper
pinot-broker:
image: apachepinot/pinot:0.9.3
command: "StartBroker -zkAddress zookeeper:2181"
restart: unless-stopped
container_name: "pinot-broker"
ports:
- "8099:8099"
depends_on:
- pinot-controller
pinot-server:
image: apachepinot/pinot:0.9.3
command: "StartServer -zkAddress zookeeper:2181"
restart: unless-stopped
container_name: "pinot-server"
depends_on:
- pinot-broker

Here, we are using the apachepinot/pinot:0.9.3 Docker image.

Start the stack by typing:

docker-compose --project-name pinot-demo up
info

Passing in the parameter --project-name pinot-demo will run all the containers in a local network named pinot-demo_default. Having a network with a specific name will make it easier to interact with Pinot later in the tutorial.

That will bring up a collection of containers including Zookeeper, Kafka, Pinot Controller, Pinot Broker, and Pinot Server.

Check their status by running:

docker-compose ps

With Launcher Scripts

We can set up a Pinot cluster using the launcher scripts that come with the Pinot distribution. Here, each Pinot component will be started as a separate JVM.

First, download the Pinot distribution, unpack it, and navigate to the bin directory, as described in the Installation Developer Guide. Next we'll start up each of the components.

tip

You'll need to run each of the commands below in a separate terminal window.

Start Zookeeper

bin/pinot-admin.sh StartZookeeper

Start Pinot Controller

bin/pinot-admin.sh StartController \
-zkAddress localhost:2181

Start Pinot Broker

bin/pinot-admin.sh StartBroker \
-zkAddress localhost:2181

Start Pinot Server

bin/pinot-admin.sh StartServer \
-zkAddress localhost:2181

Next Steps

Now that you've got Pinot running locally, it's time to see what it can do by ingesting some data.