Try StarTree Cloud: 30-day free trial

Pinot

This datasource plugin is used to connect to Pinot using the pinot-java-client (opens in a new tab). It provides multiple ways to connect to Pinot instance such as

  • using zookeeper and controller
  • using controller and broker

Zookeeper and Controller

This option is preferable when Pinot components are accessible by Thirdeye using the endpoints exposed by <pinot_controller_url>/instances/{instanceName}.

Configuration

{
  "name": "pinotQuickStartLocal",
  "type": "pinot",
  "properties": {
    "zookeeperUrl": "localhost:2123",
    "clusterName": "QuickStartCluster",
    "controllerConnectionScheme": "http",
    "controllerHost": "localhost",
    "controllerPort": 9000
  }
}

The benefit of using this configuration is that we don't need to worry about the brokers as the list of brokers to query is decided with the help of Zookeeper.
This configuration may not work when Pinot is sitting inside a remote cluster and Zookeeper is not exposed or, even if the Zookeeper is exposed, the broker URLs that it provides will mostly be internal and inaccessible from outside.

Broker and Controller

If you are facing the problem mentioned with above configuration then this configuration may solve your problem. This requires the Pinot brokers to be accessible by Thirdeye.

Configuration

{
  "name": "pinotds",
  "type": "pinot",
  "properties": {
    "zookeeperUrl": "localhost:2123",
    "clusterName": "pinot-quickstart",
    "controllerConnectionScheme": "http",
    "controllerHost": "localhost",
    "controllerPort": 9000,
    "brokerUrl": "localhost:8099"
  }
}

Here even though we provide zookeeperUrl it is just a placeholder and has no impact once we provide brokerUrl.
One drawback of this configuration is that we are providing a static broker url hence all queries will hit one broker unless all broker replicas are sitting behind a load balancer.

Pinot with Authorization (custom headers)

We can configure the datasource to pass custom headers along with the requests that are made to Pinot. The obvious usecase for this is when authorization is enabled on Pinot, and we are required to pass the Authorization header along with the requests.

{
  "name": "pinotQuickStartLocal",
  "type": "pinot",
  "properties": {
    "zookeeperUrl": "localhost:2123",
    "clusterName": "QuickStartCluster",
    "controllerConnectionScheme": "http",
    "controllerHost": "localhost",
    "controllerPort": 9000,
    "headers": {
      "Authorization": "Bearer <Token>"
    }
  }
}

Any header can be passed as a key value pair to the headers.