Try StarTree Cloud: 30-day free trial
⌂
StarTree Data Manager
Confluent Cloud

Confluent Cloud connector

Confluent Cloud is a fully-managed, cloud-native data streaming platform built on Apache Kafka. Confluent Cloud lets you build real-time data pipelines and stream data between different applications. It offers a scalable and reliable infrastructure that can handle massive amounts of data streams and process them in real-time.

Use the Confluent Cloud connector to create a table in Apache Pinot by mapping it to a particular topic. As soon as the integration is set up, StarTree automatically begins to consume data from the connected topic in Confluent Cloud.

Connect StarTree to Confluent Cloud

  1. On the Data Manager overview page, click Create a dataset. Then, on the Select Connection Type page, click Confluent Cloud.

  2. Enter the following information to create a connection with Confluent Cloud:

    • Broker URL: You can find this information in Cluster Settings of your Confluent Cloud environment.
    • Username:
    • Password:

  3. (Optional) To use Confluent Cloud Schema Registry (opens in a new tab), choose Advanced Options when creating a connection and provide the following information:

    • Schema Registry URL:
    • Username:
    • Password:

  4. Click Test Connection to verify the connection, and then click Create Connection.

When you provide schema registry information, only topics associated with that schema are displayed in the topic selection screen.

  1. In the Topic Name drop-down list, select the topic name that exists in Confluent Cloud. Note, only topics associated with the schema (that you provided schema registry information for) are available.

  2. In the Data Format drop-down list, select the appropriate format. This maps the content from Confluent Cloud to the new table in Apache Pinot.

If schema registry information isn't provided, and you select AVRO as the data format, this results in an error. If this happens, go back to the previous screen to provide schema registry information.

  1. (Optional) In Record reader config, provide additional details about the source schema.

  2. (Optional) Improve query performance by adding indexes to the appropriate columns and choose encoding types for each column.

  3. (Optional) Configure unique details such as tenants, scheduling, data retention, and a primary key for upsert.

  4. Click Next.

  5. Check the details and preview data. When ready, click Create Dataset.

AWS S3Delta Lake