Skip to main content

Ingest data

Complete the following steps to ingest data from any supported source:

  1. Create a connection to the source
  2. Add a dataset specific configuration
  3. Perform data modeling
  4. Add indexes and perform additional configuration
  5. Review and submit

Create a connection to the source​

This is specific to the source and you can find the information by choosing the specific connection page from the data source catalog.

Click Next.

Perform data modeling​

Do the following as needed to model the data:

  • Delete a column: Click the Delete button at the end of the row of the column to delete.

  • Edit a column: To change Field type, Data type and specify whether the column is multi-value, find the column to update, and then click the Edit button at the end of the row. Make changes as needed, including transformations. For information about transformation functions, see supported transformations in Apache Pinot.

  • Add a new column: This will open a modal window similar to edit column and you can create a new column and use the transformation functions to provide the logic to fill values for that column.

Click Next.

Add indexes and perform additional configuration​

Improve query performance by adding indexes to the appropriate columns and choose encoding types for each column.

Configure unique details such as tenants, scheduling, data retention, and a primary key for upsert.

Click Next.

Review and submit​

Check the details and preview data. When ready, click Create Dataset.