Complete the following steps to ingest data from any supported source:
- Create a connection to the source
- Add a dataset specific configuration
- Perform data modeling
- Add indexes and perform additional configuration
- Review and submit
Create a connection to the source
This is specific to the source and you can find the information by choosing the specific connection page from the data source catalog.
Perform data modeling
Do the following as needed to model the data:
Delete a column: Click the Delete button at the end of the row of the column to delete.
Edit a column: To change Field type, Data type and specify whether the column is multi-value, find the column to update, and then click the Edit button at the end of the row. Make changes as needed, including transformations. For information about transformation functions, see supported transformations in Apache Pinot.
Add a new column: This will open a modal window similar to edit column and you can create a new column and use the transformation functions to provide the logic to fill values for that column.
Add indexes and perform additional configuration
Improve query performance by adding indexes to the appropriate columns and choose encoding types for each column.
Configure unique details such as tenants, scheduling, data retention, and a primary key for upsert.
Review and submit
Check the details and preview data. When ready, click Create Dataset.