Try StarTree Cloud: 30-day free trial
Segment Retention

Segment Retention

Segments in Pinot tables have a retention time, after which they will be deleted. Typically, offline tables will retain segments for a longer period of time than real-time tables.

The removal of segments is done by the retention manager. By default, the retention manager runs once every 6 hours.

The retention manager purges two types of segments:

  • Expired segments - Segments whose end time has exceeded the retention period.
  • Replaced segments - Segments that have been replaced as part of the merge roll-up process.

There are a couple of scenarios where segments in offline tables won't be purged:

  • If the segment doesn't have an end time. This would happen if the segment doesn't contain a time column.
  • If the segment's table has a segmentIngestionType of REFRESH.

In addition, segments will not be purged in real-time or offline tables if the retention period isn't specified.

The retention manager initially moves these segments into a Deleted Segments area, from where they will eventually be permanently removed.