Segment Retention
Segments in Pinot tables have a retention time, after which they will be deleted. Typically, offline tables will retain segments for a longer period of time than real-time tables.
The removal of segments is done by the retention manager. By default, the retention manager runs once every 6 hours.
The retention manager purges two types of segments:
- Expired segments - Segments whose end time has exceeded the retention period.
- Replaced segments - Segments that have been replaced as part of the merge roll-up process.
There are a couple of scenarios where segments in offline tables won't be purged:
- If the segment doesn't have an end time. This would happen if the segment doesn't contain a time column.
- If the segment's table has a
segmentIngestionTypeofREFRESH.
In addition, segments will not be purged in real-time or offline tables if the retention period isn't specified.
The retention manager initially moves these segments into a Deleted Segments area, from where they will eventually be permanently removed.