Skip to main content

startree-threshold-percentile

Description

Detect an anomaly if the metric is above a maximum threshold or below a minimum threshold. Aggregation function with 2 operands: PERCENTILETDIGEST, DISTINCTCOUNTHLL,etc...

Flowchart

graph TD eventProcessor>eventProcessor<br>EVENTS]--->root guardrailThresholdProcessor>guardrailThresholdProcessor<br>THRESHOLD]--->eventProcessor metricThresholdProcessor>metricThresholdProcessor<br>TIME_OF_WEEK]--->guardrailThresholdProcessor coldStartProcessor>coldStartProcessor<br>COLD_START]--->metricThresholdProcessor anomalyDetector[[anomalyDetector<br>THRESHOLD]]--->coldStartProcessor missingDataPointsFiller[/missingDataPointsFiller<br>TimeIndexFiller\]--->anomalyDetector currentDataFetcher[(currentDataFetcher<br>DataFetcher)]--->missingDataPointsFiller root>root<br>ANOMALY_MERGER] eventsDataFetcher[(eventsDataFetcher<br>EventFetcher)]--->eventProcessor root ---> END((END))

Parameters

DATA

namedescriptiondefault value
aggregationColumnThe column to aggregate. Can be a derived metric.-
aggregationFunctionThe aggregation function to apply on the aggregationColumn. Example: AVG.-
dataSourceThe Pinot datasource to use.-
datasetThe dataset to query.-
monitoringGranularityThe period of aggregation of the timeseries. In ISO-8601 format. Example: PT1H.-
timezoneTimezone used to group by time. In TZ-identifier format.
For instance, UTC or US/Pacific.
UTC
timeColumnTimeColumn used to group by time. If set to AUTO (the default value), the Pinot primary time column is used.AUTO
timeColumnFormatRequired if timeColumn is not AUTO. Learn more.
completenessDelayThe time for your data to be considered complete and ready for anomaly detection. In ISO-8601 format. Example: PT2H. Learn more.P0D
queryFiltersFilters to apply when fetching data. Prefix with AND. Example: AND country='US'
queryLimitMaximum number of timeseries point to fetch.100000000
aggregationParameterThe second argument of the aggregationFunction. Example: for PERCENTILETDIGEST: 95.-

DETECTION

namedescriptiondefault value
maxIf the current value is above this threshold, detect an anomaly.-
minIf the current value is below this threshold, detect an anomaly.-

FILTER

Time of week

namedescriptiondefault value
daysOfWeekUsed to ignore anomalies that happen at specific time periods. A list of days. Anomalies happening on these days are ignored if timeOfWeekIgnore is true. Example: ["MONDAY", "SUNDAY"].[]
hoursOfDayUsed to ignore anomalies that happen at specific time periods. A list of hours. Anomalies happening on these hours are ignored. Example: [0,1,2,23][]
dayHoursOfWeekUsed to ignore anomalies that happen at specific time periods. A mapping of {DAY: [hours]}. Anomalies happening on these timeframes are ignored if timeOfWeekIgnore is true. Example: {"FRIDAY": [22, 23], "SATURDAY": [0, 1, 2]}{}

Guardrail metric

namedescriptiondefault value
guardrailMetricMinUsed to ignore anomalies that don't meet the guardrail threshold. Minimum threshold of the guardrail metric. If -1, no minimum threshold is applied.-1
guardrailMetricMaxUsed to ignore anomalies that don't meet the guardrail threshold. Maximum threshold of guardrailMetric. If -1, no maximum threshold is applied.-1
guardrailMetricUsed to ignore anomalies that don't meet the guardrail threshold. Metric to use as a threshold guardrail. Example: COUNT(*) and set guardrailMetricMin = 100 to ignore anomalies detected when there is less than 100 observations in the period.COUNT(*)

Special events

namedescriptiondefault value
eventFilterSqlFilterUsed to ignore anomalies that happen during events. Sql filter to apply on the events. Learn more
eventFilterLookaroundUsed to ignore anomalies that happen during events. Offset to apply on startTime and endTime to look around the timeframe. In ISO-8601 format. Example: P1D.P2D
eventFilterTypesUsed to ignore anomalies that happen during events. List of event types to fetch by. Example: ["HOLIDAY", "DEPLOYMENT"]. [] fetches all events. Use ["__NO_EVENTS"] to disable.['__NO_EVENTS']
eventFilterBeforeEventMarginUsed to ignore anomalies that happen during events. A period in ISO-8601 format that corresponds to a period that is also impacted by the event. Example: if beforeEventMargin is P1D, if event happens on [Dec 24 0:00, Dec 25 0:00[, the label will be applied to anomalies happening on [Dec 23 0:00 and Dec 25 0:00[P0D
eventFilterAfterEventMarginUsed to ignore anomalies that happen during events. Same as eventFilterBeforeEventMargin at the end of the event.P0D

POSTPROCESS

Data mutability

namedescriptiondefault value
mutabilityPeriodUse if your data is mutable. ThirdEye will maintain the detection results up to date on the mutable period. For instance, if your last 10 days of data is mutable, set P10D. At each cron detection job, the detection results for the last 10 days will be updated.P0D
reNotifyPercentageThresholdFor detection replay when data is mutable. If the percentage difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with reNotifyAbsoluteThreshold. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies.-1
reNotifyAbsoluteThresholdFor detection replay when data is mutable. If the absolute difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with reNotifyPercentageThreshold. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies.-1

Anomaly merger

namedescriptiondefault value
mergeMaxGapMaximum duration of an anomaly merger. At merge time, if an anomaly merger would get bigger than this limit, the anomalies are not merged. In ISO-8601 format. Example: P7D.
mergeMaxDurationMaximum gap between 2 anomalies for anomalies to be merged. In ISO-8601 format. Example: PT2H. To disable anomalies merging, set this value to P0D.

RCA

namedescriptiondefault value
rcaAggregationFunctionThe aggregation function to use for RCA. If the detection metric name is known to ThirdEye, this parameter is optional.
rcaIncludedDimensionsList of the dimensions (columns in the dataset) to use in RCA drill-downs. If not set or empty, all dimensions of the table are used. Learn more.[]
rcaExcludedDimensionsList of dimensions (columns in the dataset) to ignore in RCA drill-downs. If not set or empty, all dimensions of the table are used. rcaExcludedDimensions and rcaIncludedDimensions cannot be used at the same time.[]
rcaEventTypesA list of type to filter on for RCA. Only events that match such types will be shown in the RCA related events tab. Learn more.[]
rcaEventSqlFilterA Sql filter for RCA events. Only events that match the filter will be shown in the RCA related events tab. Learn more.