TimeIndexFiller
Fills missing time data points. Used to ensure data fetched with SQL is a correct timeseries for downstream nodes. The SQL language is not timeseries oriented. The GROUP BY timeColumn clause does not create a point if there is no data for a time bucket.
Inputs
A table with a time index to fill.
Outputs
The table with the time index filled. No naming constraint for outputKey and outputName.
Parameters
| name | description | default value |
|---|---|---|
component.timestamp | The name of the time index column. | timestamp |
component.monitoringGranularity | Granularity of the time index. In ISO-8601 format. Eg P1D. | |
component.minTimeInference | Inference strategy for the minimum time constraint. | FROM_DATA |
component.maxTimeInference | Inference strategy for the maximum time constraint. | FROM_DETECTION_TIME |
component.lookback | Offset to use for a time inference with lookback. In ISO-8601 format. Eg P7D. | |
component.fillNullMethod | Method to use to fill null values in the metric column. | FILL_WITH_ZEROES |
Automatic configuration with macros.
minTimeInference, maxTimeInference and lookback are not required if the __timeFilter macro was used to get the input data.
monitoringGranularity is not required if the __timeGroup macro was used to get the input data.
Available inference strategies for minTimeInference and maxTimeInference
The TimeIndexFiller needs a start and an end to fill the time index. These boundaries can be computed with different strategies.
| name | description |
|---|---|
FROM_DATA | The minimum (resp maximum) time for which to fill the index is the minimum (resp maximum) time observed in the input data. |
FROM_DETECTION_TIME | The minimum (resp maximum) time corresponds to the minimum (resp maximum) time used by the detection pipeline run. |
FROM_DETECTION_TIME_WITH_LOOKBACK | Like above, use the detection pipeline boundaries, but apply an offset. Eg: minTime - lookback. |
Available methods for fillNullMethod
When a missing time point is added, the metric value is unknown. This value can be filled with one the following methods.
| name | description |
|---|---|
FILL_WITH_ZEROES | Replace null values with zeroes. |
KEEP_NULL | Keep the null values. |
INTERPOLATE | Available soon. Interpolate a value based on the preceding and next available values. |
| Note that most of the detectors are not compatible with null values for the moment. |
Examples
Without macros
{
"name": "baselineMissingDataManager",
"type": "TimeIndexFiller",
"params": {
"component.monitoringGranularity": "P1D",
"component.timestamp": "ts",
"component.minTimeInference": "FROM_DETECTION_TIME_WITH_LOOKBACK",
"component.maxTimeInference": "FROM_DETECTION_TIME_WITH_LOOKBACK",
"component.lookback": "P7D"
},
"inputs": [
{
"sourcePlanNode": "baselineDataFetcher",
"sourceProperty": "baselineOutput"
}
],
"outputs": [
{
"outputKey": "filler",
"outputName": "baselineOutput"
}
]
}With macros
If __timeFilter and __timeGroup macros are used in the upstream DataFetcher node, and if the metric is named metric:
{
"name": "missingDataManager",
"type": "TimeIndexFiller",
"params": {
"component.timestamp": "ts"
},
"inputs": [
{
"sourcePlanNode": "baselineDataFetcher",
"sourceProperty": "baselineOutput"
}
],
"outputs": [
{
"outputKey": "filler",
"outputName": "baselineOutput"
}
]
}