StarTree Cloud Coding Competition With Grand Prizes

TimeIndexFiller

Fills missing time data points. Used to ensure data fetched with SQL is a correct timeseries for downstream nodes. The SQL language is not timeseries oriented. The GROUP BY timeColumn clause does not create a point if there is no data for a time bucket.

Inputs

A table with a time index to fill.

Outputs

The table with the time index filled. No naming constraint for outputKey and outputName.

Parameters

name	description	default value
`component.timestamp`	The name of the time index column.	`timestamp`
`component.monitoringGranularity`	Granularity of the time index. In ISO-8601 format. Eg `P1D`.
`component.minTimeInference`	Inference strategy for the minimum time constraint.	`FROM_DATA`
`component.maxTimeInference`	Inference strategy for the maximum time constraint.	`FROM_DETECTION_TIME`
`component.lookback`	Offset to use for a time inference with lookback. In ISO-8601 format. Eg `P7D`.
`component.fillNullMethod`	Method to use to fill null values in the metric column.	`FILL_WITH_ZEROES`

💡

Automatic configuration with macros. minTimeInference, maxTimeInference and lookback are not required if the __timeFilter macro was used to get the input data.
monitoringGranularity is not required if the __timeGroup macro was used to get the input data.

Available inference strategies for `minTimeInference` and `maxTimeInference`

The TimeIndexFiller needs a start and an end to fill the time index. These boundaries can be computed with different strategies.

name	description
`FROM_DATA`	The minimum (resp maximum) time for which to fill the index is the minimum (resp maximum) time observed in the input data.
`FROM_DETECTION_TIME`	The minimum (resp maximum) time corresponds to the minimum (resp maximum) time used by the detection pipeline run.
`FROM_DETECTION_TIME_WITH_LOOKBACK`	Like above, use the detection pipeline boundaries, but apply an offset. Eg: minTime - lookback.

Available methods for `fillNullMethod`

When a missing time point is added, the metric value is unknown. This value can be filled with one the following methods.

name	description
`FILL_WITH_ZEROES`	Replace null values with zeroes.
`KEEP_NULL`	Keep the null values.
`INTERPOLATE`	Available soon. Interpolate a value based on the preceding and next available values.
Note that most of the detectors are not compatible with null values for the moment.

Examples

Without macros

    {
      "name": "baselineMissingDataManager",
      "type": "TimeIndexFiller",
      "params": {
        "component.monitoringGranularity": "P1D",
        "component.timestamp": "ts",
        "component.minTimeInference": "FROM_DETECTION_TIME_WITH_LOOKBACK",
        "component.maxTimeInference": "FROM_DETECTION_TIME_WITH_LOOKBACK",
        "component.lookback": "P7D"
      },
      "inputs": [
        {
          "sourcePlanNode": "baselineDataFetcher",
          "sourceProperty": "baselineOutput"
        }
      ],
      "outputs": [
        {
          "outputKey": "filler",
          "outputName": "baselineOutput"
        }
      ]
    }

With macros

If __timeFilter and __timeGroup macros are used in the upstream DataFetcher node, and if the metric is named metric:

  {
    "name": "missingDataManager",
    "type": "TimeIndexFiller",
    "params": {
        "component.timestamp": "ts"
      },
    "inputs": [
      {
        "sourcePlanNode": "baselineDataFetcher",
        "sourceProperty": "baselineOutput"
      }
    ],
    "outputs": [
      {
        "outputKey": "filler",
        "outputName": "baselineOutput"
      }
    ]
  }

SqlExecution How-To Guides