ThirdEye infrastructure requirements

ThirdEye is available as a managed platform with StarTree Cloud (opens in a new tab).
You can also install ThirdEye from a binary file, from sources or using Docker. See Install ThirdEye for installation procedures.

ThirdEye components

ThirdEye has five different types of pods, which are the components that make up a ThirdEye deployment.

• Coordinator: This is the API Server. It also powers root cause analysis (RCA) pipelines and ad hoc charting.
• Scheduler: Runs all the cron jobs and creates tasks
• GUI: Serves the React frontend for ThirdEye
• Worker: Executes the tasks
• MySQL: Provides state/metadata persistence for ThirdEye and serves the task queue

Consider the following details in your capacity planning:

• Coordinator and Worker pods do most of the heavy lifting
• ThirdEye uses MySQL for task processing as well as for processing and storing anomalies, which can require significant DB resources
• Memory Consumption
  • The task that demands the most memory is the detection pipeline as it will fetch large chunks of data to process in-memory
  • The highest memory consumption will occur in cases where:
    • The granularity of alerts is low. For example, a process with a granularity of one minute will process significantly more data points than one with a granularity of one hour.
    • A coordinator previews an alert on a large timeframe
    • At the moment when an alert is created, because in the worker the alert is processed across all of the historical data.
• CPU Requirements
  • ThirdEye benefits greatly from parallel processing across multiple processors
  • You can configure the number of threads and workers available
  • More CPU greatly helps in executing multiple concurrent tasks and responding to API calls
• The ThirdEye backend can be run entirely in a single virtual machine. This means that the coordinator, scheduler, and worker can run in a single JVM process

Example deployments

Here we provide examples of what might be used for different deployment needs, such as production or development.

Minimal deployment for development

We run the backend services: Coordinator, Scheduler and Worker in 1 single JVM.

Production examples

This section includes two examples: the standard setup and the enterprise setup. Start with the standard setup.

Standard setup

The standard production setup involves running everything in separate pods, while keeping the coordinator, scheduled, and UI in the same node.

Enterprise setup

This is identical to the standard setup, except that it has more workers. Adding workers enables running more tasks in parallel. We may not need to scale up the coordinator since UI requests may remain limited, but in this example we show how you might choose to, if needed.

External resources

• AWS Instance Types (opens in a new tab)
• GCP
  • GCP Instance Types (opens in a new tab)
  • E2 Machine (opens in a new tab) vs C3 Machine (opens in a new tab) types. (Note: E2 Machines are less expensive.)