Scaling Workers

ThirdEye is comprised of three components

1. Coordinator
2. Scheduler
3. Worker

Out of these three the Worker component is responsible for doing the heavy lifting of running different kinds of tasks scheduled by the Scheduler. As we introduce more and more alerts, the task count grows proportionally. ThirdEye provides the flexibility to scale the workers horizontally as well as vertically to support the growing tasks.

Vertical Scaling

ThirdEye Worker has an internal thread pool which it uses to run the tasks in parallel. We can control the parallelism with the help of maxParallelTasks configuration.

When to do this?

This should be configured based on the frequent load expected on the worker, for example if there are 10 alerts which have detection scheduled for every minute then it makes sense to have 10 tasks run in parallel.

Configuration

For helm installation

...
worker:
  config:
    maxParallelTasks: 10    # default is 5

values.yaml

For bare metal installation

...
taskDriver:
  maxParallelTasks: 10

server.yaml

Horizontal Scaling

ThirdEye Workers can be scaled by replicating them.

When to do this?

This is a more flexible scaling as it allows us to scale up/down the number of workers based on sudden temporary changes in task rates, e.g. if there are 5 alerts that runs detection after every minute, but we also have another 10 alerts which just trigger once a day at 2 am, in this case it makes sense to spin up another worker before 2 am and take it down once the tasks are served. (HPA is not yet supported in official ThirdEye helm charts as the trigger to scale can vary based on the usecase)

Configuration For helm installation, provide the number of replicas needed.

...
worker:
  replicas: 2
  ...
  config:
    randomWorkerIdEnabled: true

values.yaml

For bare metal installation, run multiple workers on different ports.

...
taskDriver:
  enabled: true
  randomWorkerIdEnabled: true

server.yaml

FAQs