Here is the high level system architecture of ThirdEye.
Let's review the boxes one by one.
This is the service that ... serves the web interface. It communicates with the backend.
The Coordinator is the entry point of the REST API.
The request are (optionally) authenticated and authorized.
The coordinator can perform CRUD operations on ThirdEye entities, simulate detection pipeline, and run interactive RCA.
The Scheduler runs the task crons and create the task objects. The tasks are persisted in the database and polled by the workers. The 2 main type of tasks are detection tasks and notification tasks.
Workers run the tasks. Here is the execution principle:
- workers wait for tasks to be added to the queue
- once a worker finds a new task, it assigns it to itself (gets a lock)
- on successfully getting a lock, the worker executes the task
- upon completion, the worker captures the results and persists them into the db
UI, coordinator, and worker can be scaled independently based on their respective load.
The scheduler is not replicated.