Scheduled Jobs and Worker Queues
Background workers execute heavy or recurring tasks such as nightly risk, calibration, and data ingestion.
Explanation
Scheduled jobs trigger recurring tasks (e.g., overnight risk runs, curve rebuilding, model calibration) at defined times.
Worker queues decouple job submission from execution: tasks are enqueued and processed asynchronously by worker processes.
Each job carries metadata such as run_id and configuration so results and logs can be traced and reproduced.
workersscheduled jobsqueuesbatch
Interactive visualisation
Scheduled jobs and worker queues
Separate submission from execution.
Workers
2
Running / capacity
0 / 2
Enqueue
Tick simulates a scheduler cycle: finish one job, start queued jobs up to worker capacity.
Data ingestionQUEUED
run_id: run_a1
Model calibrationQUEUED
run_id: run_a1
Nightly riskQUEUED
run_id: run_a1
Interpretation
- Queues prevent heavy work from blocking API requests.
- Workers scale execution independently of submission.
- run_id is the bridge between jobs, logs, artefacts, and reproducibility.