Job queue-wait and duration histograms¶
Companion to the connection-budget sizing formula
(2854 measurement). The formula
takes MAX_CONCURRENT_JOBS and the pod counts as inputs; this is what
tells whether they are right.
What shipped¶
app/tasks/_job_timings.py:job.queue_wait_seconds(created_at → started_at, labeljob_type) recorded right after the runner's post-claim re-fetch, andjob.duration_seconds(claim to FINISHED, labelsjob_type,result) recorded next tofinish_job. Explicit buckets 1 s to 1 h; the OTel defaults stop at 10 s and a csv_ingest regularly runs minutes.- Same export path as the DB counters, nothing to configure in the collector.
- Grafana "specific graphs", one panel per environment: p95 by job type for both series (openshift-app-config, next to the DB failure counters).
Reading rule¶
Queue wait rising while worker CPU is saturated: add a worker or CPU.
Rising while CPU is idle: raise MAX_CONCURRENT_JOBS. Flat: the pod
count is right, only overflow is worth touching.
Tests¶
tests/unit/tasks/test_job_timings.py: value and labels of both
instruments, naive UTC stamps, missing stamps record nothing, bucket
boundaries reach minutes.