Worker hands running jobs back on SIGTERM (#2696)¶
Symptom¶
Every worker rollout (several a day on dev) stops job processing for up to
STALE_JOB_TIMEOUT_MINUTES (5 min). The old pod gets SIGTERM, uvicorn
runs the lifespan teardown, the loops are cancelled and the pool is
disposed, but the run_job tasks spawned through fire_and_forget were
never cancelled: they die with the process, and the row keeps
state = RUNNING, locked_by = <dead pod> until
sweep_stuck_running_jobs on the new pod finds it stale.
Shipped¶
DataIngestionRepository.release_job(job_id, pod_id): the CAS guard offinish_job(locked_by = pod AND state = RUNNING), the target of the stale sweep's recoverable bucket (NOT_STARTED, unlocked,attemptspreserved soclaim_job's retry cap still counts the run).run_jobcatchesCancelledErroraround the handler: cancels the handler task, rolls back both sessions, releases the row, logs "handed back to the queue" or "no longer ours", re-raises. Test:tests/unit/tasks/test_runner_handback.py.cancel_background_tasks()in_background.py: cancels every in-flight task and waits up toJOB_HANDBACK_SECONDS(20 s) for the hand-backs; from then on a cancelled task logs at INFO, since at shutdown that is the expected outcome. Test intest_background.py.- Lifespan order: safety poller cancelled first (nothing dispatches
anymore), then the drain, then the other loops, then
engine.dispose(). The drain has to run while the heartbeat still owns the lock and the pool is open. Test intest_lifespan_shutdown.py. - Helm:
worker.terminationGracePeriodSeconds: 60(was the Kubernetes default 30), rendered on the worker Deployment. 20 s of hand-back plusdispose()fits with a stalled DB; nopreStop, since nothing routes HTTP to the worker.
The same path covers the backend pods with DISPATCH_JOBS_INLINE on.
Replicas per environment¶
Multi-pod job processing already exists: claim_job is an atomic
compare-and-set on locked_by, the heartbeat detects preemption, the
sweep recovers dead owners. Going to two workers is worker.replicaCount
in the environment overlay, and the Deployment strategy switches from
surge to rolling on its own.
What gates it is the connection budget:
a worker costs DB_POOL_SIZE at rest and 3 × MAX_CONCURRENT_JOBS under
load. On dev (bouncer pool 35) a second worker at 2+7 does not fit. On
prod (90 usable) two workers at 5+15 give steady 3 × 15 + 2 × 20 = 85,
which fits without a rollout surge and not with one, so prod goes to two
workers only after DBaaS raises the bouncer pool or max_connections, or
with the worker at 5+10.
Automatic scaling, if it is ever needed¶
Horizontal. The signal is the backlog, not CPU: a worker waiting on
the bouncer or on S3 is idle by every CPU metric while jobs pile up.
count(*) FROM data_ingestion_jobs WHERE state = 'NOT_STARTED' AND
job_type IS NOT NULL is one gauge on the existing 30 s heartbeat; KEDA's
Prometheus scaler (or an HPA on a custom metric through the Prometheus
adapter) scales on it with minReplicas 1 and maxReplicas set by the
budget: (budget − backends × backend ceiling) / worker ceiling. Scale-down
is safe because the hand-back above makes a terminating worker return
its job within 20 s. A cooldown of a few minutes avoids flapping on a
single CSV upload, which creates a burst of NOT_STARTED rows that one
worker drains in seconds.
Vertical. The worker is I/O-bound (DB, S3); its CPU request of 2 is generous already. Memory is the only axis: a 144k-row CSV ingest holds the parsed rows in memory, and the 1 Gi limit is what an OOMKill would hit first. A VPA in recommendation mode on memory would tell whether the limit is right; automatic vertical scaling restarts the pod to apply, which is a rollout, so it depends on this issue's hand-back as well.
Neither is worth building before the DB budget grows: with 35 slots on
dev a second worker cannot even start, and on prod the ceiling rule
leaves room for exactly one more. The manual replicaCount per
environment, with the budget arithmetic in the overlay comment, is the
whole scaling story until then.
Rejected¶
- Waiting for the running job to finish before exiting: job 59 ran 70 min; no grace period covers that, and a rollout stuck behind one job is worse than a re-dispatch.
- Longer
STALE_JOB_TIMEOUT_MINUTESor a shorter one: the sweep is crash recovery; the graceful path should not depend on it at all. - HPA on CPU: scales on the wrong signal, see above.
Verification¶
- Unit:
test_runner_handback.py,test_background.py,test_lifespan_shutdown.py, runner suites green. helm templaterendersterminationGracePeriodSeconds: 60on the worker Deployment;helm lintclean.- Next dev rollout: the old worker logs "handed back to the queue" and
the new one logs "Poller: dispatching job N" within
POLLER_INTERVAL_SECONDS, no 5 min gap indata_ingestion_jobs.
Update 2026-09-23 — worker HPA in the chart¶
The hand-back made scale-in safe; the chart now has the matching
scale-out. helm/templates/worker-hpa.yaml mirrors the backend HPA
(worker.autoscaling: off by default, 1..4 replicas, 60 % CPU target).
CPU is the proxy for "job slots busy": a running job burns ~0.8 cores
against a 150m request, so any active job pushes the pod far past the
target and an idle pod drops back after the stabilization window. Each
pod adds MAX_CONCURRENT_JOBS slots and at most MAX_CONCURRENT_JOBS + 2
in-flight transactions at the bouncer (#2854). Dev turns it on in
openshift-app-config with MAX_CONCURRENT_JOBS 4. The honest signal,
queue depth, needs a jobs.queued gauge and a metrics adapter on the
cluster; CPU until then (#2689 follow-ups).
Update 2026-09-23 evening — HPA vs Argo CD replicas fight¶
Turning the worker HPA on in dev produced four idle workers and a pod
churn every sync: the chart rendered replicas: 1, Argo CD re-applied
it (35 scale-downs in 3 h), the HPA scaled back to 4 because the new
pods' startup CPU sat above the 60 % target, and so on (deployment
generation 182). Fix in the chart: neither deployment renders
replicas when its autoscaling.enabled is true, so the autoscaler
owns the count and Argo has nothing to reset. The backend had the same
latent bug and would have fought under real load once its HPA went
above replicaCount 2.
Update 2026-09-23 night — the worker's idle CPU was the reconciler¶
With the replicas fight fixed the worker HPA still held 4 idle pods: each
pod pulsed to ~180m every minute. Tempo showed why: pipeline reconcile
sweep took 4-5 s per run because reconcile_pipeline_statuses recomputed
every pipeline that ever had jobs, 240 on dev, five statements each, and
rewrote the finished ones too. Fix: the sweep joins pipelines and skips
TERMINAL_PIPELINE_STATUSES (now a constant on the model, shared with
pipeline_progress). A terminal pipeline never changes again, so the
sweep is O(in-flight pipelines), a few ms at rest. Regression test:
test_reconcile_skips_terminal_pipelines (fails on the old query with
checked: 2). openshift-app-config#65 had already raised the worker CPU
request to 250m so the HPA could scale in; with this fix idle sits far
below either request.