Skip to content

Worker hands running jobs back on SIGTERM (#2696)

Symptom

Every worker rollout (several a day on dev) stops job processing for up to STALE_JOB_TIMEOUT_MINUTES (5 min). The old pod gets SIGTERM, uvicorn runs the lifespan teardown, the loops are cancelled and the pool is disposed, but the run_job tasks spawned through fire_and_forget were never cancelled: they die with the process, and the row keeps state = RUNNING, locked_by = <dead pod> until sweep_stuck_running_jobs on the new pod finds it stale.

Shipped

  1. DataIngestionRepository.release_job(job_id, pod_id): the CAS guard of finish_job (locked_by = pod AND state = RUNNING), the target of the stale sweep's recoverable bucket (NOT_STARTED, unlocked, attempts preserved so claim_job's retry cap still counts the run).
  2. run_job catches CancelledError around the handler: cancels the handler task, rolls back both sessions, releases the row, logs "handed back to the queue" or "no longer ours", re-raises. Test: tests/unit/tasks/test_runner_handback.py.
  3. cancel_background_tasks() in _background.py: cancels every in-flight task and waits up to JOB_HANDBACK_SECONDS (20 s) for the hand-backs; from then on a cancelled task logs at INFO, since at shutdown that is the expected outcome. Test in test_background.py.
  4. Lifespan order: safety poller cancelled first (nothing dispatches anymore), then the drain, then the other loops, then engine.dispose(). The drain has to run while the heartbeat still owns the lock and the pool is open. Test in test_lifespan_shutdown.py.
  5. Helm: worker.terminationGracePeriodSeconds: 60 (was the Kubernetes default 30), rendered on the worker Deployment. 20 s of hand-back plus dispose() fits with a stalled DB; no preStop, since nothing routes HTTP to the worker.

The same path covers the backend pods with DISPATCH_JOBS_INLINE on.

Replicas per environment

Multi-pod job processing already exists: claim_job is an atomic compare-and-set on locked_by, the heartbeat detects preemption, the sweep recovers dead owners. Going to two workers is worker.replicaCount in the environment overlay, and the Deployment strategy switches from surge to rolling on its own.

What gates it is the connection budget: a worker costs DB_POOL_SIZE at rest and 3 × MAX_CONCURRENT_JOBS under load. On dev (bouncer pool 35) a second worker at 2+7 does not fit. On prod (90 usable) two workers at 5+15 give steady 3 × 15 + 2 × 20 = 85, which fits without a rollout surge and not with one, so prod goes to two workers only after DBaaS raises the bouncer pool or max_connections, or with the worker at 5+10.

Automatic scaling, if it is ever needed

Horizontal. The signal is the backlog, not CPU: a worker waiting on the bouncer or on S3 is idle by every CPU metric while jobs pile up. count(*) FROM data_ingestion_jobs WHERE state = 'NOT_STARTED' AND job_type IS NOT NULL is one gauge on the existing 30 s heartbeat; KEDA's Prometheus scaler (or an HPA on a custom metric through the Prometheus adapter) scales on it with minReplicas 1 and maxReplicas set by the budget: (budget − backends × backend ceiling) / worker ceiling. Scale-down is safe because the hand-back above makes a terminating worker return its job within 20 s. A cooldown of a few minutes avoids flapping on a single CSV upload, which creates a burst of NOT_STARTED rows that one worker drains in seconds.

Vertical. The worker is I/O-bound (DB, S3); its CPU request of 2 is generous already. Memory is the only axis: a 144k-row CSV ingest holds the parsed rows in memory, and the 1 Gi limit is what an OOMKill would hit first. A VPA in recommendation mode on memory would tell whether the limit is right; automatic vertical scaling restarts the pod to apply, which is a rollout, so it depends on this issue's hand-back as well.

Neither is worth building before the DB budget grows: with 35 slots on dev a second worker cannot even start, and on prod the ceiling rule leaves room for exactly one more. The manual replicaCount per environment, with the budget arithmetic in the overlay comment, is the whole scaling story until then.

Rejected

  • Waiting for the running job to finish before exiting: job 59 ran 70 min; no grace period covers that, and a rollout stuck behind one job is worse than a re-dispatch.
  • Longer STALE_JOB_TIMEOUT_MINUTES or a shorter one: the sweep is crash recovery; the graceful path should not depend on it at all.
  • HPA on CPU: scales on the wrong signal, see above.

Verification

  • Unit: test_runner_handback.py, test_background.py, test_lifespan_shutdown.py, runner suites green.
  • helm template renders terminationGracePeriodSeconds: 60 on the worker Deployment; helm lint clean.
  • Next dev rollout: the old worker logs "handed back to the queue" and the new one logs "Poller: dispatching job N" within POLLER_INTERVAL_SECONDS, no 5 min gap in data_ingestion_jobs.

Update 2026-09-23 — worker HPA in the chart

The hand-back made scale-in safe; the chart now has the matching scale-out. helm/templates/worker-hpa.yaml mirrors the backend HPA (worker.autoscaling: off by default, 1..4 replicas, 60 % CPU target). CPU is the proxy for "job slots busy": a running job burns ~0.8 cores against a 150m request, so any active job pushes the pod far past the target and an idle pod drops back after the stabilization window. Each pod adds MAX_CONCURRENT_JOBS slots and at most MAX_CONCURRENT_JOBS + 2 in-flight transactions at the bouncer (#2854). Dev turns it on in openshift-app-config with MAX_CONCURRENT_JOBS 4. The honest signal, queue depth, needs a jobs.queued gauge and a metrics adapter on the cluster; CPU until then (#2689 follow-ups).

Update 2026-09-23 evening — HPA vs Argo CD replicas fight

Turning the worker HPA on in dev produced four idle workers and a pod churn every sync: the chart rendered replicas: 1, Argo CD re-applied it (35 scale-downs in 3 h), the HPA scaled back to 4 because the new pods' startup CPU sat above the 60 % target, and so on (deployment generation 182). Fix in the chart: neither deployment renders replicas when its autoscaling.enabled is true, so the autoscaler owns the count and Argo has nothing to reset. The backend had the same latent bug and would have fought under real load once its HPA went above replicaCount 2.

Update 2026-09-23 night — the worker's idle CPU was the reconciler

With the replicas fight fixed the worker HPA still held 4 idle pods: each pod pulsed to ~180m every minute. Tempo showed why: pipeline reconcile sweep took 4-5 s per run because reconcile_pipeline_statuses recomputed every pipeline that ever had jobs, 240 on dev, five statements each, and rewrote the finished ones too. Fix: the sweep joins pipelines and skips TERMINAL_PIPELINE_STATUSES (now a constant on the model, shared with pipeline_progress). A terminal pipeline never changes again, so the sweep is O(in-flight pipelines), a few ms at rest. Regression test: test_reconcile_skips_terminal_pipelines (fails on the old query with checked: 2). openshift-app-config#65 had already raised the worker CPU request to 250m so the HPA could scale in; with this fix idle sits far below either request.