Job runner: one OTel span per job execution¶
Problem¶
Background-job SQL statements export as parentless root spans: every
UPDATE pipelines SET ... / SELECT emitted from the job runner is its
own single-span trace in Tempo. During the 2026-08-25 stage
investigation (#2360) this flooded trace search — HTTP requests nest
their SQL correctly (the request span is the parent), background jobs
don't, so job activity is impossible to read as a unit and filtering
endpoint traces from SQL noise needs {kind=server} gymnastics.
Two emitters, both outside any span:
- Job handlers — everything dispatched through
app/tasks/runner.py::run_job(handler SQL,finish_job, the post-commitrecompute_pipeline_status, heartbeat ticks). - The pipeline reconciler sweep
(
app/tasks/_pipeline_reconciler.py) — theUPDATE pipelines SET expected_recalc/status/job_countbookkeeping seen as root traces at 2026-08-25 15:47:12 comes fromrecompute_pipeline_statuson the sweep cadence.
Solution¶
Telemetry-only — zero changes to pipeline/job logic, ordering, sessions,
or transactions. opentelemetry-distro is already a direct dependency;
without the OTLP env vars (local/tests) the API is a no-op, so no config
plumbing.
run_jobopens one span per claimed job execution, namedjob <type>(reads well in Tempo search), with attributesjob.id,job.type, andpipeline.idwhen the job carries one. The span is started manually (tracer.start_span+context.attach) right after the post-claim re-fetch — before the heartbeat task is created, so the heartbeat'sUPDATEs inherit the context too — and detached + ended in the existingfinally. Manual start/attach instead of awithblock keeps the runner's heartbeat/preemption try/finally structure untouched. Async SQLAlchemy instrumentation picks the span up from the OTel context, so all SQL emitted during the job nests under it.- The reconciler loop wraps each sweep iteration in a
pipeline reconcile sweepspan (one span per orchestration step, not per statement), coveringreconcile_pipeline_statusesand the orphan-aggregation backfill.
Pre-claim SQL (the initial get_job_by_id + claim_job) stays outside
the span by design: the span represents a claimed execution, and a
no-op run_job probe (already claimed / finished) should not mint a
trace.
What changed¶
backend/app/tasks/runner.py— module-leveltracer = trace.get_tracer(__name__); span start + attribute stamping after the post-claim re-fetch;context.detach+span.end()in the existingfinally.backend/app/tasks/_pipeline_reconciler.py— sweep body wrapped intracer.start_as_current_span("pipeline reconcile sweep").backend/tests/unit/tasks/test_runner.py—test_run_job_handler_runs_inside_job_span: a local SDKTracerProvider+InMemorySpanExporterpatched onto the runner's tracer; a stub handler captures the current span mid-run and the test asserts it is the recordingjob test_jobspan (matching span id,job.id/job.typeattributes) and that exactly one span is exported afterrun_jobreturns.
Verification¶
uv run pytest backend/tests/unit/tasks/test_runner.py -x— new span test plus the existing runner matrix pass.make lint/make type-checkpass on the backend.- On stage (OTLP configured): job SQL appears as children of
job <type>traces; reconciler bookkeeping appears underpipeline reconcile sweep— no more parentless SQL root spans from background work.