Suppress per-chunk ASGI receive/send spans (#2397)¶
Delivered 2026-08-27 as #2409, implementing option A exactly as recommended below. The body is left as originally written — it's the record of the decision, not a changelog.
Split out of #2049 (was D2, promoted to B6 on 2026-08-26 — its "after A and B land" placement came from a rationale that never applied to it).
The finding¶
A single traced upload produced 805 spans — one http receive per body
chunk — each created and exported on the event loop, on the request hot
path. Measured in the v3 trace investigation (trace a7cfb477…, 3.037 s,
816 spans total).
Two costs, and they are different:
- Event-loop time. Span creation and export are synchronous work on the same loop that serves every other request. This is the one that matters.
- Trace volume. 805 spans of no analytical value per upload.
What is already verified¶
Checked against the actual pinned packages, not assumed:
-
opentelemetry-instrumentation-asgi==0.65b0supports it natively.OpenTelemetryMiddleware.__init__takesexclude_spans: list[Literal["receive","send"]] | Noneand setsself.exclude_receive_span/self.exclude_send_spanfrom it. NoSpanProcessorworkaround is needed — #2049 listed this as an open question; it is answered. -
It is a keyword argument, not an environment variable. There is no
OTEL_*constant for it anywhere in the instrumentation packages. So it cannot be switched on the way every other OTel setting in this app is switched on — via the ops-repo overlay. -
The app is instrumented inside the
FastAPI(...)constructor.backend/Dockerfile:72runsopentelemetry-instrument uvicorn app.main:app. That callsFastAPIInstrumentor._instrument(), which monkey-patchesfastapi.FastAPIwith_InstrumentedFastAPI, whose__init__callsinstrument_app(self, **_instrument_kwargs)— with the auto-instrumentation's kwargs, which don't includeexclude_spans.
So by the time app = FastAPI(...) returns in main.py, instrumentation
is already applied with the wrong settings.
That mismatch is the entire design problem. Everything below is a way to get one keyword argument past a construction step we don't control.
Options¶
A. Uninstrument and re-instrument at boot — recommended¶
Immediately after the app is constructed:
FastAPIInstrumentor.uninstrument_app(app)
FastAPIInstrumentor.instrument_app(app, exclude_spans=["receive", "send"])
uninstrument_app is public API and clean: it restores
_original_build_middleware_stack, rebuilds app.middleware_stack, restores
BackgroundTask.__call__, and clears
_is_instrumented_by_opentelemetry. Verified by reading the pinned source.
- Cost: one middleware-stack rebuild at boot. Nothing at request time.
- Risk: it must be done once, at a known point, and it must fail loudly if it throws — a silently un-instrumented app would delete every trace we have, which is far worse than 805 spans.
B. Pre-seed _InstrumentedFastAPI._instrument_kwargs¶
Set the kwarg before constructing the app. Rejected: it writes to a private class attribute of a pinned third-party package. It would break silently on upgrade — and "silently" is the operative word, since the failure mode is the spans quietly coming back.
C. Stop using opentelemetry-instrument for FastAPI; instrument manually¶
Rejected for now. It would change how every instrumentation in the app is configured, not just this one. Far larger blast radius than the problem justifies.
D. Drop the spans collector-side¶
A filter processor dropping * http receive / * http send.
This does not achieve the goal. The spans would still be created and exported on the event loop; only Tempo storage is saved. It addresses the symptom we can see and not the cost we care about. Worth doing as well if trace volume becomes a problem, but it is not this fix and must not be mistaken for it.
Sequencing — deliberately after #2263¶
Land #2263 (event_loop_lag_seconds) first, let it record a baseline,
then land this and compare.
The reason is honesty about the premise: "805 spans on the event loop are blocking it" is a hypothesis. The 683 ms of unattributed tail time in the taxonomy traces is a residual, not a measurement, and it could be serialisation, GC, span export, or queueing behind something else. The lag probe is the only instrument that discriminates. Shipping this fix first and declaring victory would be exactly the pattern this plan's own corrections list keeps catching.
Expected, and falsifiable: upload trace span count drops from ~805 to
~11. If event_loop_lag_seconds also drops during uploads, the hypothesis
holds and we have a number. If lag does not move, that is a real result —
it means the spans were cheap and the 683 ms lives somewhere else, and the
next step is D1's triage rather than more span-suppression.
What we lose¶
Per-chunk http receive visibility. We have never used it: the upload
investigation used the span count as a symptom, never the individual spans.
The upload's meaningful spans — auth, the S3 round-trips, the root — are
untouched.
If per-chunk timing is ever needed again, it is one boolean away.
Test¶
A behavioural test, not a configuration assertion: drive an upload through
the ASGI app with an in-memory span exporter and assert no span named
* http receive is produced, while the root request span still is.
That pins the actual behaviour, so it survives an instrumentation upgrade
that renames or moves exclude_spans — which is the realistic failure mode
given point 2 above.
Explicitly not in scope¶
OTEL_PYTHON_DISABLED_INSTRUMENTATIONS and OTEL_TRACES_SAMPLER are a
separate item (#2049 C2), gated on A1. This changes neither.