Debugging With Traces¶
Six recipes for turning a symptom into a cause with Tempo, the per-env Grafana "specific graphs" dashboard, GlitchTip and the OpenShift console log view. Every step below is reconstructed from a plan or an issue where it was actually done; the steps nobody has done yet are marked UNVERIFIED so you know which part you are testing for the first time. Links to the tools are in Operations. This is a multi-procedure runbook and runs past the usual page budget.
What the signals can and cannot do¶
- Tempo (enac-k8s-grafana, Explore,
Tempo data source). Services are
backendandworker; every span carriesk8s.pod.nameandk8s.namespace.name. The collector stampsroute_class(api,probe,stream,upload,job) on server spans, so{ span.route_class = "api" }works. Sampling isalways_onin all three environments; prod's is temporary, revert by 2026-09-17. Backend traces carry psycopg query spans withdb.statementin all three environments. Worker traces carry them on dev and stage only: the prod worker runs the chart default, so a prod job trace is the job span with nothing under it. Reproduce a slow prod job on stage to see its SQL. - Trace duration is not request duration. Since #2372 the browser keeps one trace id per navigation, so a "trace" spans the whole page session. Read server-span durations, table type spans, never the trace total (plan 2449).
- Logs and traces do not join by id. The backend puts no trace id in
its JSON logs or response headers. The pivot is
k8s.pod.nameplus the timestamp: read them off the span, then filter the console log view to that pod and minute. - GlitchTip events carry
contexts.trace.trace_id. Copy thetrace_id, never thespan_id, which matches no backend span. Whether thattrace_idfinds its backend spans in Tempo is UNVERIFIED end to end. Last attempt, 2026-09-07: an id from a 2026-08-25 event (before #2372 shipped) returnedNot Found, which proves nothing. Test with an event less than a day old. - Background jobs are one span named
job <type>withjob.id,job.type,pipeline.id(plan 2371). Job types:aggregation,reference_ingest,emission_recalc,module_emission_recalc,simulator_plan_prefill,csv_ingest,api_ingest,factor_ingest,unit_sync.
The TraceQL that was actually typed during #2049, verbatim:
{ duration > 30s && resource.k8s.namespace.name = "svc1751d-co2-calculator-dev" }
with table type spans and limit 200. Swap the namespace and the threshold; that query is the backbone of recipes 1, 2 and 5.
1. A user reports a slow page¶
Trigger: "the page hangs" or a ky timeout toast. ky gives up at 10 s; the backend keeps running, so the span still lands in Tempo later.
- Tempo, last 24 h, spans table, sorted by duration:
{ resource.service.name = "backend" && span.route_class = "api" && duration > 5s }
Without the route_class filter the result fills with
GET /v1/sync/pipelines/{id}/stream spans that run 6 to 170 s by
design (plan 2449).
- Open the slowest server span and compare it with its children. Two shapes seen so far:
- One SQL span is the request. #2404: a 14 192 ms
GET .../modules/purchase/it_equipment?page=1&limit=20had 23 spans, thedata_entriesSELECT took 14 149 ms and thecount(*)on the same filter 3 ms. Copydb.statement, runEXPLAIN (ANALYZE, BUFFERS)on it, look for a correlated subquery or a planner estimate that is off by orders of magnitude (2 vs 4 000 rows there). Fix was indexes plus page-first querying. - Many cheap spans, one page. #2360: GlitchTip breadcrumbs showed
about 12 failed GETs in one burst, 7 of them the identical
.../simulator/explore/unit/455/reference-year/2025/lookup. A HAR from the browser confirmed 48 API calls on one load, 35 unique, with server TTFB summing to 15.1 s and client queueing to 0.3 s. That is a frontend stampede, fixed by an in-flight promise cache. Confirm with the Network tab, not with Tempo. - If the request is a job or a stream, use
{ span.job.type = "..." }orroute_class = "stream"instead; latency alerts exclude those on purpose.
2. A 5xx alert fires¶
Trigger: HighErrorRate or ErrorRateSustainedElevated mail, or
HaproxyHighErrorRatePerRoute.
- Grafana "specific graphs", panel Backend responses by exact status code (OTel), then Backend route - HTTP responses by status class (HAProxy). If HAProxy shows 5xx and OTel shows none, the backend never saw the request: pods or Route, go to Operations.
- Tempo, same window, spans table:
{ resource.service.name = "backend" && span.http.status_code >= 500 }
UNVERIFIED: the attribute name http.status_code follows the
same old semantic-convention set the collector already matches on
(http.target, http.route); confirm once on a live 500 and delete
this note.
- Read the exception on the span's events. For a Pydantic response
validation error, GlitchTip is faster: #2226 arrived as a GlitchTip
event whose body already held the URL, method, status 500 and
ValidationError ... standby_power_w ... got a number with a fractional part. The fix was a type widening; no trace needed. { resource.k8s.namespace.name = "..." && span.http.status_code >= 500 }over 6 h tells you whether it is one route or everything. Everything at once withFATAL: remaining connection slotsin the message is recipe 4.
3. A 403 wave¶
Trigger: several users lose access at once, RoleSyncSuspiciousEmpty
or RoleSyncProviderUnavailable fires.
This one is diagnosed from logs, not traces: the request that did the damage returned 200, and the 403s come later on other requests (plan 2531).
- OpenShift console log view, application logs, search the message
Fetched roles from Accred APIfor the affected user. Readtotal_authorizationsandco2_roleson that line. total_authorizations > 0andco2_roles == 0: the mapping dropped every authorization. Look for the neighbouring warningsAuthorization missing accredunitid, skippingorAuthorization missing cf, skipping: they name the field that moved on the provider side.- Both zero: the provider returned nothing. Since #2531 the guard
refuses to persist that, and #2623 counts it as
role_sync_skipped_total{outcome=...}, which is what the two alerts watch. backend/scripts/diagnose_accred_roles.py <sciper>reproduces the mapping for one user from a pod.- Tempo adds nothing here except confirming the user's earlier requests
returned 200: filter
{ span.user.id = "<id>" }(the attribute is set on the current span at auth time).
4. DbPoolCheckoutTimeout fires¶
Trigger: DbPoolCheckoutTimeout, DbServerConnectionsHigh,
DbServerConnectionSlotsExhausted, or every request failing with
FATAL: remaining connection slots are reserved.
- Grafana "specific graphs", panels DB Pool Usage and Total DB
connections vs Postgres max_connections (100). Read the total
open series against the 100 line, not the pod ceiling: on
2026-08-31
checked_outpeaked at a healthy 13 fleet-wide while the server was full, because 53 of 85 connections belonged to pods that no longer existed (plan 2566, "Why no dashboard caught it"). - If total open is near the ceiling and pool usage is low, orphans. From psql:
select client_addr, backend_start, state, count(*)
from pg_stat_activity where datname = current_database()
group by 1, 2, 3 order by 2;
Clusters of backend_start older than the newest ReplicaSet are pods
that died without engine.dispose(). That path is fixed; if you see it
again the fix regressed.
- If pool usage is high and total open is fine, real saturation: Tempo,
{ resource.service.name = "backend" && duration > 5s }in the window. On 2026-09-03 Tempo listed 26GET /v1/sync/pipelines/{id}/streamrequests all ending at 5.0x s withQueuePool limit ... timeout 5.00(plan 2654). Long streams holding connections during a recalculation is the known shape; read plan 1723 before changing pool sizes.
5. Timeout with no trace¶
Trigger: a 504 or a client timeout, and the Tempo search for that route and minute returns nothing.
Five known reasons, in the order to eliminate them:
- Client gave up, server finished later. ky times out at 10 s, the
backend route timeout is 10 min. Widen the time window to +5 min and
search
{ duration > 10s }again. - The pod restarted mid-request. In-flight spans die unexported.
Grafana panels Container restarts (15m increase) and OOM kills,
or
increase(kube_pod_container_status_restarts_total{namespace="..."}[15m])in Explore. This was the 2026-08-25 stage incident (#2360). - Early spans dropped on export. #2050 saw a
PATCH /v1/project-plans/8011/years/2025504 after 91.5 s with about 70 s before the first captured span, and a 31.8 s hole with zero spans in another. The gap is real time; the trace is just missing it. Fall back to the pod's logs for that minute. - The work is invisible to instrumentation.
COPY FROM STDINis not traced by psycopg instrumentation (#2050 Track F0: an 18.5 s gap with zero DB spans). Worker traces on prod have no DB spans at all. - The worker's exporter timed out. #2449 found
DEADLINE_EXCEEDEDtootel-collector:4317on the worker pod; its traces were partial.oc logs deploy/co2-calculator-worker -n <ns> | grep DEADLINE.
A 504 at exactly 30 s used to mean the HAProxy router default; the backend route now carries a 10-minute timeout. Server-side signals can still log a success for a request the client already abandoned (half-closed socket, #2147), so a clean trace does not prove the user got a response.
6. A frontend error lands in GlitchTip¶
Trigger: GlitchTip mail or the Teams channel.
- Read the event first. It has the failing URL, method, status, the
response body,
componentNameandpropsData, and breadcrumbs of the requests before it. #2226 (a 500) and #2360 (a burst of identical GETs) were both solved from the event alone. - Copy
contexts.trace.trace_id. In Tempo, paste the bare 32-hex id as the TraceQL query, or use the spans table with:
{ trace:id = "<trace_id>" && resource.service.name = "backend" }
UNVERIFIED: expected to return the backend spans of that navigation since #2372. Report the result on the issue either way.
- No hit and a status 5xx in the event: fall back to the route and timestamp, recipe 2 step 2.
- Stack traces are minified: source maps are not uploaded (#1101). Match
the
componentNameto the component instead.
When you learn something new¶
Add the recipe here in the same PR as the fix, with the trace id or GlitchTip issue number that proves it. A recipe without a citation is a guess; this page holds none.