Split Grafana p99 Latency Alerting by Endpoint Class; Add GlitchTip Alerting¶
Problem¶
The current Grafana p99-latency alert evaluates request duration across
all backend HTTP traffic with a single threshold. CSV upload
(POST /api/v1/files), pipeline dispatch/recalculation
(POST /api/v1/sync/dispatch, /sync/units, /sync/units/recalculate*,
year-configuration recalculation triggers), and job/pipeline polling
under /api/v1/sync/* are naturally slow (multi-second to multi-minute
ingestion work) and trip the alert on every use — which trains the team to
ignore it. Two SSE endpoints, GET /sync/jobs/{job_id}/stream and
GET /sync/pipelines/{pipeline_id}/stream (backend/app/api/v1/data_sync.py:1378,1811),
are long-lived connections; "duration" for a stream is meaningless and
will dominate any p99 bucket it's counted in.
No alert-rule or dashboard IaC exists in this repository. Grafana
dashboards/alert rules are provisioned in the ops repos
(enack8s-app-config / openshift-app-config) — referenced only by
comment in helm/values.yaml:217. This repo's own observability surface
is: (1) an OTel Collector scaffold in docker-compose.yml (OTLP receiver
→ Prometheus exporter on :9464, otel/otel-collector-config.yaml) with
no OTel SDK instrumentation wired into backend/app — the request-duration
histogram driving the current alert is produced by the ingress/router
layer (Traefik in compose, presumably the cluster ingress in prod), keyed
on path/router labels, not by in-process spans; and (2) frontend error
capture only — @sentry/vue in frontend/src/boot/sentry.ts, wired to
EPFL's self-hosted GlitchTip, tracesSampleRate: 0.05. The backend has
no Sentry/GlitchTip SDK (grep hits on "sentry" in backend/app are
false positives — StaleStatsEntry, RecalculationStatusEntry substring
matches). No GlitchTip alert rule (new-issue / error-rate notification) is
configured for the existing frontend project.
Design¶
Path-based split is sufficient — no new backend instrumentation needed.
Route prefixes are already stable and semantically grouped
(backend/app/api/router.py):
- Job/upload/pipeline group (slow-by-design, alert on abuse/regression
only):
POST /api/v1/files(CSV upload),/api/v1/sync/*excluding the two stream routes (dispatch,units,jobs/*,pipelines,workers,active-pipelines*,recalculation-status),/api/v1/year-configurationrecalculation-trigger POSTs. - Normal API group (everything else): CRUD/read endpoints —
/units,/factors,/taxonomies,/carbon-reports,/modules*,/backoffice*,/locations,/audit,/workspace,/unit,/users,/auth,/session. - Excluded from latency alerting entirely:
GET /sync/jobs/{job_id}/streamandGET /sync/pipelines/{pipeline_id}/stream— SSE, unbounded duration by design. If connection health needs monitoring later, do it via connection-count/error-rate, not duration.
Two alert groups replace the single rule, both defined in the ops repo against the existing ingress/router-labeled duration histogram (exact metric name to be confirmed against what the ops repo's current rule already queries, since this repo doesn't own that config):
| Group | Path match | p99 threshold | Eval window |
|---|---|---|---|
| Normal API | not upload/sync/year-configuration-recalc paths, not */stream |
existing threshold (tight, e.g. 1-2s) | existing |
| Jobs/upload/pipeline | /files, /sync/* (minus /stream routes), /year-configuration/*recalculat* |
looser (e.g. 30-60s, tune from historical data) | longer (e.g. 15m) to smooth over legitimately bursty large uploads |
GlitchTip alerting: add a project alert rule on the existing frontend
GlitchTip project — new-issue notification and an error-rate-spike rule
(e.g. >N new events/5min) routed to the team's existing notification
channel (Slack/email — whatever the ops repo already uses for Grafana
alerts, for consistency). This is a GlitchTip project setting (UI/API on
the hosted instance), not code in this repo — no SDK change needed.
Backend error tracking (no Sentry/GlitchTip SDK exists there today) is out
of scope for this issue; flag as a follow-up if backend 5xx visibility is
needed beyond what's already surfaced via the frontend's HTTP-5xx
captureMessage path (frontend/src/api/http.ts).
Contradictions found while executing (2026-08-21)¶
Per this plan's own output contract ("Contradictions: anything here that the code disproves. Put this first."):
- "No OTel SDK instrumentation wired into
backend/app" (Problem section, above) is false.backend/Dockerfile:27,72runsopentelemetry-bootstrap -a requirements | uv pip install ...and starts the app underopentelemetry-instrument uvicorn .... Auto-instrumentation is active in every deployed pod (confirmed by the ops repo's ownOTEL_PYTHON_FASTAPI_EXCLUDED_URLS/OTEL_PYTHON_DISABLED_INSTRUMENTATIONSenv vars and by real SQLAlchemy/psycopg spans in the v3 traces below). The local dev.venvjust doesn't have the bootstrap-installed instrumentors, which is why the original problem statement got this wrong. - T2's index hypothesis is refuted, and now settled.
backend/app/models/factor.py:98-103already hasIndex("ix_factors_data_entry_type_year", "data_entry_type_id", "year"). Row counts (2026-08-21):building=846,other_purchases=20,915 — ~25×, tracking the observed ~19× query-time / ~13× serialisation-time ratios. Confirmed genuine large-result-set cost, not an indexing gap.values/classificationare both consumed byModuleHandlerService.get_taxonomy(backend/app/services/module_handler_service.py:76-124), so narrowing the column list isn't free — the fix is caching with a real invalidation story (factors change on ingestion, not per request). Out of #1402's scope and a real architecture decision — filed as co2-calculator#2258 rather than improvised here. - P0-3/P0-4 — answered from source, then confirmed live.
opentelemetry-instrumentation-asgi'sTraceMiddleware.__call__records the old-semconv duration histogram (name=MetricInstruments.HTTP_SERVER_DURATION, unitms— this is exactlyhttp_server_duration_milliseconds) withduration_attrs_old[HTTP_TARGET] = targetunconditionally set whenever a target is resolved. The P0-3 count-by query was then run against both dev and stage:http_targetis present and populated;http_routenever appears. The exclusion is written onhttp_target, not a guessedhttp_route. No longer blocked on anything. - §2's "the matcher is probably a no-op" — REFUTED. The v3 plan's
headline diagnosis was that
http_target!~"^.*upload.*$"matched nothing because the label was absent, making every "we filtered uploads and it didn't help" conclusion meaningless. The live query disproves it: the label exists, and/api/temp-uploaddoes contain "upload", so the old exclusion did exclude uploads. It was never a no-op — it filtered the wrong thing. §2's second reason is the real one: uploads are 1.1% of request-time and streams were 95%, so excluding uploads removed almost nothing from the tail. Worth stating plainly because the no-op theory is what justified months of confusion, and it was wrong. - P0-6 ("we have never measured probe latency") — REFUTED.
/api/healthzand/api/readyboth appear inhttp_server_duration_milliseconds. TheOTEL_PYTHON_FASTAPI_EXCLUDED_URLS: "/api/health"override is empirically a no-op — if it matched, those series would not exist. (Probable mechanism: kubelet hits the pod directly at/healthz, and the/apiprefix exists only in the synthesizedhttp.target.) Probes are nowroute_class="probe"with their own panel. ⚠️ That no-op is load-bearing — "fixing" it would delete probe metrics and with them the head-of-line-blocking canary this whole investigation wanted. - The
/api-prefix bug has a third instance, still open. The collector'stail_samplingdrop-healthpolicy matcheshttp.targetexactly against/health,/healthz,/ready; the real values are/api/healthz,/api/ready. Exact match fails, and withinvert_match: truethe trace is kept — so probe traces are exported to Tempo on every probe interval on every pod, when the intent was to drop them. Same bug class as the two already fixed. Being fixed separately under #2049; recorded here because it belongs to the same root cause as everything else in this plan.
Done (2026-08-21)¶
- GlitchTip alert rule (new-issue / error-rate) confirmed configured on the hosted instance — done outside this repo.
-
openshift-app-configdev+stageoverlays (branchfix/1402-alerting-dev-stage, uncommitted, not pushed): - Fixed the "DB Pool Usage" Grafana panel on stage — a copy-paste typo
queried
db_pool_connections{namespace="svc1751d-co2-calculator-stage"}(dev'sdprefix + stage's suffix, a namespace that doesn't exist). HighErrorRate: 0.5 → 0.02, scoped to5..only (was lumping4..), with a traffic-floor guard (> 0.1 req/s) so one 5xx in a quiet namespace can't read as a 100% error rate.- Added
BackendMetricsAbsent(absent(http_server_duration_milliseconds_count{...})) as the in-band deadman — fires if the backend stops reporting entirely, which every other rule in the file silently depends on. PodHighCPUdisabled rather than "fixed": the quota/period math was wrong (divided by raw microseconds, ratio ~1e-5 vs threshold 0.9, could never fire) — but backend/worker setrequests.cpuwith nolimits.cpuhere (HPA targets are request-relative, e.g. 1600%), socontainer_spec_cpu_quotadoesn't exist for these pods either way. Left commented out with the reason, rather than shipping a rule that still can't fire.Watchdogleft disabled: only useful wired to an external heartbeat consumer that alarms on its absence; nobody notices one more email in the shared receiver.BackendMetricsAbsentabove is the alert that actually detects something.alertmanager-email.yamlrepeatInterval: 4h → 24h (a stuckwarningwas sending 6 emails/day).- Prod overlay untouched (scoped to dev/stage per instruction); prod's dashboard JSON doesn't even have the DB Pool Usage panel at all.
- PR openshift-app-config#8 — merged.
- Verified
http_targetlive (dev + stage,count by (http_target, http_route, http_method) (http_server_duration_milliseconds_count{...})): it's populated and already collapsed to a literal/api/{tail}(e.g./v1/sync/dispatch→/api/dispatch,/v1/year-configuration/{year}→/api/{year}).http_routenever appears. Retraction (same day): the first pass here claimed "an existing collector-side transform" does this collapse — checked the actualotel-helm-charttemplate and that's false. The only default processors arefilter(health-check datapoint dropper keyed onhttp.route, which doesn't exist on this metric — a silent no-op the whole time, same bug class as #1402's original exclusion) andtail_sampling(traces only). Where the/apicollapse actually happens is still unconfirmed — now tracked as its own follow-up, see below. - Added a
route_classlabel per §4.3 — atransformprocessor in the collector's metrics pipeline setsroute_class(probe/stream/upload/job/api) fromhttp_target, once, before Prometheus export. Replaces the no-opfilterprocessor for metrics;tail_sampling(traces) carried forward unchanged.LatencyP50High/P95High/P99High(dev+stage) now filter onroute_class="api"instead of a per-rule suffix regex — this removes streams and uploads and jobs from the normal-API latency alert in one place, not three. Known fragility: the job/upload/probe classification still keys offhttp_targettail patterns (nohttp_routeto match on), with aPOST-only guard on theyear-configuration/{year}tail to avoid catching GET/PATCH on the same tail. PR openshift-app-config#9 — merged. - Fixed a real classification bug in the above, found by querying the
live label breakdown:
/healthzand/dispatchwere both landing inroute_class="api"instead ofprobe/job. Root cause:backend/app/main.py:324setsroot_path=settings.API_DOCS_PREFIX("/api", never overridden per-env), which the ASGI instrumentation prepends to every target — confirmed via/healthzitself, which bypasses the Route/HAProxy entirely (kubelet hits the pod directly) and still showed up as/api/healthz, proving the prefix is app-side. Theprobe/jobpatterns were anchored assuming no prefix;stream/uploadworked by accident (unanchored.*absorbed it). Fixed both anchored patterns to include/api. This landed after #9 had already merged, so it's in a new PR: openshift-app-config#10 (merged). - Pulled real numbers instead of guessing:
factorsrow counts:building=846,other_purchases=20,915 — the 1338 ms query is a genuine large-result-set problem, not a missing index — filed as co2-calculator#2258 (T2, out of #1402's scope, no code changed here).route_class="upload"over 4 weeks of stage: p50 421 ms / p95 1807 ms / p99 4038 ms — clean, well-resolved.route_class="job"over 4 weeks of stage: p50 60.8 ms, but p95 and p99 both saturated at 10000 ms — the histogram's highest finite bucket. Live instance of §4.2's exact warning: the true tail is unresolvable past that boundary, so a raw quantile alert there is meaningless.- Added
UploadLatencySLOBreach(>2% of uploads over 5s) andJobLatencySLOBreach(>5% of job-class requests over the 10s bucket, since that's the last one this histogram can resolve) — proportion- of-slow-requests per §4.2, not a raw quantile, with a traffic floor on both. PR openshift-app-config#10 (merged). - Found and fixed a second tail collision, same class as the
year-configuration/{year}one:GET /v1/units(plain CRUD list, meant to stayapi) andPOST /v1/sync/units(recalculation trigger, meant to bejob) both collapse to the identicalhttp.target = "/api/units"—GETwas silently landing inroute_class="job". Samehttp.methodguard pattern. PR openshift-app-config#11 (merged). - Filed the general fix as a follow-up rather than patch every
collision as it's found:
co2-calculator#2260
— two
http.method-guarded collisions found live in one session is a pattern, not a coincidence. Proposes either enabling OTel's new semconv (http.route, which should carry the real unique template instead of the collapsed/collision-pronehttp.target) or restructuring router prefixes so no two routers can produce the same leaf shape. Root cause of the/v1/<router-prefix>disappearing in the first place still isn't confirmed — flagged as the first step of whichever direction gets picked. - Split the Grafana "Latency percentile" panel by
route_class(§4.9.1) — done, PR openshift-app-config#10:sum by (le, route_class),unit: ms(was unset), a real 1000ms threshold line (was a leftover unrendered default), exemplars on (were off). Dashboard version bumped for both overlays. - Added the
route_class="probe"panel — PR openshift-app-config#12 (merged). The plan expected this to need a new backend metric; it didn't, because probes were already instrumented (P0-6 correction above). Probe latency during a burst is the direct picture of head-of-line blocking, and it is now charted.
Status¶
#1402's functional scope is complete and live in dev + stage. All four alerting/dashboard PRs are merged (ops #8, #9, #10, #11), plus the probe panel (#12).
What remains is not code — it is one observation that can only be made while a real import is running, and a short ops note. Both are listed as unchecked in Steps below. The issue should not close until the import verification has actually been done: the whole failure mode this plan warns about is shipping the measurement fix, watching the emails stop, and declaring victory.
Steps¶
- Confirm in the ops repo which metric/labels the current p99 alert
queries — confirmed live:
http_targetis populated onhttp_server_duration_milliseconds;http_routeis not. - Add
route_classin the collector (§4.3), not per-alert regex — done, see Done section above. - Exclude
/sync/jobs/{job_id}/streamand/sync/pipelines/{pipeline_id}/stream(and uploads and jobs) from the normal-API latency alerts viaroute_class="api"— done, PR openshift-app-config#9 (merged). - Pull 2-4 weeks of historical p99 for
route_class="upload"/"job"and set a threshold from it — done, see Done section above. - Add the
route_class="upload"/"job"latency alert — done (UploadLatencySLOBreach,JobLatencySLOBreach), see Done section. - Split the Grafana "Latency percentile" panel by
route_class(§4.9.1) — done, PR openshift-app-config#10:sum by (le, route_class),unit: ms(was unset), a real 1000ms threshold line (was a leftover unrendered default), exemplars on (were off). Dashboard version bumped for both overlays. -
route_class="probe"panel (§4.9.6) — done, PR openshift-app-config#12 (merged). Needed no new metric after all: probes turned out to be instrumented already (see the P0-6 correction above), so this is the head-of-line-blocking canary the plan asked for, live on both overlays. Threshold line at 500ms — a probe should be trivially cheap regardless of what else the pod is doing. - A
pipeline_duration_secondsbusiness metric/alert (§4.7.7'sPipelineSlow) — not done, and gated. Unlike the probe panel this genuinely needs a new backend metric, which means hooking pipeline completion inrunner.py/_chain.py/_pipeline_reconciler.py— recalculation internals, which the guardrails gate on a written plan reviewed by both maintainers, however additive the metric itself is. Tracked as C4 in 2049-optimize-pipeline-performance.md. This is the metric that should eventually carry the 201s number that used to fireLatencyP99High— a 3.4-minute pipeline is a product fact and belongs on a business dashboard, not in an HTTP histogram. - Re-verify the tail-based classification (
dispatch,units,jobs.*,workers,active-pipelines.*,recalculation-status,pipelines.*,health/stale-stats,admin/recompute-stats,temp-upload) against real traffic once a pipeline actually runs in stage — a wrong regex here silently mis-classifies requests, the same failure mode this plan already flagged for the oldhttp_targetno-op. (The/healthz/dispatchandunitsbugs above were exactly this failure mode, caught twice, worth checking again with a real import.) - Configure a GlitchTip alert rule on the frontend project — done outside this repo.
- Document the two alert groups and thresholds in a short ops note, so the split isn't tribal knowledge living only in the ops repo. Done: infra/03-observability-slo.md.
- Verify: trigger a CSV upload and a normal read endpoint in stage/prod-like env, confirm only the jobs/upload/pipeline group's threshold applies to the upload and the normal-API group stays tight for the read call; confirm the SSE stream endpoints never fire either alert regardless of connection duration.
2026-08-24 update¶
- Prod parity. Prod never got the
route_classtransform (2049 already noted it as "live in dev+stage" only) — its collector metrics pipeline was still on the chart-defaultfilter(the same no-op class of bug this plan found once already: it matcheshttp.route, which doesn't exist on this metric). Shipped: thetransformprocessor, the same route_class-split dashboard panels,route_class="api"scoping onLatencyP50/95/99High(which also fixes a separate bug —sum by (le, k8s_pod_name)computed a per-pod quantile, firing on any single replica), andBackendMetricsAbsent. Thresholds left untouched: scoping toapionly removes slower traffic from the distribution, so the same threshold is strictly less likely to fire — a scope fix, not a sensitivity change. -
HighErrorRateretuned from real data, all three environments. Pulled 4-week 5xx/total on 2026-08-24: dev 0.081%, stage 0.0085%, prod 0.0012% (1 in ~83,600 requests). Prod's old 4xx+5xx@50% was an outage threshold, never revisited — now 5xx-only@1% (~800x headroom over its real baseline); dev/stage keep their existing 5xx-only@2%. All three shared the same per-pod bug as the latency alerts —sum by (k8s_pod_name)on both sides of the ratio fired on any single replica breaching — fixed to a global ratio everywhere. - New alert:
ErrorRateSustainedElevated, all three environments.HighErrorRate's 5-minute window and traffic floor structurally can't see a persistent low-grade error rate — it never looks anomalous in any single slice. Same 5xx ratio, evaluated over a rolling 6hincrease()window at 0.3% (~4x dev's own noisiest baseline),for: 30m.severity: info, notwarning— the alertmanager config here is a single flat email route with no severity-based sub-routing, so this doesn't page differently, it just doesn't get lost as anotherwarning. - Not shipped to prod, blocked on data:
UploadLatencySLOBreach/JobLatencySLOBreach. The transform just started running in prod — noroute_class="upload"/"job"history exists yet to set a threshold from. Same rule this plan already applied once for stage (§ "Pulled real numbers instead of guessing"): pull 2-4 weeks of real prod data before setting a number, don't copy stage's. - Dev-only visibility gotcha, not a pipeline bug. The dev
route_classpanel appeared blank forstream/job/uploadover a 12h window despite the data existing — confirmed viahistogram_quantilereturningNaNper class (zero counter increase in that specific trailing[5m], not a broken transform). Two independent causes stacked: (1) idle 5-minute rate windows are normal for dev's low, manual-testing-driven traffic on non-probe classes; (2) atimeseriespanel doesn't render an isolated non-null point surrounded byNaNunlessshowPointsisalways. Fixed on the dev and stage panel only:showPoints: "always", rate window[5m]→[30m]. - Open question, not yet confirmed: the
tail_samplingdrop-healthpolicy (dev/stage) matches literal/health//healthz//readyonhttp.target— no/apiprefix. This plan already proved that prefix is present on every request'shttp.targetfor the metrics side, but explicitly lefttail_sampling"carried forward unchanged" and never checked it for traces. Surfaced when a 250ms probe-latency spike (visible on theroute_class="probe"panel) had no matching trace in Tempo. Whichever way the prefix bug cuts, an unconditional string-match allow/deny list can only ever be "always drop" or "always keep everything" — it can't keep the anomalous probe trace while dropping the routine ones, which is the one case worth having the trace for. Recommend replacing it with a latency-conditioned composite policy. Not yet implemented — pending confirmation this is wanted, since it changes trace volume/cost in prod too.
Related¶
The trace investigation behind this plan (200 Tempo traces + 5 OTLP
traces) also surfaced a wider set of backend performance findings —
connection pool pre_ping cost, the factors query, 2s stream polling,
duplicate OTel instrumentation, the upload path — that are out of #1402's
scope (alerting, not performance). Those are now tracked in
2049-optimize-pipeline-performance.md
(issue #2049), not duplicated here.