Observability & SLOs¶
What we alert on, why, and where the coverage still has gaps. For how the
route_class label itself came to exist, see
1402 — trim down alerting;
for the pipeline-performance follow-ups, see
2049 — optimize pipeline performance.
This page is the durable "what's live now" reference; those plans are the
history of how we got here.
The route_class label¶
Every backend request produces http_server_duration_milliseconds in
Prometheus, labelled by http_target and http_status_code. On its own
that's not enough to alert on: a 34-minute SSE stream and a 40ms health
check land in the same histogram. The ops-repo OTel Collector adds a
route_class attribute (api / probe / stream / upload / job) via a
transform processor, derived from http.target pattern-matching — before
the metric ever reaches Prometheus. Alerts and dashboard panels filter or
split on this label instead of guessing per-route regexes.
route_class="stream" is deliberately excluded from all latency
alerting. Stream duration tracks the underlying job's duration by design —
seconds for a small import, tens of minutes for a large one — so there is no
meaningful "too slow" threshold for the class as a whole. The histogram's
last finite bucket is 10s; anything past that is bucketed as unresolvable
+Inf, so a raw quantile on stream isn't just unhelpful, it's not
computable in a useful way. If a real stream-specific problem needs
detecting, it needs a different signal than request duration (e.g. a stuck
job, covered by the job-class alerts and the pipeline's own health
metrics), not a latency SLO.
The dashboard follows the same rule. The Latency ${percentile:text} by
route_class panel leaves out stream, and probe too, which has its own
panel. A percentile dropdown (P50/P95/P99, default P95 + P99) draws one
panel per selected value
(openshift-app-config#71).
Probe trace sampling¶
Metrics (route_class) and traces (Tempo, via enac-it-otel) are separate
signals from the same span data, sampled independently. The collector's
tail_sampling drop-health policy was meant to cut health-check trace
noise, matching literal /health//healthz//ready on http.target -- no
/api prefix. Every request's http.target actually carries that prefix
(same fact 1402 already proved for the metrics side), so the literal match
never fired: with invert_match: true, the policy was a total no-op --
every trace was already being kept, health and non-health alike, and none
of the intended noise reduction was happening.
Independent of that bug, a flat allow/deny list could never keep an anomalous slow probe's trace while dropping the routine fast ones anyway -- found while chasing a ~400ms P99 spike on the probe latency panel with no matching trace in Tempo. Fixed in dev only with a composite policy: non-health traffic is always kept (unchanged), health/probe traffic is kept only if a span exceeds 200ms, or by a light 5% sample otherwise. Not yet promoted to stage/prod -- verify in dev first (a deliberately slow probe should produce a trace; routine probes should drop to roughly 5% sampled), then promote. Tracked in co2-calculator#2302.
Per-environment status¶
| Signal | Dev | Stage | Prod |
|---|---|---|---|
route_class label (collector transform) |
✅ | ✅ | ✅ |
Grafana: Latency ${percentile:text} by route_class, one panel per percentile |
✅ | ✅ | ✅ |
| Grafana: probe / DB pool panels | ✅ | ✅ | ✅ |
LatencyP50/95/99High scoped to route_class="api" |
✅ | ✅ | ✅ |
BackendMetricsAbsent (deadman's switch) |
✅ | ✅ | ✅ |
HighErrorRate -- global (not per-pod) 5xx ratio |
✅ 2% | ✅ 2% | ✅ 1% (retuned from data) |
ErrorRateSustainedElevated -- 6h window, severity:info |
✅ 0.3% | ✅ 0.3% | ✅ 0.3% |
UploadLatencySLOBreach |
✅ (thresholds from 4wk stage data) | ✅ | 🟡 interim -- stage-derived thresholds, see below (#2301) |
| Job-class latency alerts | ❌ removed 2026-09-07 | ❌ removed 2026-09-07 | ❌ removed 2026-09-07 |
| Probe trace sampling -- latency-aware, not blanket drop | ✅ | ⬜ not yet promoted (#2302) | ⬜ not yet promoted (#2302) |
Prod's route_class transform only shipped recently. UploadLatencySLOBreach
is live there now, but running on stage's 4-week-derived threshold
(5s@2%) as an explicit interim value -- prod has no traffic history of its
own yet to derive a number from, and 1402's own precedent is "pull real
numbers, don't guess." Zero coverage for the weeks it takes to accumulate
that data seemed worse than a clearly-labeled approximation; the alert says
"interim" in its summary text. Retune from real prod route_class="upload"
history once available -- tracked in
co2-calculator#2301.
HighErrorRate was retuned from real data, not carried over. Pulled
4-week 5xx/total on 2026-08-24: dev 0.081%, stage 0.0085%, prod 0.0012% (1
in ~83,600 requests) -- all three environments are healthy, and prod's old
4xx+5xx@50% threshold was an outage detector, not an error-rate SLO. Prod is
now 5xx-only@1% (~800x headroom over its own baseline); dev/stage keep their
existing 5xx-only@2%. All three also shared the same per-pod bug as the
latency alerts -- sum by (k8s_pod_name) on both sides of the ratio fired
on any single replica breaching -- fixed to a global ratio everywhere.
ErrorRateSustainedElevated is new, all three environments.
HighErrorRate's 5-minute window structurally can't see a persistent
low-grade error rate -- it never looks anomalous in any single slice, and
the traffic floor keeps suppressing it there. This alert evaluates the same
5xx ratio over a 6-hour increase() window at a much lower threshold
(0.3%), for: 30m. severity: info, not warning -- alertmanager here is
a single flat email route with no severity-based sub-routing, so this
doesn't page differently, it just doesn't get lost as another warning.
Alert catalog (dev/stage; prod matches except where noted above)¶
| Alert | Fires when | Why |
|---|---|---|
HaproxyRouteHighLatency |
avg HAProxy latency > 1s for 5m, per route | Edge-level slowness, independent of app metrics |
HaproxyHighErrorRatePerRoute |
HAProxy non-2xx rate > 5% for 5m, per route | Edge-level errors (includes upstream-down cases the app never sees) |
LatencyP50/95/99High |
app P50/95/99 (route_class=api) over threshold for 5m | Normal-API latency only -- streams/uploads/jobs excluded so they can't skew it |
HighErrorRate |
global 5xx rate > threshold for 5m (2% dev/stage, 1% prod), with a traffic floor | Fast incident signal; 4xx excluded (client errors, not backend health); per-pod grouping fixed so one hot replica can't trip it alone |
ErrorRateSustainedElevated |
global 5xx rate > 0.3% over a rolling 6h window, for 30m | Slow-burn companion to HighErrorRate -- catches a persistent low-grade error rate the 5-minute window and its traffic floor structurally can't see; severity:info, doesn't page |
BackendMetricsAbsent |
no http_server_duration_milliseconds_count for 10m |
Deadman's switch -- every alert above depends on this metric existing |
UploadLatencySLOBreach |
>2% of uploads slower than 5s over 15m | Proportion-of-slow-requests, not a raw quantile -- the 1402 job-class p95/p99 saturated at the histogram's last bucket, making raw quantiles meaningless there |
DbPoolSaturationHigh |
per-pod checked_out / (size + max_overflow) > 80% for 10m |
Byte-identical to the "DB pool saturation per pod" panel query on purpose; fires for the worker too (tasks queue, not requests); more replicas hit the server wall sooner, not later |
DbServerConnectionsHigh |
max(db_server_connections) > 80 for 2m |
Server-wide, every client: pods, pgAdmin, Alembic, dumps, laptops, orphaned backends. max not sum (every pod reports the same number); 2m because dev went healthy to every-request-500 in minutes on 2026-08-31 |
DbPoolMetricsAbsent |
no checked_out or max_overflow gauge for 30m |
Deadman for DbPoolSaturationHigh: max_overflow is its denominator, and a missing denominator makes the rule permanently green, not noisy |
DbServerMetricAbsent |
no db_server_connections for 30m |
Deadman for DbServerConnectionsHigh, the one rule that sees orphans. Written by the pod heartbeat, Postgres only; separate so an environment without #2567 can be silenced alone |
DbPoolCheckoutTimeout |
db_pool_timeouts_total increases, or is born, within 15m |
DB layer 1 counted instead of inferred (#2572): a request waited DB_POOL_TIMEOUT 5s for a pool slot and gave up. Local and recoverable; sustained means demand exceeds the pod ceiling |
DbServerConnectionSlotsExhausted |
db_connect_failures_total{sqlstate="53300"} increases, or is born, within 15m |
DB layer 3: Postgres full, everyone locked out including pgAdmin and migrations. The 2026-08-31 mode; HighErrorRate sees the 500s but cannot name the cause |
DbBouncerQueued (dev, stage) |
db_pgbouncer_queued_total increases, or is born, within 15m |
DB layer 2: PgBouncer's server pool (60 on dev, 70 on stage) is full, that many transactions in flight at once, and one is queued. 10 s of warning before it errors; severity:warning |
DbBouncerQueueTimeout (dev, stage) |
db_pgbouncer_queue_timeouts_total increases, or is born, within 15m |
The queued transaction was refused after query_wait_timeout 10 s; the request or job behind it failed, the pod slot is back before /ready notices. severity:critical |
RoleSyncSuspiciousEmpty |
role_sync_skipped_total{outcome="skipped_suspicious_empty"} increases, or is born, within 15m |
Role sync refused to wipe a user's stored roles on an empty provider response (#2531/#2538); a second empty answer 2× ROLE_SYNC_TTL_MINUTES later is believed. severity:warning |
RoleSyncProviderUnavailable |
role_sync_skipped_total{outcome="skipped_provider_unavailable"} increases, or is born, within 15m |
Role sync could not reach the role provider and kept the stored roles; check /api/health/deps. severity:warning |
AccredUnavailable |
accred_request_failures_total increases, or is born, within 15m |
An Accred lookup failed with a 5xx, a 401/403/429, a timeout or a connection error, or Accred is not configured: every login is blocked and the login page says so. Co-fires with RoleSyncProviderUnavailable. severity:critical |
OtelExporterQueueSaturated |
otelcol_exporter_queue_size / otelcol_exporter_queue_capacity > 90% for 10m |
The collector's export queue is nearly full; spans and metrics are about to be dropped. severity:critical |
OtelExporterDroppingSpans |
otelcol_exporter_enqueue_failed_spans_total increases within 10m, for 10m |
Spans dropped at enqueue: traces in Tempo are partial from here on. severity:warning |
OtelCollectorSelfMetricsAbsent |
no otelcol_exporter_queue_size for 15m |
Deadman for the two rules above: the collector's own metrics stopped, so their silence means nothing. severity:warning |
The counter rules are sparse series with no samples until the first
event, so each expr has two arms: increase() for later events and
unless ... offset for series birth, which increase() alone reads as no
increase. No absent() deadman is possible for them: silence is
indistinguishable from health on an event counter. What each DB alert means
and which knob to turn: connection budget;
first response per alert: operations runbook.
No job-class latency alert, by decision (2026-09-07). Dev's
JobPollLatencySLOBreach / JobTriggerLatencySLOBreach /
JobRouteClassAbsent and stage/prod's JobLatencySLOBreach were removed.
Job endpoints are expected to run long, the 10s bucket could not resolve
"how much worse", and the deadman fired on every quiet dev weekend. The
route_class="job*" label still exists for dashboards and TraceQL; only the
rules are gone.
Known gaps (not yet actioned)¶
- Background task failures are invisible.
workers-style processing in this codebase is in-process FastAPIBackgroundTasks, not a separate worker fleet. If a task raises after the HTTP response has already been sent, the request already recorded200— nothing inhttp_server_duration_millisecondsreflects the failure. Closing this needs a backend-side counter for task outcomes (success/failure), not an ops-repo change. Filed as a follow-up, not yet built. - No span-status-based alerting. OTel span status (
status_code=ERROR, set when an exception propagates through instrumented code regardless of the HTTP response code) only reaches Tempo/enac-it-otelvia traces — nospanmetricsconnectoris configured in any overlay to turn that into a Prometheus metric.http_status_codeis therefore the only error dimension Prometheus alerting can see. Adding span-status alerting would mean adding that connector — a new pipeline component, not a config tweak. - Frontend errors aren't in this pipeline at all. They go to GlitchTip
(see the
frontend-observabilityextraction), a separate system with its own alerting surface, not OTel/Prometheus/Grafana.