2623 — Alert on role sync's suspicious-empty guard¶
The gap¶
#2538 added a
guard that refuses to wipe a user's roles on an ambiguous empty response
and logs at ERROR instead. That stops the damage; it does not make the
event visible to anyone — it was a log line nobody was notified about.
No metric, no backend error tracking (Sentry/GlitchTip is frontend-only —
APP_SENTRY_DSN), and no HTTP signal (the sync runs in a BackgroundTask,
so the triggering request already returned 200). This was proposed in
2531's own original issue body (proposed fix, item 5: "alert on it — a¶
metric, not just a log line") but never made it into #2538 or #2539.
What shipped¶
Backend counter (app/services/role_sync_service.py, shared with #2539)¶
_role_sync_skipped = get_meter(__name__).create_counter(
"role_sync.skipped",
unit="{sync}",
description="Role sync outcomes that left a user's stored roles untouched",
)
Mirrors the existing app/db.py pattern (db.pool.timeouts,
db.connect.failures — both from #2572). Incremented with
{"outcome": ...} on RoleSyncOutcome.SKIPPED_SUSPICIOUS_EMPTY and
SKIPPED_PROVIDER_UNAVAILABLE. The existing OTel collector pipeline
exports it to Prometheus as role_sync_skipped_total{outcome=...} — no new
infrastructure, same plumbing the existing request-latency alerting rides
on.
Alert rule (openshift-app-config, all three envs)¶
Added to epfl/co2-calculator/overlays/{dev,stage,prod}/monitoring/specific-namespace-alerts.yaml,
following the exact sparse-counter two-arm pattern DbPoolCheckoutTimeout
/ DbServerConnectionSlotsExhausted already use (#2572/#2573) — increase()
for later events, plus an unless ... offset arm to catch the series'
birth (which increase() reads as no increase, and the very first event
ever is the incident itself). No absent() deadman is possible for the
same reason an event counter's silence is indistinguishable from health.
No dashboard panel — #2572's sibling counters didn't get one either (sparse revocation-adjacent events are alert-worthy, not trend-worthy; a panel with near-permanently-empty data isn't useful).
Not in scope¶
Whether an empty response should count as a revocation at all is #2539's question, not this one's — this issue is purely "make the existing guard's ERROR log reach someone," independent of how #2539's two-strikes design works.