2531 — Role sync must not read "I got nothing" as "there is nothing"¶
The bug¶
RoleSyncService.sync_user_roles compared the provider's role list against
the stored one and wrote whatever came back. An empty provider response was
therefore persisted as user.roles = [], logged at INFO as
User roles updated, and the user 403'd on their next request. The sync runs
in a FastAPI BackgroundTask fired by POST /v1/session (token refresh, not
page load — ADR-017 settled GET /v1/session as a pure DB read), so the
request that caused the damage succeeded — the symptom appeared later, on a
different request, with no error anywhere.
An empty read from an external system is ambiguous. Treating it as fact manufactures a claim ("this user has zero roles") from an absence of data and then persists it. That is the no-silent-fallbacks invariant seen from the other side.
Leading hypothesis (issue discussion): the mapping loop in
AccredRoleProvider.get_roles drops an authorization on four conditions, an
Accred response-shape change trips all of them at once (the schema has moved
before — resource.cf → resource.altname), and a user with exactly one
authorization then maps to zero roles even though Accred returned data.
What shipped¶
1. Guard the wipe (app/services/role_sync_service.py)¶
If the provider returns no roles while the user has some, the sync keeps the
stored roles, does not stamp last_roles_sync_at (so the next call
retries rather than believing it succeeded), and logs at ERROR. Empty-in /
empty-out for a user who genuinely has no roles still settles normally, so
role-less users do not spin.
2. Provider failures made loud, not silent (app/providers/role_provider.py)¶
- Unconfigured Accred credentials raise
RoleProviderNetworkErrorinstead ofreturn []/return {}— one pod with rotated credentials could otherwise wipe every user it served. - Wholesale drop detection: the mapping loop skips an authorization on
four conditions, and an Accred response-shape change trips them all at
once. When authorizations arrive and none map to a role,
_log_wholesale_droplogs the offending payload's keys at ERROR — naming the field that moved — but deliberately does not raise. It raised in the first version of this fix (_raise_on_wholesale_drop); a required PR review finding reverted that (f57eefaf4) becauseAccredRoleProvider.get_user_by_user_idis also login's role-resolution call, so raising turned a user whose only authorization lackscf/altname— this issue's own leading hypothesis — from "logs in with zero roles" into "permanently locked out behind a 503". Refusing to act on the resulting empty result is already §1's job (RoleSyncService's wipe guard); this layer only has to make the cause visible.
3. Honest outcomes (RoleSyncOutcome)¶
RoleSyncResult now carries an outcome: applied, no_change,
skipped_ttl, skipped_user_not_found, skipped_provider_unavailable,
skipped_suspicious_empty. sync_user_roles catches
RoleProviderNetworkError and reports it as an outcome, so
trigger_role_sync_for_user logs one honest line instead of duplicating the
handling. Every skipped_* outcome writes nothing.
4. Login stamps last_roles_sync_at (app/services/user_service.py)¶
Login had just fetched authoritative roles from Accred but left the stale
timestamp, so the TTL gate was already expired and the very next
/v1/session re-synced — double provider load per login, and the wipe window
reopened seconds after a user recovered. _upsert_user_identity now stamps
the sync time whenever it stores provider-supplied roles.
5. One source of truth for the role namespace¶
ROLE_NAME_PREFIX is derived from RoleName in app/models/user.py and used
as the Accred searchauthorization filter. calco2. is the correct prefix;
the stale co2. references in comments, in login_test's default parameter
(which yielded a user with zero roles), and in the OpenAPI description are
fixed. Hygiene, not the trigger.
6. Close the second wipe path (app/repositories/user_repo.py)¶
Found and fixed as a required PR review finding, in the same commit as §2's
revert (f57eefaf4). Unit sync reaches bulk_upsert through map_api_user,
which builds User(**user_raw); roles is a property over roles_raw, so
the SQLModel constructor silently drops the roles= kwarg and leaves
roles_raw None. session.merge() copies every attribute, so persisting
that None overwrote the roles Accred had just resolved — a second wipe path
that §1's guard never covered, since it only guards the /v1/session
background-sync call, not unit sync. bulk_upsert now restores the existing
user's stored roles_raw whenever the incoming payload's is None; an
explicit empty list still clears, since that is a caller stating the user has
none.
Regression tests¶
| Test | Pins |
|---|---|
test_empty_provider_response_does_not_wipe_existing_roles |
the wipe itself |
test_suspicious_empty_leaves_sync_timestamp_untouched |
the next sync retries |
test_non_empty_role_change_still_applies |
the guard does not freeze roles |
test_user_with_no_roles_receiving_no_roles_is_not_an_error |
no retry loop for role-less users |
test_provider_network_error_skips_without_touching_roles |
abort path writes nothing |
test_login_stamps_sync_time_so_next_session_call_is_ttl_gated |
login stamps freshness |
test_accred_unconfigured_raises_on_get_roles_by_user_id / ..._get_user_by_user_id |
unconfigured raises |
test_accred_logs_but_does_not_raise_when_all_dropped |
schema move is visible, not fatal |
test_role_name_prefix_matches_every_role_name / test_accred_query_uses_role_name_prefix |
filter cannot drift from the enum |
test_bulk_upsert_keeps_stored_roles_when_payload_carries_none |
the second wipe path |
test_bulk_upsert_still_clears_roles_on_an_explicit_empty_list |
an explicit revocation still clears |
test_bulk_upsert_leaves_a_new_user_without_roles |
a new user isn't defaulted |
tests/unit/tasks/test_role_sync_skip.py (the JwtClaimsRoleProvider
background-sync opt-out from #2526) still passes unchanged.
Maintainer decisions (resolved in PR review, 2026-08-30/31)¶
1. The force escape hatch floated in the original draft of this plan is
declined. It would reopen the exact failure mode this PR closes, and no
production caller passes force=True today. Deprovisioning is a separate
design, tracked in
#2539: a
two-strikes roles_empty_since timer (don't wipe on the first empty; believe
it on a second empty after 2× the sync TTL), an explicit admin revoke action
(the right home for a force that bypasses the guard, since human intent is
unambiguous where an empty API response is not), and an optional
/persons/{id}-absence fast path as an optimization on top of the timer.
2. Corrected risk statement — revocation is not actually silenced. An
earlier version of this plan (and the PR description) claimed a revoked user
"keeps access until someone reads the ERROR log". That is wrong:
UserService.upsert_user (the login path) has no guard — it writes whatever
role list the provider returns, including []. A genuinely de-accredited
user loses access at their next login; this PR only stops the automatic
background sync from doing that destructively and without evidence. The
exposure window is bounded by the refresh-token lifetime — 12 h in
backend/.env, 24 h by config default — not indefinite, and not dependent on
anyone reading a log.
3. No backoff on the retry. Not stamping last_roles_sync_at on the skip
paths is what makes the next sync retry, but it also removes the TTL backoff
for exactly the users who are failing. If the cause is fleet-wide (an Accred
schema change), every /v1/session from every user re-hits Accred with no
rate limit for as long as it lasts. Deliberately deferred to
#2539 rather than
blocking this PR — carried over verbatim in that issue's body as "add a short
backoff on repeated skips before this reaches production".
4. The drop-detection hardening question is resolved by §2 above. A user
holding only an authorization that doesn't map (missing cf/altname, an
unrecognized role name) now logs in with zero roles again instead of hitting
a 503 — the soft failure this issue asked for, not a hard lockout.
Not in scope¶
AccredRoleProvider.get_roles's except ValueError: return [] is a surviving
silent fallback in the same file (missing uniqueid in userinfo). Left alone —
it belongs to the login path, not the sync path.
Deprovisioning (making an empty response eventually count as a real revocation signal) is out of scope here by design — see #2539.
Alerting on the guard firing is also out of scope here. The guard logs at ERROR (§1, §2) but nothing counts or pages on it today — no metric, and no backend error-tracking SDK (Sentry/GlitchTip is frontend-only). Filed separately as #2623 rather than folded into #2539, since it's an orthogonal observability gap (make the existing log reach someone) independent of how deprovisioning is eventually designed.
Status¶
PR #2538: both
required review findings fixed (f57eefaf4), all CI checks green, mergeable.
Still open as a draft. Outstanding before merge:
- Un-draft.
- The PR description still carries the stale risk statement corrected in "Maintainer decisions" §2 above — reword before merge so the merged history doesn't preserve the wrong claim.