Backend suite red: three unrelated causes (#2684)¶
Problem¶
tests/unit is red on CI right now; tests/integration is red on every dev
machine that has a populated backend/.env. Measured on dev @ 49a89971b:
| Suite | Before |
|---|---|
tests/unit |
1 failed, 2881 passed |
tests/integration |
12 failed, 459 passed, 1 skipped |
The 13 failures have three unrelated root causes. They share no code and must not be bundled into one PR.
Progress¶
- RC1 — delete the stale dotenv-precedence test
- RC2 —
get_settings.cache_clear()inpytest_configure+ comment - RC2 regression tests —
test_dotenv_cannot_reach_settings_under_pytest+ the CI-provabletest_cache_clear_is_what_drops_a_poisoned_dotenv_settings - RC2 scope check — cache_clear() covers modules imported after
pytest_configure(incl.app.api.v1.files, the S3 case this RC fixes); confirmed it does not coverapp.core.logging, eagerly imported byapp/__init__.pybefore that point. Filed as #2686 — see Scope. - #2686 fix —
app.core.loggingno longer binds a module-levelsettingssingleton at import time; both call sites (setup_logging(),LokiHandler.emit()) callget_settings()live instead, mirroringapp/core/crypto.py's existing pattern. Same PR as RC2, per feedback that a follow-up discovered while working the same issue belongs alongside it, not in a separately-scheduled issue. See #2686 fix. - RC3 — commit
building_rooms_unknown_room.csv - RC3 follow-up —
.gitignore's blanket*.csv(line 8) silently swallows every fixture underbackend/tests/fixtures/csv/; that is how #2253 lost this file without anyone noticing. Added!backend/tests/fixtures/csv/*.csvto unblock the whole directory. - ~~Split into three PRs against
dev~~ — delivered as one PR instead (owner call, overriding the sequencing recommendation below): the three fixes are small, already verified together, and splitting after the fact into 3 branches was judged not worth the overhead for this batch.
Suite is green locally with RC1+RC2+RC3 applied: tests/unit 2873 passed
in this session's environment (run count drifts slightly with the tree state
but there are no failures), tests/integration runs clean modulo tests that
need Docker/Postgres, which this sandbox does not have access to — those
errored on DockerException: permission denied, not on RC1–RC3 behaviour.
RC2 is also a safety finding: it is the same class as the 2026-09-08 prod-DB
drop — backend/.env reaching a process that should never see it. See
07-postmortem-prod-db-dropped.md.
RC1 — stale test left by the #1153 revert¶
Red on CI now (run 34241046819: 1 failed, 2872 passed).
b2ac6d1f1 reverted #1153 by deleting Settings.settings_customise_sources,
so pydantic's default precedence applies again: real env vars beat .env. It
added the new regression test but did not delete the old one asserting the
opposite behaviour, introduced by #1153 in 36eef0d7b.
tests/unit/core/test_config_provider_types.py::test_dotenv_db_url_overrides_stray_shell_env_var
asserts settings.DB_URL == "postgresql://from-dotenv/db" — deliberately
reverted behaviour. It is a stale test, not a code bug.
Fix — done¶
Delete the whole test_dotenv_db_url_overrides_stray_shell_env_var function
(and only that function) from
backend/tests/unit/core/test_config_provider_types.py.
Its replacement already exists and covers the dotenv source directly —
backend/tests/unit/test_manage_db_guard.py::test_env_var_beats_dotenv writes
a real dotenv file and asserts Settings(_env_file=dotenv).DB_URL == <env var>.
No coverage is lost, so no replacement test is needed.
Verify¶
cd backend && uv run pytest tests/unit -q
Expect 2887 passed (verified locally).
RC2 — backend/.env is live inside every pytest process¶
11 integration failures, all in
tests/integration/services/data_ingestion/, all one signature:
S3Error: Error moving file from tmp/X.csv to processing/1/X.csv:
An error occurred (NoSuchKey) when calling the CopyObject operation
Root cause¶
backend/.envcarries live S3 credentials (s3.epfl.ch, bucketsvc1751-…, prefixdata). Thewtpost_createhook copies that.envinto every new worktree.tests/conftest.py::pytest_configuresetsSettings.model_config["env_file"] = Noneto keep a dev's.envout of tests. Buttests/conftest.pyimportsapp.*at module level, and pytest imports conftest before it callspytest_configure. Those imports already callget_settings(), which is@lru_cached. Blankingenv_fileafterwards cannot invalidate the cache.make_files_store()(app/api/v1/files.py:44) returnsS3FilesStorewheneverS3_ENDPOINT_HOSTNAME+ access key + secret are set — which they are, from the cached.env-loaded Settings._stage_csv_under_files_store(in the failing test files) patches onlyFILES_STORAGE_PATH/FILES_ENCRYPTION_*, so the CSV is staged on local disk while the provider doesCopyObjectagainst real EPFL S3. The source key is not there →NoSuchKey.
The general failure is broader than S3: every .env key is live in the
test process. S3_* is simply the one pytest-env does not already blank
(it blanks OAUTH_*, JWT_HMAC_KEY, CREDENTIALS_*, …). Blanking S3_* in
pytest-env would be a symptom patch — do not do that.
Blast radius — nothing was written¶
Only 6 integration files touch the store. Two
(test_csv_ingest_matrix_pg.py, test_buildings_csv_pg.py) patch
make_files_store with an in-memory MagicMock and never reach S3 — which is
why they passed. No integration test calls
write_file/upload_file/delete_file/put_object directly (grep: zero
hits). Every S3-touching test died inside _move_to_processing at
file_exists (HeadObject) then move_file → copy_object (NoSuchKey). The
copy always failed, so the delete half of the move never ran.
Reads and 404s only — no object created, modified or deleted.
Fix — done¶
One line in backend/tests/conftest.py::pytest_configure:
from app.core.config import Settings, get_settings
Settings.model_config["env_file"] = None
get_settings.cache_clear()
app.api.v1.files is imported after pytest_configure, so its module-level
settings = get_settings() and its files_store = make_files_store()
singleton both pick up the clean object.
Update the surrounding comment to say why the cache_clear() is load-bearing:
blanking env_file alone is not enough, because conftest's own module-level
app.* imports have already populated the cache.
Regression tests — done¶
backend/tests/unit/test_manage_db_guard.py:
test_dotenv_cannot_reach_settings_under_pytest— the direct check (S3_*unset). Fails without the fix only on a machine with a populated.env; green on CI either way, which is exactly the blind spot that let RC2 live in the first place.test_cache_clear_is_what_drops_a_poisoned_dotenv_settings— CI-provable companion: builds a poisoned cachedSettingson purpose (its owntmp_pathdotenv, not a real one) and provescache_clear()specifically, not just "this machine's.envhappens to be clean", is what evicts it.
Scope, checked empirically — not everything is covered¶
get_settings.cache_clear() in pytest_configure only fixes modules
imported after pytest_configure runs. app.api.v1.files is one of
those, which is why the S3 claim above is real and verified. But
app/__init__.py:15 (import app.modules as _modules_pkg) eagerly imports a
long chain before pytest_configure ever runs, and app.core.logging is
in that chain — its module-level settings = get_settings() stays bound to
a .env-poisoned Settings object for the whole test session, cache_clear()
notwithstanding. Confirmed by direct reproduction: a populated .env with
LOKI_ENABLED=true + a fake LOKI_URL makes setup_logging() attempt a
real outbound connection to that URL during the test session (LokiHandler:
failed to push log: ConnectError). Checked the same way for app.db,
app.core.security, app.providers.* — none of those are in the eager
chain, so this is a single-module leak, not a repeat of RC2's blast radius.
Filed as #2686. Originally deferred out of this PR — reconsidered on feedback that a follow-up discovered while working the same issue, on the same branch, belongs fixed alongside it rather than mentally parked for a separately scheduled PR. See #2686 fix below.
#2686 fix — app.core.logging reads settings live, not at import¶
Two candidates were considered:
- Make
app.core.logging's settings access lazy — callget_settings()insidesetup_logging()/LokiHandler.emit()instead of binding a module-levelsettingssingleton at import time. - Detect pytest in
Settings.model_configitself (e.g.env_file=None if "pytest" in sys.modules else ".env") — fixes this and any of the ~19 other modules with the same import-timesettings = get_settings()pattern in one place, but plants test-awareness in production config code, in the exact file already flagged "precedence has flipped twice, read this before touching."
Went with (1). It's not a new pattern — app/core/crypto.py already
does exactly this (settings = get_settings() as the first line inside the
function that needs it, not at module scope). Mirrors existing code rather
than inventing a config-level mechanism, and stays contained to the one
module with the confirmed bug.
tests/unit/core/test_logging_redaction.py::test_loki_handler_is_wrapped_in_a_queue_not_attached_directly
already exercised this exact path via monkeypatch.setattr(logging_module.settings,
...), which stopped working once the module-level singleton was removed —
switched to the established monkeypatch.setenv(...) +
get_settings.cache_clear() pattern (same as test_crypto.py), and its
docstring now also carries the #2686 regression-guard rationale: a
QueueHandler only appearing after a live env-var change proves
setup_logging() isn't reading a value frozen at import time.
Verify¶
cd backend && uv run pytest tests/unit/core/test_logging_redaction.py -v
Reproduction used to confirm both the bug and the fix: populate
backend/.env with LOKI_ENABLED=true + a fake LOKI_URL, run any
backend test. Before the fix: setup_logging() attempts a real outbound
connection (LokiHandler: failed to push log: ConnectError). After: no
connection attempt — get_settings().LOKI_ENABLED reads False as
expected under pytest.
Verify¶
With a populated backend/.env present and no env blanking on the command
line:
cd backend && uv run pytest tests/integration -q
Before: 12 failed, 459 passed. After: 1 failed, 473 passed — only RC3
remains (verified locally).
Probe confirming the mechanism, before vs after: S3_ENDPOINT_HOSTNAME set?
True → False; make_files_store() S3FilesStore → LocalFilesStore.
Follow-up (do not do in this PR)¶
The 4 files using _stage_csv_under_files_store are fragile by construction:
they patch settings and hope the store resolves to local. The 2 files that
patch make_files_store with an in-memory fake are the robust pattern and are
why those tests passed. Converting the remaining files to the fake-factory
pattern is worth its own issue — it removes the dependency on store selection
entirely rather than relying on the settings being clean.
RC3 — CSV fixture #2253 never committed¶
test_buildings_csv_pg.py::test_building_rooms_csv_unknown_room_is_rejected_as_row_error:
FileNotFoundError: csv_fixture_path: ('building_rooms', 'unknown_room') is
registered but no file found. Tried: backend/tests/fixtures/csv/building_rooms_unknown_room.csv
a47816f03 (#2253) added the test and the conftest.py mapping (line 470)
but never committed the CSV. git log --all on that path is empty — it has
never existed in the repo.
Nobody noticed because PR CI runs make test-cov-xml = tests/unit only;
integration runs in a separate daily job.
Fix — done¶
Create backend/tests/fixtures/csv/building_rooms_unknown_room.csv:
unit_institutional_id,building_location,building_name,room_name,room_type,note,kg_co2eq
{unit_institutional_id},ECUBLENS,BC,BC-150,office,,
{unit_institutional_id},ECUBLENS,BC,ZZ-999,office,,
Constraints this content satisfies — the test does
template.format(unit_institutional_id=...), so the column is a str.format
placeholder (unlike the sibling building_rooms_smoke.csv, which hardcodes
SEED-BC-0). The header must match that sibling. BC-150 is the room the
test seeds into building_rooms; ZZ-999 must be absent from it, so the row
is a hard row error. One known + one unknown row is what makes the parent
report WARNING (partial success) rather than SUCCESS. The file must
contain no other { or }, or str.format will raise.
Verify¶
cd backend && uv run pytest \
tests/integration/services/data_ingestion/test_buildings_csv_pg.py -q
Expect the unknown-room test to pass (verified locally: 1 passed).
The trap that caused it¶
.gitignore:8 is a blanket *.csv — every fixture under
backend/tests/fixtures/csv/ was already tracked only because someone force-
added it (git add -f) at some point. #2253 didn't force-add this one, so
git add/git commit silently no-opped and the PR looked complete. Fixed
with a directory-scoped negation so the next fixture in that directory does
not need -f:
*.csv
!backend/tests/fixtures/csv/*.csv
Sequencing¶
Three unrelated causes → three PRs, in this order. Do not bundle.
- RC1 — deletes a stale test, turns
devgreen. Smallest, most urgent. - RC2 — the
cache_clear()line + its regression test + the comment. - RC3 — the missing fixture.
Each targets dev per the guardrails.
End state¶
Measured on this branch with RC1+RC2+RC3 applied, with a populated
backend/.env present and no env blanking on the command line:
| Suite | Before | After |
|---|---|---|
tests/unit |
1 failed, 2881 passed | 2887 passed |
tests/integration |
12 failed, 459 passed, 1 skipped | 474 passed, 1 skipped |
Not in scope¶
- Converting
_stage_csv_under_files_storeusers to the fake-factory pattern (own issue — see RC2 follow-up). - Making PR CI run the integration suite. It is slow by design and split deliberately; the daily job is the right place. But the daily job did not surface RC3 loudly enough — worth a separate look at its alerting.
- Anything about
backend/.envcontents. The.envis correct for running the app; the bug is that tests could see it.