Concurrent CSV upload move failures (#2528)¶
Goal: no Error moving file reaches a user — the move retries a
transient failure instead of making the job terminal on the first one,
and the app's dead falsy-return branch (fact 4, a leftover from before
enacit4r-files 1.4.0 aligned both backends on raising) is removed
rather than left as unreachable code.
Found by the #2295 load suite (PR #2526). Read
2295-load-tests-locust.md for the
harness and 1559-ingestion-idempotent-tmp-to-processing-move.md
for the existing idempotency guard this plan builds on.
Correction (2026-09-02): the 56% figure is void¶
The maintainer's own follow-up (issue #2528 comment, 2026-08-30T11:31Z)
pulled the actual exception out of data_ingestion_jobs.meta.status_history
for the dev-DB run:
An error occurred (NoSuchKey) when calling the CopyObject operation:
The specified key does not exist.
NoSuchKey is not a transient fault — the source object was never in
the bucket, because that run's backend/.env had S3_BUCKET commented
out (FILES_STORAGE_PATH=./files_storage active), so the local backend
process wrote every temp-upload to its own laptop disk while
RUN_BACKGROUND_POLLER=False handed the ingest jobs to the dev pods,
which correctly looked in S3 and found nothing. The 56% measured a split
file store, not concurrency, and retrying the move would not have
helped — retrying a copy of a key that does not exist just fails N
times.
What this voids: the 56% number itself, the "70× local/S3 gap" framing in H3 below, and Open Questions 1–2 as originally posed (they existed to size a rate that turns out not to exist).
What survives, unaffected by the topology bug: every finding below
came from reading the code, not from the 56% run, and none of it depends
on that number — facts 1, 3, 4, 5, 6, 7, and H1. The local-store run's
3/378 failures (0.8%) are the only measurement left standing, because
NoSuchKey-by-split-store cannot arise when both the upload and the job
run against the same local disk. That residue is real and still
unexplained — see "The one open measurement" below.
Real measurement (2026-09-02): clean topology, real S3, zero move failures¶
Ran the actual reproduction the Correction called for: one process serving
both temp-upload and the job (poller off, inline dispatch — the #2220
boot guard enforced this), real S3 (s3.epfl.ch), against the dev DB.
make perf-load PERF_CLASSES=CsvUploadUser, 5 users then 20 users, 10
minutes each.
| Run | Flows | Error moving file / S3Error / FilesStoreError |
|---|---|---|
| 5 users | 104 | 0 |
| 20 users | 100 | 0 |
Verified two ways: grepped the full backend log (both runs) for those
three strings — zero hits — and independently queried every result =
'ERROR' job's meta.status_history in the DB for the run windows. Every
failure in both runs (37 at 5 users, ~60 at 20 users) was a CSV-content
mismatch against this dev DB's current seed — No factors available for
module=X year=Y, a location/room lookup miss, a duplicate institutional
ID — none were a move failure. This is a gap in the generated perf-test
fixtures (make perf-csvs output vs. this DB's factor/location seed),
not a product bug; it doesn't need fixing for this issue.
This closes the plan's open measurement. At both 5 and 20 concurrent
uploaders, against real S3, with the topology bug from the Correction
ruled out, the tmp → processing move did not fail once — better than
the 0.8% local-store baseline this plan set as the target. H1 (fact 3,
no retry) is still a real defect worth fixing on its own merits — an
unretried move is one transient S3 fault away from failing a job — but
H2 (client leak) and H3 (throttling) have now been tested, not just
theorized, and neither produced a single failure at this scale. Open
Questions 1-3 are resolved below.
Found, and explicitly out of scope for #2528: at 20 users, 67% of
flows failed for a completely different reason —
psycopg.errors.ProtocolViolation: query_wait_timeout on plain SELECT
queries (e.g. the users login lookup), consistent with PgBouncer's
well-known default query_wait_timeout (120s) — several endpoints
independently clustered at almost exactly 119-120s, and the FLOW metric's
484s worst case looks like a few of a flow's sequential steps each
separately hitting that wall. This is dev-DB connection-pool contention,
unrelated to file storage; it is not analyzed further in this plan. Given
the timing (PgBouncer was put in front of the dev DB the day before this
run) and #2566's own postmortem calling out a stray local test process as
a contributing factor there, this run's own connection footprint (2
workers × 15 connections) may well have been sufficient on its own to
trigger it — that hasn't been established either way. The local backend
used for this measurement was stopped immediately once the contention
appeared, to avoid compounding it.
The path, traced¶
One CSV upload flow crosses these:
| Step | Code | S3 calls |
|---|---|---|
POST /v1/files/temp-upload |
backend/app/api/v1/files.py → files_store.write_file into tmp/<ts>/ |
put_object + metadata sidecar |
POST /v1/sync/dispatch |
job row, then run_job under the MAX_CONCURRENT_JOBS=4 semaphore |
— |
| move | DataIngestionProvider._move_to_processing (backend/app/services/data_ingestion/base_provider.py) |
head_object, copy_object, delete_object, + sidecar |
| read | files_store.get_file(processing_path) |
get_object |
| archive | _move_to_processed |
head_object, copy_object, delete_object, + sidecar |
The move itself lands in enacit4r-files 1.4.0 (enacit4r_files/services/s3.py),
pinned by tag in backend/pyproject.toml.
Evidence: what is proven from the code¶
-
The reported message is the dependency's, character for character.
S3FilesStore.move_filewraps its body intry/except Exceptionand doeslogging.exception(f"Error moving file from {source_path} to {destination_path}"), thenraise S3Error(f"Error moving file from … to …: {e}") from e. So the failure is an exception raised insideS3Service.move_file— i.e. incopy_objectordelete_object. It is not a falsy return, and it is not the metadata sidecar (that is caught separately and only ever logs a warning). -
The move is copy + delete, not a rename.
S3Service.move_filecallscopy_file()(copy_object) and, only if that returned non-False,delete_file()(delete_object) — two network round trips, non-atomic.LocalFilesStore.move_fileisshutil.move, a same-filesystem rename with no network at all. That asymmetry is why the S3 path can fail on a transient network fault where the local path structurally cannot — it is a mechanism, not (per the Correction above) a measured gap. -
The app never retries the move.
_move_to_processingcalls it once. The exception propagates through_setup_and_validatetoprocess_csv_in_batches, whose handler writesstate=FINISHED, result=ERRORitself. Because the provider makes the job terminal, the runner'sattempts < max_attemptsretry never engages — which is exactly why every failed job showsattempts = 1. That is not evidence against a retry bug; it is evidence there is no retry at all. -
Update (2026-09-02): the contract drift is already fixed upstream — the app just hasn't caught up.
backend/pyproject.tomlnow pinsenacit4r-files1.4.0 (bumped from the 1.2.0 this plan was written against). Verified straight from the installed package (.venv/lib/python3.14/site-packages/enacit4r_files/services/):
| Store | On failure at 1.2.0 (plan's original basis) | On failure at 1.4.0 (now pinned) |
|---|---|---|
S3FilesStore.move_file |
raises S3Error |
still raises S3Error |
LocalFilesStore.move_file |
except Exception: logging.error(…); return False |
now raises FilesStoreError, chained; docstring: "Matches S3FilesStore.move_file (#24) so a caller can handle both backends with one except" |
S3Error(FilesStoreError) — one except FilesStoreError catches
both. Neither backend returns False on failure any more. That
makes _move_to_processing's if not await self.files_store.move_file(...)
fully dead on both backends, not just S3 — a failing move raises
straight out of that if's own await, so the
raise Exception("Failed to move file from …") line and
_diagnose_move_failure never execute in practice today. The raw
dependency exception is what actually propagates, matching fact 1.
This resolves Open Question 5 below: the upstream tag bump is
itself the alignment — nothing further needs opening in
enacit4r-files. What's left is app-side catch-up:
_diagnose_move_failure's docstring and test_base_provider.py:111
still describe a backend that returns bare False, which is true of
neither store now, and the falsy-return branch is dead weight rather
than a defense. Step 1 is revised accordingly: catch
FilesStoreError and retry — there is no falsy-return path left to
handle alongside it.
-
One S3 client per job, never closed.
make_files_store()builds a freshS3Service;base_csv_provider,base_factor_csv_provider,base_reduction_objective_csv_providerandcsv_providers/reference_dataeach call it lazily per provider instance, so every ingestion job constructs its own client. 1.4.0 keeps that client alive for the life of theS3Service(its docstring says "Call close() on application shutdown") and nothing in the app ever callsclose(). Every finished job leaves an aiobotocore client and its aiohttp connector behind. -
Pool and retries are at their defaults.
S3Service.__init__takesmax_pool_connections: int = 10andmake_files_store()does not pass it. ItsConfigsets noretries=, and neither the repo nor the chart setsAWS_RETRY_MODE/AWS_MAX_ATTEMPTS, so botocore'slegacymode applies (pinned: aiobotocore 2.12.4, botocore 1.34.106). -
The user cannot recover without re-uploading.
copy_objectfailing meansdelete_objectnever runs, so the CSV is still intmp/<ts>/— no data loss, but nothing GCstmp/.POST /v1/sync/jobs/{id}/recoveronly accepts stale RUNNING rows (stale_running_clause), so a job that went FINISHED/ERROR here is terminal. Re-upload + re-dispatch is the only path, and the orphan stays.
Root-cause hypotheses, ranked¶
H1 — the move has no retry, so any transient S3 fault is fatal. Proven (fact 3), and sufficient on its own to explain the shape: fails at 5 uploaders, worse at 20, near-zero on a store with no network. This is the one certain defect, and it is worth fixing whatever turns out to be underneath it.
H2 — client accumulation from never calling close().
Mechanism proven (fact 5) and confirmed firing — 373 "Unclosed client
session" / "Unclosed connector" warnings logged across the 2026-09-02
runs (see "Real measurement" above) — but tested at magnitude and not
observed to cause a single failure at 5 or 20 concurrent uploaders over
10 minutes each. At MAX_CONCURRENT_JOBS=4 × 2–3 replicas and ~18
flows/min in production the accumulation is bounded, and aiohttp closes
transports on GC; this run didn't run long or hard enough to say whether
a much longer sustained load changes that.
H3 — endpoint-side throttling or connection caps on the S3-compatible
store. Tested, not observed. 20 concurrent uploaders against
s3.epfl.ch produced zero SlowDown, ClientConnectorError, or
ConnectTimeoutError (see "Real measurement" above). Stays a plausible
mechanism at higher concurrency than 20 — which the original issue never
claimed — but there is no evidence for it at the scale this issue is
actually about.
H4 — tmp/ path collision. Weak, confirmed weak by code.
folder_name is computed once per POST /temp-upload request
(backend/app/api/v1/files.py:358-360) as
str(datetime.datetime.now(UTC).timestamp()).replace(".", "") —
Python's datetime.now() carries microsecond resolution, so two
requests would need to land in the same microsecond and pick the same
filename. CsvUploadUser draws from a handful of fixed filenames, so
the collision requires both conditions at once. At ~182 flows over 10
minutes the odds are tiny, and the symptom would be an occasional
"source no longer exists", not a large fraction of flows.
H5 — a retry re-moving an already-moved file. Falsified in the
issue by attempts = 1, and independently by #1559's destination-exists
check already at the top of _move_to_processing.
The one open measurement¶
The only real signal left after the Correction is the local-store run's
3/378 failures (0.8%) — on a store where the split-topology
NoSuchKey cannot arise, so these are genuine tmp → processing
failures. LocalFilesStore.move_file returns False rather than
raising, so these three went through the falsy-return branch and
_diagnose_move_failure (fact 4) — that diagnosis string, if the job
rows still exist, would say more than the code alone can.
Update (2026-09-02): intranet/DB access is available now (it wasn't
when this was first written). Checked: zero data_ingestion_jobs rows
with moving file in status_message in the dev DB for 2026-08-29 to
2026-08-31, the window of that local run. Either it ran against a
different DB (the reporter's own laptop, not the shared dev DB the S3
run used) or those rows have since rotated out — either way, this
specific residual is unrecoverable from here. It stays open in the
sense that it was never explained, but it is no longer actionable
without the original run's own local database, and shouldn't block
closing anything else in this plan.
logging.exception also wrote a traceback next to every Error moving
file line in the void 56% run, but that traceback is no longer useful:
the Correction already identifies the exception (NoSuchKey) and its
cause (split topology), so pulling dev-pod logs for that window would
just confirm what is already known. Do not re-run this kubectl
command (kept here only so nobody re-derives it independently):
kubectl --context svc1751d-co2-calculator-dev/… -n svc1751d-co2-calculator-dev \
logs -l app.kubernetes.io/name=co2-calculator-backend --since=48h --tail=-1 \
| grep -A 20 'Error moving file'
The fix¶
Scope decision (maintainer, PR #2535, 2026-08-30T11:49Z): "Fix the
enacit4r-files contract drift app-side now, and upstream too... Keep
the plan; drop the priority until a clean measurement exists." The
upstream half is done (fact 4, revised — 1.4.0). Read against the
Correction above, the app-side half means: Step 1 ships now — catch
the now-shared exception, remove the dead falsy-return branch, retry —
correct regardless of what any concurrency measurement eventually shows.
Steps 2 and 3 are deferred — both were sized and ordered against the
void 56% run (pool sizing against an assumed connection-exhaustion rate,
"ship Step 3 first" against an assumed throttling signal); revisit them
once a clean-topology measurement exists to size against.
Step 1 — bounded idempotent retry at the single choke point (ship now)¶
DataIngestionProvider._move_to_processing, in base_provider.py. All
four CSV providers route through it, so this is one guard instead of
four.
- Both backends raise on failure at 1.4.0 (fact 4, revised) —
S3ErrororFilesStoreError, andS3Error(FilesStoreError), so oneexcept FilesStoreErrorcatches both. Delete the falsy-return branch and itsif not await move_file(...)check — it is dead code on both stores now, not a defense to preserve. Wrap the call intry/except FilesStoreErrorinstead._diagnose_move_failuremoves into thatexceptblock: it still adds detail ("source already gone" vs "still present") that the raised exception's message alone may not spell out, so it survives as a diagnostic on the exception path, not as the trigger for detecting failure. - Catch
FilesStoreErrorfrom the destination probe too:path_existscorrectly raises on anything that is not a 404 (a permission or transport fault is not "the file is absent"), so a transport fault can surface on either call. - Re-check
file_exists(processing_path)at the top of every attempt. That check already exists from #1559 and is what makes the retry idempotent — a copy that succeeded before the connection dropped is detected, not redone. - ~4 attempts, exponential backoff with jitter (0.25 / 0.5 / 1.0 s), then raise. No swallow, no fallback, no "misc" outcome. Zero added latency on the happy path.
_move_to_processedreuses the same helper but keeps its non-raising caller: the data is already committed there, only archival bookkeeping is at stake, and that is a deliberate, documented asymmetry.
Keep the helper under 40 lines and 2 nesting levels; imports at top.
Scope this brings with it (so the estimate is honest): the two stale
docstrings from fact 4, plus the existing tests that pin the old
single-shot, falsy-return behaviour and will need rewriting — not just
extending, and not just re-pointed at a retry — since the scenario they
mock (move_file returning False) can no longer happen on either
backend at 1.4.0 and must become side_effect=FilesStoreError(...):
test_base_provider.py (lines 103, 122, 134, 183, 198, all
move_file = AsyncMock(return_value=False)),
test_base_factor_csv_provider.py::test_finalize_and_commit_move_file_failure,
and test_base_csv_provider.py around line 1418. They must keep
asserting that an exhausted retry still fails loudly.
Step 2 — one S3Service per process, with a sized pool (atomic, deferred)¶
- Replace the four per-provider
make_files_store()calls with the process-wide singleton already inapp/api/v1/files.py. - Add
S3_MAX_POOL_CONNECTIONStoSettings(default 32 ≈MAX_CONCURRENT_JOBS × 8, sized to cover concurrent jobs plus concurrent uploads on one pod) and pass it toS3Service. - Close it in the FastAPI lifespan shutdown.
These ship together, not separately. Consolidating onto the singleton without raising the pool would move job traffic onto the same 10-connection pool the upload endpoint uses — creating contention on a path that currently works.
Two wrinkles to settle in review:
close()exists onS3Serviceonly, not onFilesStore/LocalFilesStore, so the lifespan hook needs anisinstance(files_store, S3FilesStore)discriminator. The cleaner alternative is addingasync def close()to theFilesStorebase upstream inenacit4r-filesand bumping the tag. Maintainer picks.- The three providers use inline
from app.api.v1.files import make_files_storeimports to dodge a circular import (against the imports-at-top rule). Importing the singleton has the same problem, so either keep the inline import or movemake_files_storeout ofapi/v1/files.pyinto a service module. Prefer the move.
Step 3 — botocore retry tuning, zero code (deferred)¶
enacit4r-files documents AWS_RETRY_MODE / AWS_MAX_ATTEMPTS as the
supported knobs. Set AWS_RETRY_MODE=standard and AWS_MAX_ATTEMPTS=5
on the backend and worker deployments in the chart. standard mode
retries a broader, better-defined set (throttling including SlowDown,
5xx, connection errors) than the legacy default. Cheap and a one-line
revert, but there is no longer a traceback or a rate to justify shipping
it ahead of a measurement — revisit alongside Step 2.
Explicitly out of scope¶
Park as follow-up issues unless the maintainer says otherwise: tmp/
orphan GC; making a move-failed job recoverable without re-upload;
replacing the timestamp folder name with a uuid4.
Regression tests¶
Deterministic, in CI — backend/tests/unit/services/data_ingestion/test_base_provider.py,
mirroring the #1559 tests already there. Patch asyncio.sleep so the
backoff costs nothing.
| Test | Setup | Fails today because |
|---|---|---|
…retries_transient_storage_error |
file_exists → False; move_file side_effect=[S3Error(…), True] |
the first exception propagates; there is no retry |
…retries_local_store_error |
move_file side_effect=[FilesStoreError(…), True] |
same as above via the base exception class — guards against an except S3Error that misses LocalFilesStore's FilesStoreError |
…raises_after_exhausting_retries |
move_file always raises |
(guards the fix: the error must still escape — no silent fallback — and the attempt count must be bounded) |
…rechecks_destination_between_attempts |
move_file raises once, then file_exists → True |
asserts move_file awaited exactly once: idempotency must hold across retries, not just across job attempts |
…retries_when_existence_probe_fails |
file_exists side_effect=[S3Error(…), False] |
a probe fault must be retried, never misread as "destination absent" |
…providers_share_one_files_store |
assert provider.files_store is the process singleton |
each provider builds its own S3Service today |
Acceptance gate for Step 1 is the six unit tests above (CI, deterministic). The load-test acceptance gate below is for Steps 2/3, if they're ever un-deferred — Step 1 doesn't need it, it's correct independent of any rate:
make perf-load PERF_CLASSES=CsvUploadUser PERF_USERS=20 PERF_TIME=10m
Already run clean, 2026-09-02 — see "Real measurement" above: 5 and
20 users, real S3, topology fixed (poller off, inline dispatch,
S3_BUCKET uncommented). Zero Error moving file at either stage,
comfortably under the local-store baseline. Re-run only if Steps 2/3 are
revived and need a fresh before/after comparison — start with the 5-user
stage as a cheap smoke, and watch for the dev-DB connection-pool
contention noted in "Real measurement" before pushing past 20 users.
Rollout and verification¶
- Ship Step 1 now (contract-drift fix, maintainer-approved). CI gate only — no perf-load run needed to ship this, since it is correct independent of any concurrency rate.
- ~~Open the upstream
enacit4r-filesPR~~ Done — the pin is at 1.4.0 and the two backends'move_filecontracts already agree (Open Question 5). - ~~Get a clean concurrency measurement~~ Done, 2026-09-02 — see "Real measurement" above. Zero move failures at 5 and 20 users on real S3. Steps 2/3 stay deferred; there is no measured problem for them to fix.
- Steps 2 + 3 only get revived if a future run — at higher concurrency,
or in production — actually shows
Error moving file/S3Error. If that happens, ship them together (sized against each other — see the "ship together" note in Step 2) and watch, before and after: that error count,Unclosed client sessionwarnings (373 of these did fire in the 2026-09-02 run — fact 5's leak is real, just not yet causing failures at this scale), pod fd count, event-loop lag (app/tasks/_event_loop_lag.py), and GlitchTip forS3Error. - Sweep
tmp/for orphans left by any failed runs (including the void 56% run and the 3 local failures) and record the count in the issue — it sizes the GC follow-up (Open Question 6).
Open questions¶
- ~~The traceback~~ Answered, and void. The maintainer pulled it
from
meta.status_history:NoSuchKeyonCopyObject, caused by the split file store, not a transient fault (see Correction). No dev-pod log fetch needed or wanted for this window. - ~~Flat vs climbing~~ Answered by measurement. A clean run at both 5 and 20 users produced zero move failures to characterize — see "Real measurement" above. Neither H2 nor H3 manifested at this scale.
- Answered by measurement, for now:
s3.epfl.chhandled 20 concurrent uploaders (104-111 requests/run) with zero throttling or connection-cap symptoms — noSlowDown, noClientConnectorError.S3_MAX_POOL_CONNECTIONSsizing stays moot until a run at a higher concurrency than 20 (which the original issue never claimed) shows otherwise. close()on theFilesStorebase upstream (cleaner, needs a tag bump), or anisinstancediscriminator in the lifespan (no dependency change)? Only matters once Step 2 is un-deferred.- Done. The maintainer's PR #2535 comment asked for the divergent
move_filecontract to be aligned upstream, and it has been: the pin moved toenacit4r-files1.4.0, whereLocalFilesStore.move_filenow raisesFilesStoreErrorto matchS3FilesStore.move_file'sS3Error(fact 4, revised above). Nothing further to open upstream — Step 1 just needs to catch the now-shared base exception. - Do the
tmp/GC and "recover a move-failed job without re-upload" ship here, or as follow-up issues? Still open. Also still open: the 3/378 local-store failures (0.8%) have no diagnosis yet — see "The one open measurement" above — so #2528 should stay open even after Step 1 ships, not be closed as resolved.