Delivered. DataIngestionProvider._move_to_processing /
_move_to_processed (backend/app/services/data_ingestion/base_provider.py:64-108)
implement the idempotency check exactly as designed below, with regression
coverage in backend/tests/unit/services/data_ingestion/test_base_provider.py.
Found still-current while investigating
#2220 — this
plan's own frontmatter was stale (said "proposed"), which is corrected here
per the guardrails' "keep plans aligned with shipped code."
Idempotent tmp->processing (and processing->processed) file moves on job retry¶
Problem¶
Symptom. After a backend restart, CSV ingestion intermittently fails with:
Processing failed: Failed to move file from tmp/<ts>/headcount_data.csv to processing/<job_id>/headcount_data.csv
Root cause: the tmp -> processing move is not idempotent on retry.
- The move consumes the source.
BaseCSVProvider._setup_and_validate(backend/app/services/data_ingestion/base_csv_provider.py:1113-1116, confirmed still current) callsfiles_store.move_file(tmp_path, processing_path)(ashutil.move) and raises if it returns falsy. move_filereturnsFalseonly when the source is missing. Inenacit4r_files/services/local.py:308, a missing source raisesFileNotFoundErrorinternally, caught and converted toreturn False. So the error means the tmp source is already gone, not a permissions/disk problem.- A retry finds the source already consumed — the first attempt already
shutil.moved it intoprocessing/<job_id>/. The move is a one-shot consume with no "already done" branch. - A restart triggers the retry.
sweep_stuck_running_jobs(backend/app/repositories/data_ingestion.py:1135) resets staleRUNNINGjobs toNOT_STARTEDsoclaim_jobre-dispatches them. Its own docstring flags a no-heartbeat duplicate-run hazard (a long job pastSTALE_JOB_TIMEOUT_MINUTES, or a crash after the move). So: restart after the move → job re-queued →_setup_and_validateruns again → move fails on the already-moved file.
This explains "sometimes": it only fails when the crash/restart lands after the move but before the job is marked FINISHED. Data is never lost — it sits safely in processing/<job_id>/; the retry just refuses to recognize it.
Affected code (same pattern duplicated across all four DataIngestionProvider subclasses):
| Move | File | Line (verified 2026-07-07) |
|---|---|---|
| tmp->processing | base_csv_provider.py |
1113-1116 |
| tmp->processing | base_factor_csv_provider.py |
260-265 |
| tmp->processing | base_reduction_objective_csv_provider.py |
261-270 |
| tmp->processing | csv_providers/reference_data.py |
228-230 |
The later processing -> processed moves share the same retry hazard:
base_csv_provider.py:1378(issue cited 1452; drifted, still present)base_factor_csv_provider.py:552base_reduction_objective_csv_provider.py:448csv_providers/reference_data.py:247
DataIngestionProvider (shared ABC, backend/app/services/data_ingestion/base_provider.py:24) is the natural home for a centralized helper. files_store.file_exists exists on the same service as move_file (enacit4r_files/services/local.py:253), so the idempotency check needs no new dependency.
Design¶
Make each move idempotent: if the destination already exists, a prior attempt already did the move — skip it instead of failing.
filename = tmp_path.split("/")[-1]
processing_path = f"processing/{self.job_id}/{filename}"
if await self.files_store.file_exists(processing_path):
logger.info(f"File already at {processing_path} (prior attempt); skipping move")
else:
if not await self.files_store.move_file(tmp_path, processing_path):
raise Exception(f"Failed to move file from {tmp_path} to {processing_path}")
Centralize as two helpers on DataIngestionProvider (base_provider.py) rather than patching four call sites independently (DRY, and it forecloses a fifth copy of the bug landing in a future provider):
_move_to_processing(tmp_path: str) -> str— tmp->processing, returnsprocessing_path._move_to_processed(processing_path: str) -> str— processing->processed, returnsprocessed_path.
Both check destination existence before calling move_file, log-and-skip when already present, and raise only when the move is actually attempted and fails. Call sites (all four providers, both move points each) replace their inline move_file + raise blocks with a call to the shared helper.
Steps¶
- Add
_move_to_processing(tmp_path) -> strand_move_to_processed(processing_path) -> strtoDataIngestionProviderinbackend/app/services/data_ingestion/base_provider.py, each guarded by afile_existscheck on the destination before callingmove_file. - Replace the inline tmp->processing move in
base_csv_provider.py:1113-1116withself._move_to_processing(tmp_path). - Replace the inline tmp->processing move in
base_factor_csv_provider.py:260-265with the same helper call. - Replace the inline tmp->processing move in
base_reduction_objective_csv_provider.py:261-270with the same helper call. - Replace the inline tmp->processing move in
csv_providers/reference_data.py:228-230with the same helper call. - Replace the four processing->processed moves (
base_csv_provider.py:1378,base_factor_csv_provider.py:552,base_reduction_objective_csv_provider.py:448,csv_providers/reference_data.py:247) withself._move_to_processed(processing_path). - Add a regression test: place the file in
processing/<job_id>/(simulating a prior attempt) with no source intmp/, run the provider's setup, and assert it succeeds (reads fromprocessing/) instead of raising "Failed to move file". Cover at least one provider directly and rely on the shared helper for the rest. - Out of scope: the deeper
STALE_JOB_TIMEOUT_MINUTES/ no-heartbeat duplicate-run issue (Plan 310C per thesweep_stuck_running_jobsdocstring) is the underlying trigger for duplicate runs. The idempotent move fixes this symptom directly without touching the job runner; leave 310C as a separate, later plan.