Clarify re-upload vs recalculate messaging on partial CSV ingestion¶
Problem¶
Headcount (and any module) CSV upload that stops partway through (issue: processed only 3000 rows) leaves the job in a state whose message tells the user to "recalculate" without saying the file needs to be re-uploaded. Recalculating only recomputes emissions for rows already committed to the DB — it cannot add the rows that were never ingested. Users read "recalculate" as the fix, click it, and the missing rows stay missing.
Root cause is shared, not headcount-specific: BaseCSVProvider (backend/app/services/data_ingestion/base_csv_provider.py), used by every module CSV importer including headcount, computes SUCCESS / WARNING / ERROR from rows_processed / rows_skipped in _compute_ingestion_result and writes a generic summary in _finalize_and_commit:
status_message = (
f"Processed {stats['rows_processed']} rows: "
f"{stats['rows_with_factors']} with factors, "
f"{stats['rows_without_factors']} without factors, "
f"{stats['rows_skipped']} skipped"
)
This message never mentions re-uploading. Separately, the frontend (UploadCard.vue + useUploadCard.ts) surfaces the job's status_message verbatim in the error/warning banner (getErrorDetails), and independently shows a "Recalculation Needed" badge with a "Retry recalculation" button (data_management_recalculation_needed, data_management_recalculate_retry in frontend/src/i18n/backoffice_data_management.ts) whenever recalculation status is stale. Both can be visible at once, and neither tells the user which action actually restores the missing data.
If the job dies mid-stream (exception/timeout before _finalize_and_commit runs), the terminal message is whatever the failure handler wrote — also not reviewed for "you must re-upload" wording.
Design¶
Two distinct situations, both currently collapse into the same vague UI:
- Partial ingestion (needs re-upload):
rows_skipped > 0because the job stopped early (crash/timeout) orrows_processed < total CSV rows. Recalculating cannot fix this — the source rows were never written. The message must say so explicitly and point at re-upload, not recalculate. - Recalculation genuinely sufficient: all rows were ingested (
rows_skipped == 0/ jobSUCCESS), but emissions are stale because factors or config changed. Here "Retry recalculation" is the correct and complete action — no re-upload needed.
Backend fix (single shared spot, covers headcount and every other CSV-backed module):
- In
_compute_ingestion_result/_finalize_and_commit, when result isWARNING(partial) or an early-terminationERROR, append an explicit instruction tostatus_message, e.g.:"... {rows_skipped} skipped. Re-upload the file to import the missing rows — recalculation alone will not add them." - Do the same in whatever failure/exception path sets the terminal job state before reaching
_finalize_and_commit(mid-stream crash/timeout), so a hard stop at row 3000 also gets a "re-upload" message instead of a bare exception string. - Only emit "recalculation is sufficient" phrasing (or say nothing extra) when the job actually completed with all rows processed.
Frontend fix:
UploadCard.vue/useUploadCard.tsalready rendersstatus_messagefor WARNING/ERROR jobs — no new component needed, the backend message change flows through automatically.- Where the "Recalculation Needed" badge/CTA can appear alongside a WARNING/ERROR data job (
hasErrorOrWarning(lastDataJob)), suppress or relabel the recalc CTA so it doesn't read as the sole fix — e.g. keep both visible but make the upload card's own re-upload button (data_management_reupload_data, already shown whenlastDataJobexists) the primary call-to-action, and gate "Retry recalculation" copy to not imply it replaces re-upload.
No backend API/schema change; status_message is a free-text field already piped to the frontend.
Steps¶
- In
base_csv_provider.py::_finalize_and_commit, extend the WARNING-pathstatus_messageto explicitly instruct re-upload whenrows_skipped > 0. - Locate the mid-stream failure handler that sets job state before
_finalize_and_commitruns (job crash/timeout path) and apply the same re-upload wording there. - Confirm
_compute_ingestion_resultclassification is untouched (SUCCESS stays silent — no re-upload prompt when all rows landed). - Frontend: verify
UploadCard.vue's error/warning banner renders the updatedstatus_messagewithout truncation. - Frontend: where
hasErrorOrWarning(lastDataJob)is true, ensure the "Recalculation Needed" / "Retry recalculation" CTA doesn't read as a standalone fix (copy tweak inbackoffice_data_management.ts, or conditional ordering so re-upload is primary). - Add/update i18n strings (en/fr) for the new backend-driven wording if any static frontend copy changes.
- Regression test: CSV ingestion that stops partway (simulate
rows_skipped > 0/ mid-stream failure) assertsstatus_messagecontains re-upload instruction, distinct from the all-rows-succeeded case.