MODULE_UNIT_SPECIFIC jobs must not auto-retry after a crash (#2700 Part 2)¶
Correction to the original #2700 scope¶
The plan that shipped Part 1 (2700-job-session-connection-release.md)
carried over #2689's framing of "one commit for the whole ingest — job
59, 144k rows, ~70 min." That's not what the code does.
_process_batch (base_csv_provider.py:1481-1535) already commits
data_session after every INGEST_COPY_BATCH_SIZE batch (default
50,000 rows, config.py:565), and has since aade48bf1b (2026-06-10) —
months before #2689. Verified directly: nothing between one batch's
commit and the next batch's COPY touches data_session — row
resolve/validate/enrich is pure Python, the only DB touch inside
_process_batch itself is a per-batch year-cache lookup that's a no-op
once warm. There is no connection held open mid-upload left to release;
that part of the original ask is already done by the existing batching.
What batching already exposed¶
Committing per batch means a crash between batches leaves durably committed partial data under a job that isn't FINISHED. Whether a later retry over that job is safe depends entirely on whether the handler's own inserts are idempotent — and that's asymmetric:
MODULE_PER_YEAR(bulk, nocarbon_report_module_idpinned):_delete_existing_entries_for_module_per_year(base_csv_provider.py:731-789) deletes the whole year/type slice before the row loop, on every attempt. A from-scratch retry after a partial commit is correct by construction — it deletes whatever the dead attempt left, then reinserts the complete file. Wasteful, not wrong.MODULE_UNIT_SPECIFIC(single-report upload,carbon_report_module_idpinned): append-only, no delete step, ever (base_csv_provider.py:746, confirmed in code — the delete branch's condition never matches this entity type).data_entrieshas exactly one DB uniqueness constraint,uq_member_role_per_module(models/data_entry.py:165-183), scoped todata_entry_type=memberonly. Every other type — purchases, equipment, travel, buildings, process emissions — has none. A retry after a partial commit re-COPYs every row the dead attempt already wrote, duplicated, silently.
The path that makes this reachable: sweep_stuck_running_jobs
(repositories/data_ingestion.py:1203-1279) resets any stale-RUNNING
job with attempts < max_attempts straight to NOT_STARTED for the
next claim_job to pick up — no entity-type awareness at all. A pod
crash (or, per the maintainer, DB-pool exhaustion under load — no
observed case of a plain crash) mid-batch-loop on a MODULE_UNIT_SPECIFIC
job is exactly this scenario.
What we're not doing, and why¶
Building an idempotent-resume mechanism for MODULE_UNIT_SPECIFIC
(content-based dedup keys, a persisted batch checkpoint, ON CONFLICT
handling per data-entry type) is real design work with its own risk —
manufacturing a natural key for rows that don't have one is exactly the
kind of thing #1559's clean destination-exists check can't generalize
to. Maintainer decision: don't build it. A human deletes the
partial rows and re-uploads. Raised to product ownership already;
accepted as correct for this spec. The engineering job here is making
sure that's what actually happens — loudly — instead of the system
quietly corrupting the report with duplicates.
Fix¶
1. sweep_stuck_running_jobs — MODULE_UNIT_SPECIFIC never lands in
the "recovered → NOT_STARTED" bucket, regardless of attempts remaining.
It always goes to "abandoned → FINISHED+ERROR", with a status_message
that says what happened and what to do:
"Job stopped mid-run (pod lost / crashed) and cannot be safely retried automatically — this upload has no duplicate protection. Verify data_entries for this report against the source file before re-uploading; delete any rows this job may have already written."
2. POST /jobs/{id}/recover (data_sync.py:2161-2225) — the manual
recovery path has the identical footgun (resets to NOT_STARTED,
attempts=0, same append-only handler runs again). Block it for
MODULE_UNIT_SPECIFIC with a 409 carrying the same explanation;
scope_row.entity_type is already fetched here for the permission
check, so this is one added branch, not a new query.
3. Leave claim_job untouched. Once a MODULE_UNIT_SPECIFIC job
can no longer reach NOT_STARTED via either path above, claim_job
never sees it again — nothing to guard there.
4. The "abandoned batches never get recalculated" gap (found during
Part 2 research) stays unfixed, deliberately. Today, once a
MODULE_UNIT_SPECIFIC job is abandoned, whatever it committed sits
un-recalculated forever. After this fix that's more likely to be the
outcome, not less — but it's the correct outcome: the maintainer's
resolution path is delete-then-re-upload, and a fresh re-upload chains
its own recalc over the (now clean) data. Building recalc-for-orphaned-
partial-data would be effort spent making duplicated data more usable,
backwards from the actual fix.
Regression tests¶
tests/unit/repositories/test_sweep_stuck_running_jobs.py (existing —
extend): a stale MODULE_UNIT_SPECIFIC job with attempts=0,
max_attempts=3 (plenty of retries left) must land in the abandoned
bucket, not recovered. Fails on current code (goes to NOT_STARTED).
tests/unit/api/test_recover_job.py (existing — extend, or new if it
doesn't cover entity_type): POST /jobs/{id}/recover on a stale
MODULE_UNIT_SPECIFIC job returns 409 with the duplicate-risk message,
not a reset job. Fails on current code (recovers normally).
Explicitly out of scope¶
- Batch-size tuning (
INGEST_COPY_BATCH_SIZE=50,000vs.310-overview.md's original 1k-5k recommendation) — no evidence either number is causing a problem; not touched here. - Any dedup/idempotent-insert mechanism for
MODULE_UNIT_SPECIFIC— explicitly rejected above. - The pre-row-loop setup phase (
_resolve_carbon_report_modules,base_csv_provider.py:899) usesflush()notcommit(), so it's folded into the first batch's transaction rather than committing on its own. Noticed while verifying the "anything left to relieve" question above; separate from this fix and not addressed here — flag for a future pass if aMODULE_PER_YEARfile spanning many not-yet-provisioned units turns out to make that phase slow enough to matter (it self-heals on retry either way, unlike the problem this plan fixes).