Lock the recalc step only¶
Follow-up to 2527 B1,
found while load-testing dev for #2572: eight csv_ingest uploads for
one unit, module and year showed two running and six not started, with
the two running ones parked on the module lock one behind the other.
What was wrong¶
csv_ingest_handler took the per-module lock exclusively before
_run_ingest. The comment justified it with the pre-import DELETE
cascading into data_entry_emissions, which only holds for
MODULE_PER_YEAR. A MODULE_UNIT_SPECIFIC upload is append-only: its
only delete is its own rows from a crashed attempt (#2700).
The lock could not simply go, because it was what kept the scoped recalc
dedup safe. EMISSION_RECALC_SCOPED_DEDUP collapsed a new child onto any
active sibling, including a RUNNING one. Without the ingest lock:
- upload A commits, chains recalc A, recalc A starts reading entries
- upload B commits, chains recalc B, deduped onto the RUNNING A
- B's rows never get emissions, nothing sweeps for entries without emissions, the total is silently short
What shipped¶
| Piece | Change |
|---|---|
app/tasks/_locks.py |
acquire_factor_recalc_lock(module_write=False): shared factor gate only, no module lock |
app/tasks/ingestion_tasks.py |
csv_ingest passes module_write=False for MODULE_UNIT_SPECIFIC; comment rewritten to say what each entity type needs and why |
app/models/data_ingestion.py |
ACTIVE_JOB_STATES, EMISSION_RECALC_SCOPED_ACTIVE_STATES = (NOT_STARTED, QUEUED), active_states_sql(); all three partial indexes render their WHERE from these tuples |
app/tasks/_chain.py |
DedupConfig.active_states; the pre-check SQL renders its state list from it, so index and pre-check cannot drift |
migration 7e026b9941cd |
drops and recreates uq_emission_recalc_active_scoped without RUNNING in its WHERE; the autogenerated trgm-index drop was a false positive and is pruned |
The resulting sequence: A and B insert concurrently. Recalc A runs. When B commits, its recalc is created because A is RUNNING, waits on the module lock behind A, then rewrites the scope and covers both. A third upload C that lands while B is still NOT_STARTED dedups onto B, which will read C's rows.
The unscoped dedup keeps RUNNING as a target: its writer
(factor_ingest) holds the exclusive gate until the recalc commits, so
a second factor upload cannot commit behind a running whole-slice
recalc.
Tests¶
tests/unit/tasks/test_scoped_dedup_ordering.py: scoped config and scoped index exclude RUNNING, unscoped config and index include it.tests/unit/tasks/test_ingestion_handlers.py: unit-specificcsv_ingestpassesmodule_write=False, per-year passesTrue.tests/integration/services/data_ingestion/test_emission_recalc_dedup_pg.py: with a RUNNING scoped sibling, chaining creates a new child; the existing NOT_STARTED test pins the other half.
Accepted trade-off¶
A crash-retried unit-specific ingest deletes its own rows from the previous attempt, and that cascade can queue on a running recalc's row locks. Rare and bounded, and the alternative is the head-to-tail behaviour this plan removes.
Verify on dev¶
Upload two files for the same unit and module a few seconds apart. Both jobs start immediately, the pipeline page shows two recalc children for the second pipeline only if the first recalc was already running, and both uploads' rows have emissions afterwards.