Skip to content

Lock the recalc step only

Follow-up to 2527 B1, found while load-testing dev for #2572: eight csv_ingest uploads for one unit, module and year showed two running and six not started, with the two running ones parked on the module lock one behind the other.

What was wrong

csv_ingest_handler took the per-module lock exclusively before _run_ingest. The comment justified it with the pre-import DELETE cascading into data_entry_emissions, which only holds for MODULE_PER_YEAR. A MODULE_UNIT_SPECIFIC upload is append-only: its only delete is its own rows from a crashed attempt (#2700).

The lock could not simply go, because it was what kept the scoped recalc dedup safe. EMISSION_RECALC_SCOPED_DEDUP collapsed a new child onto any active sibling, including a RUNNING one. Without the ingest lock:

  1. upload A commits, chains recalc A, recalc A starts reading entries
  2. upload B commits, chains recalc B, deduped onto the RUNNING A
  3. B's rows never get emissions, nothing sweeps for entries without emissions, the total is silently short

What shipped

Piece Change
app/tasks/_locks.py acquire_factor_recalc_lock(module_write=False): shared factor gate only, no module lock
app/tasks/ingestion_tasks.py csv_ingest passes module_write=False for MODULE_UNIT_SPECIFIC; comment rewritten to say what each entity type needs and why
app/models/data_ingestion.py ACTIVE_JOB_STATES, EMISSION_RECALC_SCOPED_ACTIVE_STATES = (NOT_STARTED, QUEUED), active_states_sql(); all three partial indexes render their WHERE from these tuples
app/tasks/_chain.py DedupConfig.active_states; the pre-check SQL renders its state list from it, so index and pre-check cannot drift
migration 7e026b9941cd drops and recreates uq_emission_recalc_active_scoped without RUNNING in its WHERE; the autogenerated trgm-index drop was a false positive and is pruned

The resulting sequence: A and B insert concurrently. Recalc A runs. When B commits, its recalc is created because A is RUNNING, waits on the module lock behind A, then rewrites the scope and covers both. A third upload C that lands while B is still NOT_STARTED dedups onto B, which will read C's rows.

The unscoped dedup keeps RUNNING as a target: its writer (factor_ingest) holds the exclusive gate until the recalc commits, so a second factor upload cannot commit behind a running whole-slice recalc.

Tests

  • tests/unit/tasks/test_scoped_dedup_ordering.py: scoped config and scoped index exclude RUNNING, unscoped config and index include it.
  • tests/unit/tasks/test_ingestion_handlers.py: unit-specific csv_ingest passes module_write=False, per-year passes True.
  • tests/integration/services/data_ingestion/test_emission_recalc_dedup_pg.py: with a RUNNING scoped sibling, chaining creates a new child; the existing NOT_STARTED test pins the other half.

Accepted trade-off

A crash-retried unit-specific ingest deletes its own rows from the previous attempt, and that cascade can queue on a running recalc's row locks. Rare and bounded, and the alternative is the head-to-tail behaviour this plan removes.

Verify on dev

Upload two files for the same unit and module a few seconds apart. Both jobs start immediately, the pipeline page shows two recalc children for the second pipeline only if the first recalc was already running, and both uploads' rows have emissions afterwards.