Remove primary_factor_id from DataEntry.data
Problem¶
DataEntry.data["primary_factor_id"] is a persisted denormalization of "which
factor matches this entry's classification". Persisting it forces an entire
invalidation apparatus, and that apparatus still leaks:
- Stale factors cannot be deleted. Rows missing from the latest CSV upload
are only flagged (
GET /v1/factors/stale,factors.py:29) because entry payloads may still hold their ids (factor_repo.py:151). - Reuploads must preserve factor ids.
upsert_factorskeys identity on(data_entry_type_id, year, emission_type_id, classification::text)exactly so existing ids survive (factor_repo.py:138). - Recalc must rematch. Plan 310B Part 6 + 310D built a bulk rematch layer
in
emission_recalculation.py:93-262whose only job is refreshing the stored id against current factors. - It still breaks. When a factor's classification shape changes (e.g.
building_rooms gained
energy_typeas a third classification key), the upsert cannot match old rows, both generations survive, andget_by_classification500s withMultipleResultsFoundon the first lookup that spans them. - The field is internal noise: the frontend never reads it, and report
exports explicitly scrub it (
carbon_report_module_repo.py:1238).
Decision¶
The entry's classification fields (kind_field, subkind_field,
kind_field_override code) are the sole source of truth. The matching factor
is derived state: resolved when needed, cached per request/slice, never
persisted on the entry.
DataEntryEmission.primary_factor_id (real FK column,
data_entry_emission.py:1082) is unchanged: it records the factor actually
used at computation time, is rewritten on every recalc, and backs the
results/rollup joins (data_entry_repo.py:539,591). Deliberately accepted
trade-off: dynamic resolution costs one bulk factor SELECT per
(data_entry_type, year) per request — robustness over peak performance.
Design — Phase 1: dynamic resolution¶
FactorResolver (new service)¶
Promote the Plan-310D lookup code out of emission_recalculation.py:105-165
into app/services/factor_resolver.py:
- One bulk SELECT per
(data_entry_type, year), memoized on the instance. - In-memory resolution mirroring today's semantics exactly:
(kind, subkind)match → kind-only fallback, and the override-key-first rule for handlers withkind_field_override(_lookup_factor_id/_lookup_factor_id_with_override). - Instance lifetime: one API request or one recalc slice. No cross-request cache, no invalidation problem.
Consumers switch to the resolver¶
- Emission compute —
prepare_createresolves the factor and injects it intoctx; Strategy Aresolve_computations(schemas/data_entry.py:356) keeps readingctxunchanged._get_building_energy_typereceives the resolvedFactorinstead of dereferencing an id (data_entry_emission_service.py:229). - Create/update — (as shipped) the create path drops factor resolution
entirely: nothing in the create flow consumed the factor, so
CarbonReportModuleWorkflow.createno longer calls the handler service (emission compute resolves on its own). The update path keeps a renamedresolve_factor_if_changedfor its side-effects (clear subkind/override on kind change, repopulate defaults) withresolve_factorreturningOptional[Factor]and never stamping. The old override-path guard that raised on a falsy kind moved to the validation layer:PurchaseHandlerUpdaterejects a provided-but-blank/nullpurchase_institutional_code(key-absent still means "not updating"). - Recalc — delete the rematch block (
emission_recalculation.py:192-262); resolution at compute time is the rematch. The prefetched dicts move into the resolver; the strict-drop contract is preserved (no factor match → emission recomputes to none → dashboard missing-factor signal). - List enrichment — primary path (join through the emission FK) is
unchanged; the JSON fallback (
data_entry_repo.py:788) resolves through the resolver instead, gated on the entry carrying a kind value, and tolerates ambiguous factor data per row (log + empty factor columns) instead of failing the whole list — display enrichment only; the compute/update paths keep surfacing ambiguity loudly. - CSV entry ingest — stops writing
primary_factor_idinto row payloads (base_csv_provider.py:1242-1280); keeps its "every row must match a factor" validation against the same in-memory map. - Cleanup — remove the export scrub (
carbon_report_module_repo.py:1238) and every remaining read/write ofdata["primary_factor_id"].
Data migration (added at v1.0, when "reseeds drop the DB" stopped being
true): revision 954eac6c95da strips the dead primary_factor_id keys
from data_entries.data and removes explicit-null
purchase_institutional_code keys (which the new update validator would
otherwise reject on every PATCH). Post-deploy operator step: re-upload the
current factor CSVs per module/year so the replace-semantics sweep deletes
stale factor generations that accumulated before this shipped.
Design — Phase 2: replace-semantics factor ingest¶
With no entry-side id references, stale factor rows lose their reason to exist (as shipped):
FactorRepository.delete_stale_for_year(year, *, det_ids, threshold_job_id)deletes rows in the covered scope whoselast_seen_job_idis NULL or predates the threshold. The threshold is passed explicitly by the ingest (its own job id) because mid-pipeline neither the running job (statenot yet FINISHED) nor the superseded one (is_currentalready flipped) is visible to a job-state lookup.- The factor CSV provider calls it right after a successful upsert, in the
same transaction, before the 310C recalc fan-out dispatches; the deleted
count lands in the job's
meta.stats.factors_deleted. - Destructive-semantics guards (explicit decisions): the sweep runs only
on full-SUCCESS uploads — a partial upload (WARNING) must not delete the
factors of rows it failed to re-ingest — and is scoped to the dets the
job actually upserted, so a partial multi-type module CSV can never wipe
sibling dets it didn't carry. Both are pinned by unit tests
(
test_module_per_year_factor_csv_provider.py). - The emission FK is
ondelete="CASCADE": affected emission rows vanish and the chained recalc rebuilds them. GET /v1/factors/stale,list_stale_for_year, and_latest_factor_job_per_detare removed — stale rows can no longer exist, and with them went the generic sweep mode and its per-det SQL CASE threshold (no UI ever consumed the endpoint).- The 2-key/3-key duplicate scenario (building_rooms
energy_typereshape) is pinned impossible by a regression test that first reproduces theMultipleResultsFound500, then shows the sweep killing it (test_factor_replace_semantics_pg.py).
Phase 2 ships as its own PR on top of Phase 1.
Testing¶
- Regression test for the originating bug: upload 2-key building factors,
reupload 3-key, assert the lookup returns only the new generation and the
/valuesendpoint no longer 500s (Phase 2 makes this pass by deletion). FactorResolverunit tests inherit the existing rematch-lookup coverage (kind/subkind fallback chain, override-key-first, ambiguity errors).- Strategy-B rematch integration tests repointed at the resolver path.
- Recalc integration: classification change on a factor reupload relinks entries through recompute alone (no stored-id refresh step).
Out of scope¶
DataEntryEmission.primary_factor_idcolumn and its consumers.get_by_classificationpublic signature (internals may gain the resolver).- Any frontend change beyond regenerated
openapi.d.ts.