recompute-stats never reached Simulator Plan reports (#2775)¶
Context¶
POST /v1/sync/admin/recompute-stats is the admin backfill trigger: it fans
out one root aggregation job per (module_type_id, year) scope so every
carbon_report_module.stats row gets re-derived under current code after a
stats-shape change. Running it for 2025 on dev dispatched 8 jobs (one per
module type, jobs 242-249) and none of them touched a Simulator Plan module.
A Plan report's carbon_reports.year is its planning target; its entries
price against reference_year. That is exactly what resolve_factor_year
reads first (app/utils/factor_year.py), and the plan year is deliberately a
year no factor set exists for — on dev all 84 Simulator_Plan reports carry
reference_year = 2025 with year spanning 2020-2043.
Every scoping query in the recompute path keyed on carbon_reports.year, so a
plan was stranded three times over:
list_module_type_year_scopesfilteredCarbonReport.year == yearand emittedCarbonReport.yearas the scope key, so picking 2025 in the operator UI never produced a scope containing a 2043 plan report.filter_scopes_with_current_factorsthen asked "does 2043 have a current FACTORS job?" → no → counted inskipped_no_factors.list_by_module_type_and_year— the aggregation handler's own module selection — filtered the same way, so even a dispatched job would not have collected the plan's modules.
Only the seven plan reports that coincidentally sit at year = 2025 were ever
recomputed, which is why the symptom read as "it only recomputes Calculator".
The same seed query backs find_stale_aggregations, so the stale-stats
endpoint had the mirror-image bug: a plan at 2043 raised a scope no
aggregation can ever target, reported as no_aggregation_ever forever.
What shipped¶
A single SQL expression, effective_factor_year_col() in
app/utils/factor_year.py, next to the Python resolve_factor_year it is the
twin of — so the two answers to "which year does this report price against"
cannot drift:
func.coalesce(col(CarbonReport.reference_year), col(CarbonReport.year))
Applied at the three scoping sites:
| Site | Was | Now |
|---|---|---|
data_ingestion.list_module_type_year_scopes |
CarbonReport.year (select + filter) |
effective_factor_year_col() |
data_ingestion.find_stale_aggregations (seed subquery) |
CarbonReport.year |
effective_factor_year_col() |
carbon_report_module_repo.list_by_module_type_and_year |
CarbonReport.year == year |
effective_factor_year_col() == year, opt-in via include_reference_year |
CarbonReport.reference_year is written in exactly one place —
SimulatorPlanService.set_reference_year, a Simulator Plan route — so the
COALESCE is a no-op for Calculator and Explore reports.
The module slice is widened for the admin trigger only¶
list_by_module_type_and_year backs both aggregation paths: the admin
trigger and the recalc-chained aggregation that runs after every ingest or
factor upload. Widening it unconditionally would have changed the hot path:
emptied = await svc.emptied_module_ids(candidates)
affected = [m for m in candidates if m.id in affected_scope or m.id in emptied]
emptied_module_ids runs over the full candidate list and is not filtered by
affected_scope, and the recalc path recomputes with bump_status=True. So a
routine 2025 factor upload could have flipped validated plan modules back to
IN_PROGRESS — a change to validated data on the pipeline's hot path, which is
not what this issue is for.
The widening is therefore opt-in: create_root_aggregation_job sets
meta.config.include_reference_year_reports, and aggregation_handler passes
it through to the slice query. Recalc-chained aggregations don't set it and
keep exactly their current candidate set.
Open question for review: a 2025 factor change does invalidate plans baselined on 2025, so arguably the recalc path should reach them too. That is a deliberate follow-up, not an oversight — it touches validated emission data and needs its own plan.
Why this shape¶
Plans fold into the existing (module_type_id, 2025) scope rather than
raising one of their own. That means no new scopes, no new jobs, and no extra
pooled DB connections — the 8 jobs an operator dispatches today stay 8 jobs.
It also dissolves the factor filter's objection without touching it: the scope
year is now 2025, which has factors.
The operator UI needs no change either. Its year dropdown is restricted to
years with a year-configuration row (RecomputeStatsCard.vue), so a plan year
like 2043 could never have been selected; picking the baseline year now sweeps
in every plan baselined on it.
Deliberately out of scope¶
- Simulator Explore — decided, won't fix. See below.
- Re-pricing.
recompute_stats_manyre-aggregates existingdata_entry_emissionsrows into the stats JSON; it does not recompute emissions. This change makes the stats backfill reach plans, which is what the endpoint is for. Whether a factor change re-prices plan entries isemission_recalc's concern and was not investigated.
Decision: Explore stays out of the backfill¶
Explore has the same symptom as Plan — its scope year is decoupled from its factor year — but not the same fix, and we are choosing not to fix it.
create_explore sets year = now.year and reference_year = None, while the
factor year resolves to N-1 via _resolve_latest_started_year. Confirmed on
stage: a sandbox created 2026-09-14 returns year: 2026, factor_year: 2025,
and its (module_type, 2026) scope has no current FACTORS job, so the trigger
skips it.
Plan was fixable in ~8 lines because its factor year is a stored column
(reference_year — user input, not a derived value) that the scope query can
read. Explore's is derived from today's date plus users.provider and
year_configurations, with nothing stored to key on. Reproducing it on the
scope path means a CASE plus extra joins in two places (the scope listing
and the handler's module slice) — no single-site version exists.
Two options were rejected:
| Why not | |
|---|---|
Persist the factor year (a column, or reusing reference_year) |
Smallest diff, but Explore's factor year is N-1 of today — genuinely dynamic, so a stored copy is wrong after the year rollover. Exactly what "don't store derived values when they resolve from factors/lookups" exists to prevent, and it reverses #2656. |
| Exempt Explore scopes from the factor filter | A scope like (module_type, 2026) holds Explore's 9 modules and thousands of factorless Calculator 2026 modules. Admitting it dispatches a job that recomputes all of them to zeros — precisely what the filter prevents. |
Why doing nothing is correct here: Explore sandboxes are recreated on every
"start exploration" with the previous ones deleted in the background (#2656),
and their modules carry NULL stats until someone edits them
(bulk_create_carbon_report_modules_of_carbon_report leaves stats unset). A
stats-shape change therefore self-heals on the next exploration — there is no
durable wrong state for a backfill to repair. The only exposure is a user
holding one sandbox open across a shape-changing deploy.
What would reopen this: Explore sandboxes gaining a long lifespan, or their
stats being read somewhere durable (a report, an export, a cross-unit
aggregate). If it reopens, note that scoping Explore as year - 1 is a much
cheaper approximation than resolving the real N-1/N-2 — it degrades to "skipped"
rather than to a wrong number, because the scope year only gates whether the
aggregation runs and never feeds the stats math (recompute_stats_many derives
report_year from the reports themselves, not from the job's scope).
Tests¶
backend/tests/integration/services/data_ingestion/test_recompute_stats_endpoint_pg.py:
test_recompute_stats_scopes_plan_by_reference_year— a plan atyear=2043 / reference_year=2025alongside a Calculator 2025 report with factors dispatches one job, at year 2025, withskipped_no_factors == 0. Before the fix the plan raised a second, factorless 2043 scope.test_reference_year_slice_collects_plan_modules— the handler's own module query returns the plan's module for the widened 2025 slice, and not for the narrow (recalc-chained) 2025 slice nor the 2043 planning slice. This proves a dispatched job reaches the modules and that the recalc path is unchanged.
backend/tests/unit/tasks/test_aggregation_handler.py pins the default: a plain
aggregation job calls list_modules_for(..., include_reference_year=False).