Skip to content

Manual perf-test CSV generator: one file per data_entry_type (#2161)

Backfilled — delivered directly, no plan-first pass (small, mechanical extension of an existing script with an explicit ceiling table given).

What changed

backend/scripts/generate_perf_test_csvs.py previously generated 3 coarse CSVs (travel, buildings, purchase) whose row counts were the sum of several DataEntryTypeEnum ceilings, hiding which type the rows actually targeted and mixing e.g. all 7 purchase categories into one arbitrary file. It now has one generator per calculator DataEntryTypeEnum (23 types), sized at #2161's real per-unit-year ceilings (same numbers documented in 2161-ceiling-scale-perf-fixtures.md's CEILING_PER_UNIT_YEAR table).

Each generator samples real rows from backend/INPUT_DATA/*_factors.csv / *_reference.csv (gitignored, developer-supplied) so every emitted row resolves to a real factor — an unresolvable value fails the row since #2050 Track J1, so inventing values would measure the error path, not the module. Where a type shares its factor table with siblings (equipment's scientific/it/other, purchase's 7 categories), one parametrized generator filters the shared table by category rather than duplicating near-identical code per type.

building_construction_renovation (data_entry_type 32, still named building_embodied_energy in the model) has no CSV ingest — it's derived server-side from building rows during ingest. The generator table documents this and returns no file rather than fabricating one.

Why not app/seed/ceilings.py

2161-ceiling-scale-perf-fixtures.md plans a CEILING_PER_UNIT_YEAR dict in app/seed/ceilings.py as the single source of truth for these numbers, but that module doesn't exist yet on this branch (it's Task 1 of that plan, not merged). This script hardcodes the same numbers directly, mirroring its own prior style (a module-level CEILINGS dict) rather than depending on unmerged code. When app/seed/ceilings.py lands, re-key this script's CEILINGS off CEILING_PER_UNIT_YEAR to keep the two from drifting.

Also added

backend/scripts/README.md — a one-page index of every script in the folder (this generator plus the pre-existing audit/migrate/dedupe/seed/db scripts), each with its purpose and invocation, since none existed before.