Bootstrap year configurations from INPUT_DATA¶
Problem¶
After every make db-drop / make clean-db, a developer had to redo the whole
backoffice click-path by hand: create the year, wait for the Accred unit sync,
upload ~15 factor CSVs plus the 3 reference CSVs, then fill the three
reduction-objective file slots and re-enter the institutional goals. Several
times a week, ~15 minutes each.
Second problem found on the way: app/seed/seed_generic_factors.py (behind
make seed-data) pointed at backend/seed_data/ while listing the current
filenames, which only exist in backend/INPUT_DATA/ — equipment_factors.csv,
purchases_centralized_factors.csv, headcount_students_factors.csv,
buildings_construction_renovation_factors.csv. It died on
FileNotFoundError.
What shipped¶
make bootstrap-years (in backend/Makefile, YEARS ?= 2025 2026) runs
app.seed.bootstrap_years. Per year, in order:
- Year configuration — insert the
year_configurationrow withgenerate_default_year_config()if absent. Years are validated againstsettings.MIN_CONFIGURABLE_YEARand the current year, the same boundsPOST /year-configuration/{year}enforces. - Unit sync — build the same
unit_syncDataIngestionJobthe endpoint enqueues (ensure_pipeline_exists→create_ingestion_job→meta.config.target_year) and awaitrun_job(job_id)instead of firing it in the background.run_jobconverts handler failures into FINISHED+ERROR rather than raising, so the job row is re-read afterwards and a non-SUCCESSresult raises. This is what createsunits,users,carbon_reports,carbon_report_modulesand theconfiguration_completedstamp that unblocks uploads. - Factors —
seed_all_factors(session, year)overFACTOR_SEEDS. - Reference data —
seed_all_reference_data(session, year). - Emission recalculation — one
module_emission_recalcjob per module, mirroringPOST /sync/recalculate-emissions/{module_type_id}. See Why recalculation has to run below. - Reduction objectives + goals — the three CSVs plus goals
2030 / 10 % / ref 2016,2035 / 10 % / ref 2016,2040 / 10 % / ref 2016. - Open the year —
is_started = True.
Data entries are deliberately not seeded;
Files¶
| File | Change |
|---|---|
backend/app/seed/bootstrap_years.py |
New — the orchestrator + --years CLI |
backend/app/seed/seed_reference_data.py |
New — LocalReferenceCSVProvider + REFERENCE_SEEDS |
backend/app/seed/seed_reduction_objectives.py |
New — CSV loading, file storage, DEFAULT_GOALS |
backend/app/seed/seed_generic_factors.py |
Repointed at INPUT_DATA/; seed_factors/main parameterized by year |
backend/app/tasks/_background.py |
New wait_for_background_tasks() drain helper |
backend/Makefile |
bootstrap-years target |
Reuse over reinvention¶
LocalReferenceCSVProvidersubclasses the productionReferenceDataCSVProviderand overrides onlyvalidate_connection,_stage_file(read from disk),_move_to_processed(no-op) and_update_job(no-op) — exactly the shapeLocalFactorCSVProvideralready uses incsv_providers/local_seed.py. Header validation, the locations COPY + natural-key UPSERT, the per-mode replace and the building-rooms delete-and-insert are inherited untouched.- Reduction-objective CSVs go through the endpoint's own
validate_reduction_objective_csv(), and are stored using the endpoint's ownget_files_storage_path()/generate_unique_filename(), so the on-disk layout cannot drift from a real upload. - Every seeded CSV plants a
create_seed_stub_job(...)row so the backoffice data-management cards show the seed instead of staying blank.
Why recalculation has to run¶
Seeded factors go in through LocalFactorCSVProvider, which bypasses
factor_ingest and therefore never fans out the emission_recalc children a
real upload chains. get_recalculation_status_by_year flags any type whose
latest FACTORS job is newer than its latest computed DATA_ENTRIES job — so
without this step backoffice showed "Recalculation needed" on every module
right after a clean bootstrap.
The fan-out targets are read off FACTOR_SEEDS, not off
get_recalculation_status_by_year: a multi-type CSV (equipment_factors.csv,
purchases_common_factors.csv) plants its stub job with
data_entry_type_id = NULL, and that query filters NULLs out, so it does not
name every type that just received factors.
module_emission_recalc (rather than per-type emission_recalc) is used
because its handler already writes the per-type stub jobs the status query
matches on, and chains a single aggregation child per module instead of one
per type.
Those chained aggregations are dispatched through fire_and_forget. A CLI has
to wait for them before the event loop closes or they are cancelled mid-run and
left stuck in RUNNING — hence wait_for_background_tasks() in
app/tasks/_background.py.
Reference seeds¶
CSV in INPUT_DATA/ |
data_entry_type | module_type |
|---|---|---|
travel_planes_reference.csv |
plane (20) |
professional_travel (2) |
travel_trains_reference.csv |
train (21) |
professional_travel (2) |
building_rooms_reference.csv |
building (30) |
buildings (3) |
seed_locations.py / seed_building_rooms.py are untouched — they still back
make seed-data from seed_data/, and _NATURAL_KEY_EXPR still lives in
seed_locations.py where reference_data.py imports it.
Verification¶
Ran twice against a local Postgres, second run confirming idempotency
(identical counts, no unique-index collisions on is_current or factor
identity):
year_configuration: 2025 + 2026,is_started=t,configuration_completedset, 3 goals and 3 file entries eachfactors: 24 296 per yearlocations: 4 478 plane + 51 299 trainbuilding_rooms: 19 925data_entries: 0- 18 seeded
is_currentingestion jobs per year (15 FACTORS + 3 REFERENCE_DATA) needs_recalculation = falsefor every(module, data_entry_type)— no "Recalculation needed" badges in backoffice- every
data_ingestion_jobsrow FINISHED/SUCCESS, including the chainedaggregationchildren — nothing stuck in RUNNING