ARCHIVED: This plan was not delivered. Preserved for context.
Plan: 707 — Implement sync_module_factors (Backend only)¶
Using example of Research facilities module, this feature aims at allowing a backoffice manager to update factors that are already stored in the database. The update logic is module data entry type dependent.
TL;DR: Implement the sync_module_factors stub in app/api/v1/data_sync.py. For (module_type, data_entry_type, year), the endpoint walks all factors stored for those parameters and recomputes their values fields by pulling actual DataEntryEmission totals — using classification.researchfacility_id to locate the correct Unit → CarbonReport → CarbonReportModule → emissions. Reuses the existing DataIngestionJob + background-task infrastructure, so callers poll via SSE identically to sync_module_data_entries.
Phase 1 — New ingestion method¶
Step 1 – Add IngestionMethod.computed = 3 to DataIngestionMethod enum¶
- File:
data_ingestion.py - Distinguishes "compute from existing emissions" from csv/api imports.
- Requires an Alembic migration:
ALTER TYPE ingestion_method_enum ADD VALUE 'computed'(native PostgreSQL enum — no transaction wrapper).
Phase 2 — Factor-update provider¶
Step 2 – Abstract BaseFactorUpdateProvider (new file)¶
backend/app/services/data_ingestion/factor_update_provider.py- Extends DataIngestionProvider; defines abstract compute_factor_values(factor, year, session) → dict | None.
- Concrete ingest() loop: fetch all factors for (data_entry_type_id, year) via FactorRepository.list_by_data_entry_type, call compute_factor_values, apply FactorRepository.update, track stats.
Step 3 – Concrete ResearchFacilitiesAnimalFactorUpdateProvider (new file)¶
backend/app/services/data_ingestion/computed_providers/research_facilities_animal.py- For each factor:
- Read researchfacility_id from factor.classification.
UnitRepository.get_by_institutional_id(researchfacility_id)→ Unit.CarbonReportRepository.get_by_unit_and_year(unit_id, year)→ CarbonReport.CarbonReportModuleRepository→ find module for (carbon_report_id, module_type_id=research_facilities).DataEntryEmissionRepository.get_stats(carbon_report_module_id, aggregate_by="emission_type_id")→ sums per emission_type.- Map EmissionType leaf → source name (e.g. process_emissions → processemissions, buildings__rooms → building_rooms, etc.).
- Return updated values dict: only overwrite kgco2eq_sum keys; leave shares / use_unit / total_use untouched.
- Surface missing unit/carbon-report with a clear error — no silent skip.
Step 4 – Register in ProviderFactory¶
- File:
provider_factory.py - Add entry: (ModuleTypeEnum.research_facilities, IngestionMethod.computed, TargetType.FACTORS, EntityType.MODULE_PER_YEAR) → ResearchFacilitiesAnimalFactorUpdateProvider.
Phase 3 — API endpoint¶
Step 5 – Implement sync_module_factors stub (depends on Steps 1–4)¶
- File:
data_sync.py - Fix route path typo: {module_id} → {module_type_id}.
- Remove duplicate permission check body (decorator already handles it).
- Validate syncRequest.year is mandatory.
- Set entity_type = EntityType.MODULE_PER_YEAR, call ProviderFactory.create_provider(..., ingestion_method=IngestionMethod.computed), create job, schedule background task, return SyncStatusResponse — mirrors sync_module_data_entries exactly.
Phase 4 — Tests¶
Step 6 – Unit tests for ResearchFacilitiesAnimalFactorUpdateProvider¶
- New file:
backend/tests/unit/services/data_ingestion/test_research_facilities_animal_factor_update.py - Cases: happy path (all sources), unit not found, carbon report not found, partial sources present.
Step 7 – Extend test_provider_factory.py (parallel with step 6)¶
- Assert new (research_facilities, computed, FACTORS, MODULE_PER_YEAR) key resolves to the new provider.
Relevant files¶
data_sync.py— fill in stubdata_ingestion.py— add computed = 3backend/app/services/data_ingestion/factor_update_provider.py— new abstract basebackend/app/services/data_ingestion/computed_providers/research_facilities_animal.py— new concrete providerprovider_factory.py— registerfactor_repo.py— reuse list_by_data_entry_typedata_entry_emission_repo.py— reuse get_statsversions— new migration for native enum valuebackend/tests/unit/services/data_ingestion/test_research_facilities_animal_factor_update.py— newtest_provider_factory.py— extend
Verification¶
- make lint + make typecheck (ruff + mypy strict) pass on all changed files.
- New test file: all 4 cases pass.
- Provider factory test: new registration asserted.
- Manual: POST /data-sync/factors/research_facilities/mice_and_fish_animal_facilities body {"ingestion_method": 3, "target_type": 1, "year": 2025} → returns job_id; SSE polling → FINISHED/SUCCESS; Factor.values.kg_co2eq_sum_processemissions updated per facility.
- ModulePerYearFactorCSVProvider tests still pass (no regression on CSV factor ingestion).
Decisions¶
- IngestionMethod.computed = 3 avoids conflating this with api in job history.
- year is mandatory for this endpoint (unlike data-entry sync).
- Route typo {module_id} → {module_type_id} fixed as part of this PR.
- ResearchFacilitiesCommonFactorUpdateProvider (single kg_co2eq_sum) is out of scope.
Further considerations¶
- CarbonReportModuleRepository gap — confirm whether a method get_by_carbon_report_id_and_module_type already exists; if not, add it before Step 3 is executable.
- Emission aggregation scope — get_stats aggregates all DataEntryEmission rows for a carbon_report_module_id; confirm that the research_facilities module stores all lab-emission sub-types (process, buildings, rooms, equipment, purchases) under a single CarbonReportModule, or whether separate modules need to be joined.
- Alembic native-enum migration — ALTER TYPE … ADD VALUE cannot run inside a transaction; verify alembic.ini settings (transaction_per_migration) or add op.execute with explicit connect_args={"isolation_level": "AUTOCOMMIT"}.