996 — _process_row factor counter always reports rows_without_factors¶
Root cause (confirmed)¶
_process_row(backend/app/services/data_ingestion/base_csv_provider.py:1357) always doesreturn data_entry, None, None, kg_co2eq_override— position 3 (factor) is a literalNone, never the result of anything.- The caller,
process_csv_in_batches(base_csv_provider.py:1055-1058), doesif factor: stats["rows_with_factors"] += 1 else: stats["rows_without_factors"] += 1— so every successfully processed row lands inrows_without_factors. - Confirmed to pre-date #988: parent commit
b39fec7ealready hadreturn data_entry, None, None. #988 only appended the 4th element (kg_co2eq_override). Matches the issue's origin note anddocs/code-review/988-copilot-feedback-fix-data-entry-stop-persisting-computed-fields-to-dataentry-data.md, "Skipped after verification" section.
Premise gap found during investigation¶
The issue's suggested fix assumes "the resolution logic is already in
_process_row" and this is pure return-value plumbing. True for exactly one
of four ingestion paths, false for the rest:
- MODULE_PER_YEAR, priority-3 type inference
(
csv_providers/module_per_year.py:167-216): whendata_entry_type_idisn't given by job config or a CSV category column, it infers the type vialookup_data_entry_type_by_kind, which scansfactors_map. A hit here is a genuine "this row's kind/subkind matched a factor" signal — but it's discarded; only the resultingDataEntryTypeEnumsurvives. - MODULE_PER_YEAR, priority 1/2 and all of MODULE_UNIT_SPECIFIC
(
csv_providers/module_unit_specific.py:55-99):data_entry_typecomes straight from job config or a category column. Nofactors_maplookup ever runs. Both subclasses' docstrings claim "factor validation is handled by ModuleHandlerService when it queries the database in_process_row" — checked, and it doesn't hold: ModuleHandlerService(app/services/module_handler_service.py) only has taxonomy-tree and typeahead methods for the UI; no per-row match check.require_factor_to_match(app/schemas/data_entry.py:136,279) is read exactly once in the whole backend — at setup time in_guard_factors_required(fail-fast whenfactors_mapis empty for a module that needs it) — never per row.validate_createis a synchronous pydantic call with no DB session, so it isn't doing a live factor lookup either.- Conclusion: for most rows, "does this row have a matching factor" is not computed anywhere in the ingestion path today. The docstring is stale/aspirational, not a pointer to real code.
Related, out of scope for this issue¶
api_providers/base_tableau_api_provider.pydeclares the identicalrows_with_factors/rows_without_factorspair (lines 46-47, 853-854) and never increments either — professional-travel API ingestion always reports 0/0 for both. Same flavor of bug, different class hierarchy (DataIngestionProvider, notBaseCSVProvider). Not touched here; flag as a possible follow-up issue.frontend/src/stores/backofficeDataManagement.ts:67-68types the two fields but nothing renders them structurally — they only reach the user pre-formatted inside the backend's summary sentence (base_csv_provider.py:1440-1441). No frontend behavior change either way.
Options¶
A — Make the counter real for the one path that already checks. Return
whether lookup_data_entry_type_by_kind matched, for MODULE_PER_YEAR
priority-3 rows only. Priority-1/2 and MODULE_UNIT_SPECIFIC rows still need a
default (e.g. always counted as "with factor," since no check applies to
them) — closer to correct than today, but the semantics differ by entity
type and the default is itself a judgment call.
B — Extend the check to every row. Add a factors_map existence check to
the priority-1/2 path too, reusing the kind/subkind key logic that today only
lives inside seed_helper.lookup_data_entry_type_by_kind. Makes the stat
uniform and honest, but is new logic on a path that currently has none — a
design change, not plumbing.
C — Drop the counter (recommended). Delete
rows_with_factors/rows_without_factors from StatsDict, its
initialization, the increment, and the summary sentence
(base_csv_provider.py:110-111,898-899,1055-1058,1440-1441); update the 4
backend tests that reference the keys; drop the two optional fields from the
frontend type. A number nobody can currently compute correctly is worse than
no number ("no silent fallbacks"). Smallest diff; matches the issue's own
stated impact ("user-visible only via job summary text").
Decision¶
Option C, confirmed by Guilbert 2026-09-01. Smallest, most honest diff, and nothing structurally depends on the counter downstream. A and B both require deciding what "has a factor" means for entity types that never check today — a real design conversation, not a bug fix — parked, not pursued here.
Regression test¶
Assert the job summary/status message no longer mentions factor counts and
StatsDict no longer carries rows_with_factors/rows_without_factors;
assert process_csv_in_batches still completes end-to-end (no KeyError
from the removed keys, no leftover reference in the summary string).
Implementation checklist¶
-
base_csv_provider.py: removedrows_with_factors/rows_without_factorsfromStatsDict, init, theif factor: ... else: ...increment (keptstats["rows_processed"] += 1), and the summary sentence fragments. -
_process_rowreturn type: dropped the always-None3rd tuple element (factor) — signature, docstring, everyreturnstatement, and the caller's unpack all updated to a 3-tuple(DataEntry | None, str | None, kg_co2eq_override). - Found and fixed a ripple the checklist hadn't accounted for:
csv_providers/local_seed.pyoverrides_process_row(forwards tosuper()) and had its own now-stale 4-tuple return annotation — updated to match. - ~~
_resolve_handler_and_validate: drop the deadfactor: Any | Noneparameter~~ — descoped. That parameter is independent of_process_row's return tuple (it's an input, alwaysNone, ignored by both subclass overrides) and touching it would have rippled into 11 more test call sites acrosstest_module_per_year_csv_provider.pyandtest_module_unit_specific_csv_provider.pyfor zero behavior change. Left as a separate, still-dead cleanup opportunity — not part of the scope Guilbert approved for this issue. - Updated the backend tests referencing the two keys:
test_base_csv_provider.py(_build_stats/_make_statshelpers, thetest_stats_dict_structureassertions, the_finalize_and_commitsummary-message assertion, and 10_process_rowcall sites unpacking the old 4-tuple),test_module_per_year_csv_provider.py, andtest_module_unit_specific_csv_provider.py.test_professional_travel_api_provider.pyconfirmed out of scope (exercisesBaseTableauApiProvider's separateStatsDict, untouched) and left alone. -
frontend/src/stores/backofficeDataManagement.ts:67-68: dropped the two optional fields — grep confirmed nothing else infrontend/srcreferenced them. - Regression test:
test_finalize_and_commit_moves_file_and_updates_job's summary-message assertion andtest_stats_dict_structure's key assertions were flipped to the new expectation first and confirmed to fail against the pre-fix code (RED) before the production change (GREEN). Fulltests/unit/services/data_ingestion/suite (305 tests) passes;make lintclean.
2026-09-02 — folded in the identical Tableau-provider counter¶
Guilbert asked to fix the related-but-out-of-scope finding
(api_providers/base_tableau_api_provider.py) in this same PR rather than as
a separate follow-up issue. Investigation confirmed the same conclusion as
the CSV path, more starkly: rows_with_factors/rows_without_factors are
declared on StatsDict and zeroed in _init_stats(), but grepping every
stats[...] mutation in base_tableau_api_provider.py and
professional_travel_api_provider.py turns up no increment site at all —
the per-record loop (_inject_module_ids) only ever resolves
carbon_report_module_id, never a factor. Applied the same Option C:
-
base_tableau_api_provider.py: removed the two keys fromStatsDictand_init_stats(). -
professional_travel_api_provider.py: removed the twof"... with/without factors ..."fragments from_success_status_message(the only subclass that referenced them;headcount_members_api_provider.pyandresearch_facilities_api_provider.pynever did). -
test_professional_travel_api_provider.py: updated the twoTestRecordRowErrorstats literals; addedTestSuccessStatusMessage.test_summary_does_not_claim_a_factor_countas a real regression test (calls the production method, not a self-contained literal) — confirmed RED against the pre-fix message, GREEN after. -
docs/src/frontend/data-management/api-and-troubleshooting.md: the SSE payload example still showedrows_with_factors/rows_without_factors— corrected. (Leftdocs/src/implementation-plans/1398-...mdalone: astatus: deliveredhistorical plan quoting the code as it stood at delivery time, not a live contract.) - Full
tests/unit/services/data_ingestion/suite (306 tests) passes;make lint/make type-checkclean.