Load tests with locust (#2295)¶
Goal: find which endpoints exceed 1 s under realistic concurrency, locally first, then against the dev platform (~10× slower DB).
Complementary to the #2161 plan (single-request, in-process, one unit at ceiling): this suite is real HTTP against a running stack, many units, many concurrent users.
What shipped¶
backend/app/seed/ceilings.py— #2161's per-type ceilings as code (single source;scripts/generate_perf_test_csvs.pynow derives itsCEILINGSfrom it). Pinned bytests/unit/seed/test_ceilings.py.backend/tests/performance/locustfile.py— rewritten (old file was dead scaffolding hitting nonexistent/api/v1/users). Four scenarios:ExplorerReadUser(workspace home, merged modules-stats, unit totals, module/submodule reads, explore GET),ExploreCreateUser,PlanUser(create → set reference year → poll prefill → read → delete),CsvUploadUser(temp-upload → dispatch → poll pipeline; flow wall time recorded as aFLOWstats row).backend/tests/performance/report_slow.py— prints every (stage, endpoint) with p95 over a threshold (default 1000 ms) from the run CSVs.- Backdrop seeding knobs on the existing COPY seeder
(
app/seed/random_generator/):SEED_NUM_UNITS,SEED_NUM_USERS,SEED_YEARS,SEED_CEILING_SCALE(per-type ceiling × scale, all VALIDATED, instead of the random ~67/module density).seed_year_configurationnow also stamps provider TEST sologin-testusers see the seeded years.seed_allrefuses non-local DB hosts unlessSEED_ALLOW_REMOTE=1(backend/.env commonly points at the shared dev DB). backend/Makefiletargets:perf-csvs,perf-seed,perf-load,perf-sweep,perf-report,perf-db-dump,perf-db-restore.
How to run (local)¶
# one-time, with backend/.env DB_URL pointing at localhost
make run-db
cd backend
make db-migrate
make seed-data # locations, rooms, factors
make perf-seed # 600 units × 5 years × 0.1 ceiling ≈ 6M rows
make perf-csvs # upload CSVs from real factor data
make perf-db-dump # snapshot the expensive backdrop
uv run uvicorn app.main:app --workers 4 --port 8000
# then, per stage or the whole ladder
make perf-load PERF_USERS=50 PERF_CLASSES=ExplorerReadUser
make perf-sweep
make perf-report # endpoints with p95 > 1s
Reports land in backend/tests/performance/reports/ (gitignored), one
CSV+HTML pair per stage. Restore the backdrop anytime with
make perf-db-restore instead of re-seeding.
Against dev: make perf-dev (2026-09-23) — targets
https://co2-calculator-dev.epfl.ch/api (the /api prefix is required, the
bare host is the SPA), logs VUs in via login-test (dev is a DEBUG build) and
refuses write scenarios without PERF_ALLOW_WRITES=1. For a host without
login-test, export PERF_AUTH_COOKIE=<auth_token JWT> and use perf-load.
Dev ladder, 2026-09-23/24¶
make perf-dev (#2924, #2929, #2930, #2931) against dev on PgBouncer
transaction mode, pool 70, backend HPA 2–6 pods × 1 core (osac #66):
capacity ≈ 150 req/s for ExplorerReadUser (148 at 600 users, 138 at 800
with the first 500s, 109 at 1000 with 11 % failures). Saturation is backend
CPU on the busiest single-worker pod (0.93 core at 600 users), with the
bouncer pool filling as a symptom of queued requests holding their
transaction. Pod → Postgres round trip is 1.0 ms, not 14 ms (see the 2527
plan correction). Baseline for before/after, tag pool70_before: 100 users
32 req/s p95 630 ms; 600 users 142 req/s p50 1.1 s p95 2.4 s. Use 3-minute
runs when Prometheus gauges matter (30 s scrape). Overload answers 500 where
it should answer 503, and the shared-cookie session read can answer 401
under DB pressure — both tracked in the goal document linked from #2295.
What the first local run taught us (2026-08-29)¶
- The random seeder had drifted from the dev schema:
carbon_projects/carbon_reportsgrew partial/extended unique indexes the seeder'sON CONFLICTtargets no longer matched — fixed inseed_carbon_reports.py. ItsSEEDED_UNIT_LEVEL = 5also predated #930'sUnit.level == 4workspace filter, which made seeded units invisible — now 4. - Unit listing (
get_user_units) is membership + level-4 based; a GlobalScope backoffice user sees no units, and merged modules-stats validate against that same allow-list. Module reads/uploads additionally needmodules.*permissions the admin lacks. Hence the role model: unit roles drive the load; the seeder maps its first four fake units to the TEST leaf iids so principal/standard own ceiling-loaded units, andPERF_UNIT_IDSfeeds admin-wide reads. - The pipeline endpoint serializes job state/result enums by NAME
(
"FINISHED","ERROR"), not int value. - A fresh simulator plan has no year rows:
PATCH /{plan_id}withstart_year/end_year/default_reference_yearcreates them and enqueues the prefill job. - The full upload chain (temp-upload → dispatch → csv_ingest → emission_recalc → aggregation) was verified green end-to-end as the principal login-test user.
Traps encoded in the locustfile (don't rediscover these)¶
RequestOriginMiddleware403s every cookie-authed non-GET withoutSec-Fetch-Site/Origin; the client sendsSec-Fetch-Site: none.- Auth is an httpOnly cookie, not a bearer token;
login-test302s and needs an exactRoleNamevalue (calco2.backoffice.adminfor global scope) — a wrong value yields a user with zero roles, not an error. - Job/pipeline states are int enums (
state==3finished, then checkresult;2= error). PollGET /v1/sync/pipelines/{id}, never the SSE/streamroutes. - Trailing slashes are load-bearing on
carbon-reports/unit/...andproject-plans/unit/...routes (otherwise you measure 307s).
Deliberate simplifications (ceilings for later)¶
- Backdrop emissions are the seeder's placeholders (
primary_factor_idNULL): reads skip factor-join enrichment cost. The CSV-upload scenario exercises the real ingest+recalc path during the test, so the production-path cost is measured where it is heaviest. Upgrade path: run the recalc workflow over the backdrop, or raiseSEED_CEILING_SCALEand accept the longer seed. - All VUs of a role share one test user; per-unit-scoped users (role provider scopes) are not simulated.
- Default scale is 0.1 (≈6M rows). The full 600×5×21050 ≈ 63M is a knob
away (
SEED_CEILING_SCALE=1), not a code change.