Ops runbook and tools documentation¶
Problem¶
The developer docs had no day-to-day operations entry point. The triage
content (service map, DSI queues) sat under "Security Documentation";
the tool URLs (Grafana, ArgoCD, console logs) lived only as hardcoded
strings in frontend/src/pages/system/LogsPage.vue; Tempo, Icinga,
GlitchTip and Matomo were not linked anywhere. infra/01-overview.md
described Jaeger, Kibana, Elasticsearch, grafana.epfl.ch, and
hand-managed stage/prod secrets, none of which exist today. Everything
else lived in the lead developer's head.
Decisions¶
- Internal dashboard hostnames are accepted as public: they already ship in the public repo and sit behind VPN and SSO. The docs link them.
- Secret locations stay out of the public docs beyond the vault hostname. Names, emails and restore commands stay in the private ops repo.
- No new pattern: pages live under the existing
infra/folder, the nav section is renamed "Infrastructure & Operations" and the incident page goes first.
Delivered¶
infra/04-operations.md— what to expect, first ten minutes, links per environment, symptom-to-owner map with DSI queues (moved from the security page), our-side symptom table, recovery, after the incident.infra/06-debugging-with-traces.md— six symptom-driven recipes, each step cited to the plan or issue where it was done.infra/05-tools-and-access.md— every tool, what it is for, how to get in; the EPFL service account and groups behind the service; onboarding steps.infra/01-overview.md— rewritten from the GitOps overlays: routes, namespaces, what runs in a namespace, deploy path, secrets, backups.- Stale claims removed from
02-system-overview,03-subsystem-map,05-environments,07-global-conventions,11-deployment-topology,encryption: Jaeger, Azure Key Vault, manual stage/prod secrets. security-documentation.mdpoints at the operations page for the service map.frontend/error-monitoring.mdno longer links the vault project. Home page gets a "Something is broken?" entry.
Follow-ups¶
infra/06-debugging-with-traces.mdshipped with three UNVERIFIED markers: GlitchTiptrace_idlookup in Tempo, thehttp.status_codespan attribute name, and the log-view pod filter. Verify each once and delete the marker.- Which EPFL group feeds the
admin-SVC1751rolebinding, and whatco2-calculator-opsactually grants. Two groups exist by accident; kept both on 2026-09-07 until understood, then merge into one. - Verify end to end that a GlitchTip
trace_idfinds its backend spans in Tempo; the docs say "expected, not verified" until then. - Renew service account
svc-calcco2-epfl-apibefore 2026-10-20.