Operations: Something Is Broken¶
Where to look first, who to call, and what to expect, for dev, stage and prod. Read this page before anything else during an incident. Severity tiers, communication and personal-data rules are in Incident Response; a full rebuild is the private Disaster Recovery Plan. This is a multi-procedure runbook and runs longer than the usual page budget on purpose.
What to expect¶
- Support is best effort, working hours. There is no on-call rotation and nobody is paged.
- Prometheus/Alertmanager and Icinga mail the
co2-calculator-sysadminsgroup, and nothing else. GlitchTip mails the same group and posts to the private co2-calculator Teams channel. - End users report problems to the EPFL service desk,
1234@epfl.ch, which routes them to us.
First ten minutes¶
- Icinga (status page,
host group
co2-calculator): is the host UP, arehttps,https-apiandpgsqlgreen? Redhttps-apiandpgsqltogether means the database. Redhttpsalone means the Route or the frontend. - Alert mail: which alert, which environment, since when. Every alert name is explained in the alert catalog.
- Grafana "specific graphs" for that environment (links below):
latency by
route_class, DB pool checkouts, HTTP errors by status code. This is the dashboard the lead opens first. - ArgoCD: is the application Synced and Healthy? A
chore(manifest)commit close to the symptom start is the usual suspect. Revert it in the GitOps repo; ArgoCD rolls back on its own. - Traces in Tempo: enac-k8s-grafana,
Explore, Tempo data source, filter by service and route in the time
window. Since #2372
a GlitchTip event's
trace_idis expected to match the backend spans in Tempo. Not yet verified end to end: treat a miss as unconfirmed, not as proof the request never reached the backend. Recipes: Debugging with traces. - Pod logs in the OpenShift console (links below), or:
oc login --web
oc logs deploy/co2-calculator-backend -n <namespace> --tail=200
oc get externalsecret,pods -n <namespace>
Links per environment¶
| Where | dev | stage | prod |
|---|---|---|---|
| App | co2-calculator-dev.epfl.ch | co2-calculator-stage.epfl.ch | co2-calculator.epfl.ch |
| Grafana, specific graphs | open | open | open |
| Grafana, namespace | open | open | open |
| ArgoCD application | open | open | open |
| Console, application logs | open | open | open |
| OpenShift console | open | open | open |
Cross-environment: Icinga, Tempo, GlitchTip, Matomo. The same links are on the backoffice Logs page inside the app. What each tool is for and how to get an account: Tools and access.
Symptom to owner¶
Follow an arrow backwards from the symptom to whoever owns it. Anything in the DSI box has a service number and a support queue: open the incident directly in that queue. A ticket in the wrong queue is re-routed by hand and costs hours at the worst possible moment.
graph TB
User([EPFL user · VPN]) -->|HTTPS| Route
subgraph ours["Our workload — ENAC-IT, no DSI ticket"]
Route[OpenShift Route · TLS edge]
FE[Frontend · Quasar SPA]
BE[Backend · FastAPI + worker]
Route -->|/| FE
Route -->|/api| BE
FE -->|XHR + auth cookie| BE
end
subgraph dsi["EPFL DSI — open the ticket in the queue"]
K8s[OpenShift / Kubernetes<br/>SVC1219 · SI_KUBERNETES]
PG[(PostgreSQL DBaaS<br/>SVC1757 · SI_POSTGRESQL)]
S3[(Object storage S3<br/>SVC1057 · ticket via 1234)]
Entra[(Entra ID / AD<br/>SVC0026 · SI_AD)]
end
subgraph other["Other dependencies — no DSI queue"]
Infi[ENAC-IT Infisical<br/>secrets, all environments]
Tab[EPFL Tableau<br/>travel + headcount]
ECB[European Central Bank<br/>exchange rates]
GH[GitHub + Quay<br/>CI, images, GitOps]
Obs[ENAC-IT observability<br/>Tempo, Icinga, GlitchTip]
ES[(EPFL OPDo Elasticsearch<br/>audit records)]
end
K8s ==>|hosts| Route
GH ==>|ArgoCD sync| K8s
K8s -.->|External Secrets Operator pulls| Infi
BE -->|SQL| PG
BE -->|S3 API · ingestion files| S3
BE -->|OIDC code exchange| Entra
User -.->|login redirect| Entra
BE -->|scheduled pull| Tab
BE -->|scheduled pull| ECB
BE -.->|OTLP| Obs
BE -->|audit sync| ES
| Dependency | Service | Queue | Open a ticket when |
|---|---|---|---|
| OpenShift / Kubernetes | SVC1219 | SI_KUBERNETES |
Pods will not schedule, Route down, wildcard certificate, cluster-wide failure |
| PostgreSQL (DBaaS) | SVC1757 | SI_POSTGRESQL |
Database unreachable, restore or point-in-time recovery request |
| Object storage (S3) | SVC1057 | 1234@epfl.ch + bucket name |
Bucket unreachable, S3 credentials rejected |
| Entra ID / Active Directory | SVC0026 | SI_AD |
Login broken, group or role claims missing |
Numbers and queues are as supplied by EPFL DSI. The linked service page carries the current service manager; read it there rather than trusting a name copied into a doc.
Object storage has no published SI_ queue. Open the ticket at
1234@epfl.ch and give the bucket name: it is S3_BUCKET in the backend
secret, set from the GitOps overlay. Buckets are ordered and managed
through the VPSI XaaS portal (https://portal-xaas.epfl.ch,
FAQ).
Infisical, Tableau, the ECB feed, GitHub and the ENAC-IT observability
stack are not DSI services; an SI_ ticket for them goes nowhere.
Contact ENAC-IT (roster in the DRP) for Infisical, Tempo, Icinga and
GlitchTip; the others are vendor or data-source issues.
Ours: symptom, cause, action¶
Every alert in openshift-app-config carries a runbook_url that lands
here, or on
which knob, when an alert fires
for the DB rules. Keep both headings as they are, or update the three
overlays in the same change.
| Symptom | Usual cause | Do |
|---|---|---|
CreateContainerConfigError or CrashLoop after a rollout |
A secret key is missing or a setting is invalid | oc get externalsecret -n <ns> and oc describe pod; fix the key in Infisical or the overlay, then oc rollout restart deploy/co2-calculator-backend |
ImagePullBackOff |
Tag not in Quay yet, or the pull robot credential expired | Check the GitHub Actions run, then the quay-reg-cred ExternalSecret |
DbPoolSaturationHigh, then DbPoolCheckoutTimeout |
DB layer 1: this pod's pool (DB_POOL_SIZE + DB_MAX_OVERFLOW) is full. Behind the bouncer a backend pod should have scaled out; on the worker, jobs are queuing on connections |
Check the HPA, then the Grafana DB pool panel; which knob; read plans 2566 and 1723 before touching pool sizes |
DbBouncerQueued, then DbBouncerQueueTimeout |
DB layer 2: more than 60 (dev) or 70 (stage, prod) transactions in flight at once filled PgBouncer's server pool (transaction pooling on all three since 2026-09-24). Reference event 2026-09-17 17:07 to 17:13, under the old 25-slot session pool and 120 s wait: total open crossed 25, fresh logins queued 120 s while holding pod pool slots, backend pools hit 100%, 5 s pool timeouts, /ready red, a 503 band. At today's 10 s wait the same overrun is a short error band, not a lockup |
Nothing local heals it. Stray client first (application_name in pg_stat_activity), then a long transaction holding slots (COPY factor upsert, long routes), then a bigger default_pool_size from DBaaS; which knob |
DbServerConnectionSlotsExhausted |
DB layer 3: Postgres itself is full (53300). Orphaned backends from dead pods, or a laptop on the shared DB. The 2026-08-31 mode: everyone is locked out, pgAdmin and migrations included | Count orphans: select state, count(*), max(now()-state_change) from pg_stat_activity where usename='app' group by state; pg_terminate_backend the idle ones; SI_POSTGRESQL if max_connections must grow; which knob |
| Many users get 403 at once | Role sync received an empty list from the provider | Plan 2531; check /api/health/deps for the role provider |
AccredUnavailable, logins land on "EPFL Accred is unavailable" |
api.epfl.ch answered 5xx, timed out or was unreachable (upstream, nothing here to restart). The status_code label tells it apart: 401/403 is our API key rejected, 429 is rate-limiting, unconfigured is missing Accred settings |
Confirm with curl -u <user>:<key> https://api.epfl.ch/v1/persons/<id> or the EPFL IT status page; for 401/403 rotate ACCRED_API_KEY in Infisical; RoleSyncProviderUnavailable co-fires and clears on its own; /api/health/deps |
| Upload or recalculation never finishes | Stuck job | GET /api/health/stale-stats; plans 1219 and 1559 |
HighErrorRate, LatencyP99High |
A slow or failing route | Grafana errors by status code, then Tempo for the slowest traces on that route |
BackendMetricsAbsent |
Backend pods down, or the OTel collector pod down | oc get pods -n <ns>; the collector is otel-collector |
Icinga ssl-cert warning |
Cluster wildcard certificate nearing expiry | Not ours: SI_KUBERNETES |
Icinga pgsql CRITICAL, app still up |
Probe path to DBaaS, or DBaaS maintenance | Check /api/health/deps; if the app is failing too, SI_POSTGRESQL |
Recovery¶
- Bad deploy: revert the GitOps commit. ArgoCD reconciles within minutes.
- Database: nightly dump on the
db-dumpsPVC. Restore: the step-by-step runbook and one-shot script in the private ops README (exercised for real on 2026-09-08, see the post-mortem); point-in-time recovery is aSI_POSTGRESQLticket. - Files: not recoverable, by decision. Ask the user to re-upload.
- Anything larger: the Disaster Recovery Plan, linked at the top.
After the incident¶
Follow Incident Response § After the incident. Write the postmortem as a page in this folder, in the shape of the OAuth redirect postmortem, and add it to the nav in the same PR.