Skip to content

Operations: Something Is Broken

Where to look first, who to call, and what to expect, for dev, stage and prod. Read this page before anything else during an incident. Severity tiers, communication and personal-data rules are in Incident Response; a full rebuild is the private Disaster Recovery Plan. This is a multi-procedure runbook and runs longer than the usual page budget on purpose.

What to expect

  • Support is best effort, working hours. There is no on-call rotation and nobody is paged.
  • Prometheus/Alertmanager and Icinga mail the co2-calculator-sysadmins group, and nothing else. GlitchTip mails the same group and posts to the private co2-calculator Teams channel.
  • End users report problems to the EPFL service desk, 1234@epfl.ch, which routes them to us.

First ten minutes

  1. Icinga (status page, host group co2-calculator): is the host UP, are https, https-api and pgsql green? Red https-api and pgsql together means the database. Red https alone means the Route or the frontend.
  2. Alert mail: which alert, which environment, since when. Every alert name is explained in the alert catalog.
  3. Grafana "specific graphs" for that environment (links below): latency by route_class, DB pool checkouts, HTTP errors by status code. This is the dashboard the lead opens first.
  4. ArgoCD: is the application Synced and Healthy? A chore(manifest) commit close to the symptom start is the usual suspect. Revert it in the GitOps repo; ArgoCD rolls back on its own.
  5. Traces in Tempo: enac-k8s-grafana, Explore, Tempo data source, filter by service and route in the time window. Since #2372 a GlitchTip event's trace_id is expected to match the backend spans in Tempo. Not yet verified end to end: treat a miss as unconfirmed, not as proof the request never reached the backend. Recipes: Debugging with traces.
  6. Pod logs in the OpenShift console (links below), or:
oc login --web
oc logs deploy/co2-calculator-backend -n <namespace> --tail=200
oc get externalsecret,pods -n <namespace>
Where dev stage prod
App co2-calculator-dev.epfl.ch co2-calculator-stage.epfl.ch co2-calculator.epfl.ch
Grafana, specific graphs open open open
Grafana, namespace open open open
ArgoCD application open open open
Console, application logs open open open
OpenShift console open open open

Cross-environment: Icinga, Tempo, GlitchTip, Matomo. The same links are on the backoffice Logs page inside the app. What each tool is for and how to get an account: Tools and access.

Symptom to owner

Follow an arrow backwards from the symptom to whoever owns it. Anything in the DSI box has a service number and a support queue: open the incident directly in that queue. A ticket in the wrong queue is re-routed by hand and costs hours at the worst possible moment.

graph TB
    User([EPFL user · VPN]) -->|HTTPS| Route

    subgraph ours["Our workload — ENAC-IT, no DSI ticket"]
        Route[OpenShift Route · TLS edge]
        FE[Frontend · Quasar SPA]
        BE[Backend · FastAPI + worker]
        Route -->|/| FE
        Route -->|/api| BE
        FE -->|XHR + auth cookie| BE
    end

    subgraph dsi["EPFL DSI — open the ticket in the queue"]
        K8s[OpenShift / Kubernetes<br/>SVC1219 · SI_KUBERNETES]
        PG[(PostgreSQL DBaaS<br/>SVC1757 · SI_POSTGRESQL)]
        S3[(Object storage S3<br/>SVC1057 · ticket via 1234)]
        Entra[(Entra ID / AD<br/>SVC0026 · SI_AD)]
    end

    subgraph other["Other dependencies — no DSI queue"]
        Infi[ENAC-IT Infisical<br/>secrets, all environments]
        Tab[EPFL Tableau<br/>travel + headcount]
        ECB[European Central Bank<br/>exchange rates]
        GH[GitHub + Quay<br/>CI, images, GitOps]
        Obs[ENAC-IT observability<br/>Tempo, Icinga, GlitchTip]
        ES[(EPFL OPDo Elasticsearch<br/>audit records)]
    end

    K8s ==>|hosts| Route
    GH ==>|ArgoCD sync| K8s
    K8s -.->|External Secrets Operator pulls| Infi
    BE -->|SQL| PG
    BE -->|S3 API · ingestion files| S3
    BE -->|OIDC code exchange| Entra
    User -.->|login redirect| Entra
    BE -->|scheduled pull| Tab
    BE -->|scheduled pull| ECB
    BE -.->|OTLP| Obs
    BE -->|audit sync| ES
Dependency Service Queue Open a ticket when
OpenShift / Kubernetes SVC1219 SI_KUBERNETES Pods will not schedule, Route down, wildcard certificate, cluster-wide failure
PostgreSQL (DBaaS) SVC1757 SI_POSTGRESQL Database unreachable, restore or point-in-time recovery request
Object storage (S3) SVC1057 1234@epfl.ch + bucket name Bucket unreachable, S3 credentials rejected
Entra ID / Active Directory SVC0026 SI_AD Login broken, group or role claims missing

Numbers and queues are as supplied by EPFL DSI. The linked service page carries the current service manager; read it there rather than trusting a name copied into a doc.

Object storage has no published SI_ queue. Open the ticket at 1234@epfl.ch and give the bucket name: it is S3_BUCKET in the backend secret, set from the GitOps overlay. Buckets are ordered and managed through the VPSI XaaS portal (https://portal-xaas.epfl.ch, FAQ).

Infisical, Tableau, the ECB feed, GitHub and the ENAC-IT observability stack are not DSI services; an SI_ ticket for them goes nowhere. Contact ENAC-IT (roster in the DRP) for Infisical, Tempo, Icinga and GlitchTip; the others are vendor or data-source issues.

Ours: symptom, cause, action

Every alert in openshift-app-config carries a runbook_url that lands here, or on which knob, when an alert fires for the DB rules. Keep both headings as they are, or update the three overlays in the same change.

Symptom Usual cause Do
CreateContainerConfigError or CrashLoop after a rollout A secret key is missing or a setting is invalid oc get externalsecret -n <ns> and oc describe pod; fix the key in Infisical or the overlay, then oc rollout restart deploy/co2-calculator-backend
ImagePullBackOff Tag not in Quay yet, or the pull robot credential expired Check the GitHub Actions run, then the quay-reg-cred ExternalSecret
DbPoolSaturationHigh, then DbPoolCheckoutTimeout DB layer 1: this pod's pool (DB_POOL_SIZE + DB_MAX_OVERFLOW) is full. Behind the bouncer a backend pod should have scaled out; on the worker, jobs are queuing on connections Check the HPA, then the Grafana DB pool panel; which knob; read plans 2566 and 1723 before touching pool sizes
DbBouncerQueued, then DbBouncerQueueTimeout DB layer 2: more than 60 (dev) or 70 (stage, prod) transactions in flight at once filled PgBouncer's server pool (transaction pooling on all three since 2026-09-24). Reference event 2026-09-17 17:07 to 17:13, under the old 25-slot session pool and 120 s wait: total open crossed 25, fresh logins queued 120 s while holding pod pool slots, backend pools hit 100%, 5 s pool timeouts, /ready red, a 503 band. At today's 10 s wait the same overrun is a short error band, not a lockup Nothing local heals it. Stray client first (application_name in pg_stat_activity), then a long transaction holding slots (COPY factor upsert, long routes), then a bigger default_pool_size from DBaaS; which knob
DbServerConnectionSlotsExhausted DB layer 3: Postgres itself is full (53300). Orphaned backends from dead pods, or a laptop on the shared DB. The 2026-08-31 mode: everyone is locked out, pgAdmin and migrations included Count orphans: select state, count(*), max(now()-state_change) from pg_stat_activity where usename='app' group by state; pg_terminate_backend the idle ones; SI_POSTGRESQL if max_connections must grow; which knob
Many users get 403 at once Role sync received an empty list from the provider Plan 2531; check /api/health/deps for the role provider
AccredUnavailable, logins land on "EPFL Accred is unavailable" api.epfl.ch answered 5xx, timed out or was unreachable (upstream, nothing here to restart). The status_code label tells it apart: 401/403 is our API key rejected, 429 is rate-limiting, unconfigured is missing Accred settings Confirm with curl -u <user>:<key> https://api.epfl.ch/v1/persons/<id> or the EPFL IT status page; for 401/403 rotate ACCRED_API_KEY in Infisical; RoleSyncProviderUnavailable co-fires and clears on its own; /api/health/deps
Upload or recalculation never finishes Stuck job GET /api/health/stale-stats; plans 1219 and 1559
HighErrorRate, LatencyP99High A slow or failing route Grafana errors by status code, then Tempo for the slowest traces on that route
BackendMetricsAbsent Backend pods down, or the OTel collector pod down oc get pods -n <ns>; the collector is otel-collector
Icinga ssl-cert warning Cluster wildcard certificate nearing expiry Not ours: SI_KUBERNETES
Icinga pgsql CRITICAL, app still up Probe path to DBaaS, or DBaaS maintenance Check /api/health/deps; if the app is failing too, SI_POSTGRESQL

Recovery

  • Bad deploy: revert the GitOps commit. ArgoCD reconciles within minutes.
  • Database: nightly dump on the db-dumps PVC. Restore: the step-by-step runbook and one-shot script in the private ops README (exercised for real on 2026-09-08, see the post-mortem); point-in-time recovery is a SI_POSTGRESQL ticket.
  • Files: not recoverable, by decision. Ask the user to re-upload.
  • Anything larger: the Disaster Recovery Plan, linked at the top.

After the incident

Follow Incident Response § After the incident. Write the postmortem as a page in this folder, in the shape of the OAuth redirect postmortem, and add it to the nav in the same PR.