2845 — Taxonomy cache broadcast blocked by NetworkPolicy¶
Symptom¶
Every factor_ingest job on dev ended with three httpx.ConnectTimeout
spans on POST http://<pod-ip>:8000/internal/cache/taxonomy/clear, one per
live backend pod, all failing at TCP connect inside the 200 ms budget
(trace b9e7d1…, job 430, 2026-09-17). The job still succeeded: the
broadcast is best-effort and logs
taxonomy cache broadcast: pod … unreachable, then the 60 s TTL takes over.
Root cause¶
helm/templates/network-policies.yaml applies default-deny on Ingress to
every pod and only re-opens the backend port to the OpenShift router
(networkPolicies.ingressFromRoutes). No rule admitted traffic from pods of
the release itself, so a SYN from the worker (or another API pod) to
backend:8000 was dropped. Policies are enabled on dev and stage
(openshift-app-config overlays), off on prod — the broadcast only worked
where nobody looked.
Change¶
helm/templates/network-policies.yaml: newallow-backend-from-podspolicy — ingress tocomponent: backendonbackend.service.targetPortfrom any pod carrying the chart'sselectorLabels(worker and backend alike). Nothing else changes; the router rule stays as is.helm/Makefile:check-network-policiesrenders the chart with policies enabled and fails if the pod-to-pod rule is missing — the regression check for this fix.
No app code changes. The broadcast module already does the right thing once the network lets it through.
Verification¶
make -C helm check-network-policiespasses.- After deploy on dev, run a factor ingest: the worker log no longer prints
the
unreachablewarning and the trace carries noConnectTimeoutspans.
Left open¶
- Whether a 3/3 broadcast failure should stay a warning or raise. Today it is a silent fallback by design (TTL backstop); this incident shows that a permanently dead broadcast is invisible. Decide with the lead; not changed here.
- Prod still has
networkPolicies.enabled: false(see the prod overlay comment). Enabling it is an ops change, tracked there, not in this fix.