Back to overview
Resolved

Postmortem: Loss of Ingress on Pangolin Cloud

Nov 4, 2026 at 7:04pm UTC
Affected services
HTTP Ingress

Resolved
Nov 4, 2026 at 7:04pm UTC

Incident Summary
On November 4, from 16:29 to 16:58 UTC, the website experienced a 29-minute loss of ingress on Pangolin Cloud nodes. During this window, the system was unable to process incoming requests.

Timeline

16:29 UTC: The v1.26 update was initiated, immediately resulting in a loss of ingress and dropped incoming requests.

16:58 UTC: The namespace configuration was corrected, restoring Traefik's ability to process requests.

Root Cause
The outage was caused by a sequencing error during the recent 1.26 update. Gerbil containers were updated without simultaneously updating Traefik, which shares a network namespace with Gerbil. Because they were updated out of sync, Traefik became orphaned on the old namespace and lost its ability to route incoming traffic.

Resolution and Action Items
The monitoring systems swiftly detected the orphaned namespace, and engineering intervened to manually resolve the routing failure. To prevent this from occurring in the future, the deployment update scripts have been successfully patched to ensure Traefik and Gerbil are updated together and maintain their shared network namespace.