Incidents and maintenance reported on status page for Pangolin https://status.pangolin.net/ https://d1lppblt9t2x15.cloudfront.net/logos/847b93693b42a1f69a73044c8420ace1.png https://status.pangolin.net/ en https://status.pangolin.net/incident/1049043 Thu, 03 Sep 2026 14:57:00 -0000 https://status.pangolin.net/incident/1049043#9d785bf5846feff7d5e069c7ac3e02504ec2589142d38fc37c88cc5306535f34 Incident ## Postmortem: ElastiCache Routing Misconfiguration & Multi-Region Outage **Date:** September 2, 2026 **Incident Duration:** ~6 hours 30 minutes **Impact:** Partial service degradation across EU and US-West regions due to primary cache inaccessibility. ### Executive Summary On September 2, 2026, a scheduled ElastiCache service upgrade triggered a failover of Pangolin’s primary cache node to a replica located within a newly provisioned VPC subnet. A routing table misconfiguration prevented traffic from the **EU** and **US-West** regions from reaching this new subnet. The sudden loss of cache access caused thread pool starvation and cascading service failures across downstream applications. Service was fully restored after updating cross-region routing rules and recycling connection pools across affected clusters. ### Root Cause Analysis 1. **Subnet Provisioning Gap:** The maintenance event failed over the ElastiCache primary to a replica inside a newly provisioned subnet. 2. **Missing Routing Entries:** Cross-region VPC peering routes and security group rules for EU and US-West were not updated to include the cidr block of the newly provisioned subnet. 3. **Cascading Failure:** Downstream microservices lacked resilient circuit breakers for cache unavailability. Unhandled connection timeouts quickly exhausted application worker threads, causing cascading service crashes. 4. **Stale Connection Persistence:** Even after routing was restored, persistent TCP connection pools in certain client nodes continued trying to reach the old unreachable endpoints until services were manually restarted. ### Impact Assessment * **Regions Affected:** EU and US-West regions experienced severe latency and request failures. US-East remained largely functional due to direct local routing. * **Cache Layer:** Read/write operations to the primary ElastiCache cluster failed completely for impacted regions. * **Client/Node Connections:** Certain client services required explicit service restarts to drop stale sockets and re-establish active connections to the updated cluster endpoint. ### Corrective Actions & Preventative Measures #### Completed * **Route Remediation:** Updated all cross-region VPC routing tables and security groups to allow full traffic access to the new ElastiCache subnet. * **Connection Refresh:** Executed targeted pod and service restarts across EU and US-West deployments to clear dead TCP sessions. * **Customer Advisory:** Published status updates advising self-hosted nodes and client connections to restart lingering services. #### Planned * **Infrastructure as Code (IaC) Enforcement:** Update CDK subnet modules to automatically update all cross-region peering route tables whenever new subnets are created. * **Pre-Flight Failover Audits:** Implement automated integration tests that validate cross-region connectivity across all standby cache nodes prior to scheduling upgrades. * **Graceful Cache Degradation:** Implement circuit breakers and fallback mechanisms so that cache connectivity loss degrades performance gracefully rather than causing total service failures. https://status.pangolin.net/maintenance/920356 Wed, 10 Jun 2026 23:48:06 -0000 https://status.pangolin.net/incident/920356#dcbbabc42ad141eae3903eeca9ddb08356af10b8f60f6900d7a1600c70ea8318 Maintenance The Pangolin dashboard may experience slowdowns or momentarily changes in operation for around 30-60 minutes while the team performs system updates. Resources should continue to operate as expected. https://status.pangolin.net/incident/896245 Thu, 14 May 2026 22:12:00 -0000 https://status.pangolin.net/incident/896245#437661d0482da22fad48aed292c97bb51e9e343b2fc70bbcdef38972ef12e525 Incident ## Postmortem: Client Registration Disruption **Date:** May 14, 2026 **Incident Duration:** ~3 hours **Impact:** Intermittent failure of client registration and connection establishment across US-East regional Points of Presence (PoPs). --- ### Executive Summary On May 14, 2026, the **US-East region** experienced a degradation in connection processing. A surge in holepunch requests, stemming from suboptimal code execution paths, led to resource exhaustion within the supporting infrastructure. This created a bottleneck in source address verification, preventing clients from successfully completing the registration handshake. Service was restored following targeted code optimizations and an expedited infrastructure scaling event. ### Incident Timeline * **T-00:00:** Observed sharp increase in holepunch request volume at US-East regional PoPs. * **T+00:15:** Cache and database infrastructure report elevated latency and resource utilization. * **T+00:30:** Inability to verify client source addresses leads to widespread registration failures. * **T+01:45:** Engineering identifies specific code inefficiencies contributing to the request volume. * **T+02:30:** **Resolution:** Optimized logic is deployed; supplementary infrastructure capacity is provisioned. * **T+03:00:** Connectivity metrics return to baseline; all regional systems confirmed operational. ### Root Cause Analysis The disruption was the result of a **resource exhaustion cascade** triggered by internal logic inefficiencies: 1. **Request Volume Anomalies:** Suboptimal code paths generated a disproportionate volume of holepunch requests relative to standard site traffic. 2. **Downstream Pressure:** The high frequency of these requests overwhelmed the regional database and caching layers, leading to delayed response times. 3. **Verification Bottleneck:** The system requires successful holepunch completion to validate client source addresses for security and functionality. Due to infrastructure timeouts, this validation could not be completed. 4. **Registration Inhibition:** Failure to verify source addresses prevented the system from authorizing new client connections. ### Impact Assessment * **Client Connectivity:** Users in the US-East region were unable to register or establish stable connections. * **Infrastructure:** Database and cache utilization reached critical thresholds * **Verification Logic:** The source address validation mechanism was temporarily unable to process inbound requests, leading to a "fail-closed" state for new connections. ### Corrective Actions & Preventative Measures #### Completed * **Logic Optimization:** Refined the code responsible for holepunch request handeling to reduce unnecessary overhead. * **Capacity Expansion:** Scaled the regional database and cache clusters to better accommodate peak request volumes. https://status.pangolin.net/maintenance/856941 Wed, 25 Mar 2026 03:55:35 -0000 https://status.pangolin.net/incident/856941#978646c989f9f5a9b78e8828bf30d53f502941b3a2bbf62038f1989c6fd6a733 Maintenance The Pangolin dashboard will be unavailable for around 30 minutes while the team performs system updates. Resources should continue to operate as expected. Remote nodes may experience downtime. https://status.pangolin.net/incident/853774 Fri, 20 Mar 2026 17:29:00 -0000 https://status.pangolin.net/incident/853774#a75b9ec63cbb865f085b0fc76977f1cc73ab79a66da38443972163fc9a3caac8 Incident ## Postmortem: EU Cloud Region Service Disruption **Date:** March 20, 2026 **Incident Duration:** ~2:00 AM PST – 9:00 AM PST **Impact:** Total service outage for the EU Cloud Region. Exit nodes remained functional, but downstream resources were inaccessible. ### Executive Summary On March 20, 2026, at approximately 2:00 AM PST, the **EU Cloud Region** experienced a total service outage. The failure originated from database write timeouts to the **US-East region**, which triggered a cascading failure within the Kubernetes clusters. Lack of region-specific monitoring delayed initial detection. Service was restored by forcing a full redeployment of the affected services. ### Incident Timeline * **02:00 PST:** Database write timeouts begin between EU region and US-East database. * **02:15 PST:** High concentration of write failures causes Kubernetes health checks to fail. * **02:20 PST:** Pods enter rapid restart cycles. * **03:00 – 07:00 PST:** Pods enter `CrashLoopBackOff` state; Kubernetes back-off timers increase until containers cease running. * **08:00 PST:** Engineering team identifies the regional failure despite status page silence. * **09:00:** **Resolution:** A forced redeployment of all EU pods is executed. * **09:10:** Traffic flow resumes; service is confirmed healthy. ### Root Cause Analysis The outage was caused by a **cascading failure** initiated by cross-region latency or connectivity issues: 1. **Dependency Failure:** The EU region relies on the US-East database for write operations. Timeouts here caused application threads to hang. 2. **Health Check Sensitivity:** Kubernetes liveness/readiness probes were tied to database connectivity. When the writes failed, the probes failed, killing the pods. 3. **The Death Spiral:** Rapid restarts and subsequent `CrashLoopBackOff` states prevented the system from recovering automatically once the database became reachable again. 4. **Monitoring Blind Spot:** The monitoring stack was configured primarily for US regions. Consequently, the EU-specific failure did not trigger the automated alerting system or update the public status page. ### Impact Assessment * **Services:** All EU-hosted applications were offline. * **Exit Nodes:** Not directly impacted. However, because the backend resources they serve were down, sites reported as "offline" to end-users. * **Internal Visibility:** Status pages incorrectly reported "All Systems Operational" during the event due to the lack of regional monitoring granularity. ### Corrective Actions & Preventative Measures #### Completed * **Alerting Update:** Monitoring has been expanded to include all specific geographic regions (including EU). * **Notification Logic:** Media alerts are now configured to trigger for regional-specific failures, ensuring engineers are notified regardless of US-region health. #### Planned * **Resiliency Tuning:** We will adjust the `CrashLoopBackOff` and restart policies in Kubernetes to ensure containers continue to attempt restarts more gracefully during prolonged outages. * **Database Deep Dive:** A secondary deep-root-cause analysis is underway to determine why the cross-region database writes timed out initially. * **Circuit Breaking:** Implementing circuit breakers to prevent database timeouts from failing pod health checks entirely.