# INCIDENT POSTMORTEM & ROOT CAUSE ANALYSIS (RCA)
**Incident Title:** Q4-INC-842: Production API Gateway 502 Bad Gateway Outage
**Date of Incident:** November 5, 2026
**Incident Severity:** SEV-1 (Critical Outage)
**Incident Commander:** Sarah Jenkins (Staff Reliability Engineer)
**Lead Investigator:** Michael Zhang (Principal Backend Engineer)
**Status:** Completed & Remediated
---
### 1. EXECUTIVE SUMMARY
On November 5, 2026 between 14:12 UTC and 14:48 UTC (duration: 36 minutes), our core API gateway experienced elevated 502 Bad Gateway responses affecting 42% of incoming client requests. The issue was triggered by a memory exhaustion deadlock in the connection pool following an unthrottled burst of automated webhook deliveries. Service was fully restored following roll-back of deployment v2.4.2 and scaling connection pool limits. Zero customer data loss occurred.
---
### 2. CUSTOMER & BUSINESS IMPACT
- **Downtime Duration:** 36 minutes (14:12 - 14:48 UTC)
- **Requests Affected:** 142,500 requests failed with HTTP 502 / 504 status codes
- **Customer Impact:** Users were unable to complete document conversions or submit new processing jobs
- **SLA Impact:** Monthly uptime decreased from 99.99% to 99.91% (within 99.9% monthly SLA threshold)
---
### 3. CHRONOLOGICAL INCIDENT TIMELINE (ALL TIMES IN UTC)
| Time | Event Description |
| :---: | :--- |
| **14:05** | Production deployment of release `v2.4.2` begins via GitHub Actions. |
| **14:12** | PagerDuty SEV-1 alert fires: "API Gateway 5xx rate > 5%". |
| **14:15** | Incident Commander opens war room channel `#inc-842-outage` on Slack. |
| **14:22** | Lead Investigator discovers connection pool saturation on database replica. |
| **14:31** | Decision made to execute immediate rollback to stable release `v2.4.1`. |
| **14:42** | Rollback deployment finishes; Kubernetes pods begin health check recovery. |
| **14:48** | Error rate drops to 0.01%; Incident Commander declares all systems operational. |
---
### 4. ROOT CAUSE & THE "FIVE WHYS" ANALYSIS
1. **Why did the API gateway return 502 Bad Gateway?**
Because worker processes hung waiting for available database pool connections until client socket timeouts fired.
2. **Why was the database connection pool exhausted?**
Because release `v2.4.2` introduced an asynchronous event logger that opened a new connection per event without closing it in the finally block.
3. **Why was the unclosed connection not caught in testing?**
Because unit tests ran in mock mode where connection pool thresholds were not simulated under high concurrency.
4. **Why did connection pooling lack automated circuit breakers?**
Because pool saturation health checks were configured as soft warnings rather than blocking readiness probes.
5. **Why was there no automated canary deployment?**
Because canary progressive rollouts had not yet been implemented on the secondary cluster.
---
### 5. WHAT WENT WELL & WHAT WENT WRONG
#### What Went Well:
- PagerDuty monitoring detected the anomaly within 7 minutes of deployment.
- The team executed a clean, zero-data-loss rollback within 17 minutes of diagnosis.
- Customer support status page was updated within 12 minutes of the initial alert.
#### What Went Wrong:
- Integration test suite lacked automated leak detection for database connection handles.
- Rollback took 11 minutes due to slow Docker image pulling on cold node instances.
---
### 6. CORRECTIVE ACTION ITEMS & PREVENTIVE MEASURES
| Item # | Preventive Action Description | Owner | Target Date | Status |
| :---: | :--- | :---: | :---: | :---: |
| ACT-01 | Refactor event logger to use connection pool singleton with automated release | Michael Z. | Nov 8, 2026 | Done |
| ACT-02 | Add automated load-test step in CI simulating 10k concurrent connections | David P. | Nov 12, 2026 | In Progress |
| ACT-03 | Configure Kubernetes readiness probes to fail fast when pool usage >85% | Sarah J. | Nov 15, 2026 | Scheduled |
| ACT-04 | Pre-warm base Docker images on all production worker nodes | David P. | Nov 18, 2026 | Scheduled |