# Incident Postmortem: [INC-XXXX] [Brief Summary of Outage]
**Date:** YYYY-MM-DD
**Incident Commander:** [Name / Handle]
**Severity Level:** SEV-1 / SEV-2 / SEV-3
**Status:** Complete / Action Items In Progress
---
## 1. Executive Summary
Between [HH:MM UTC] and [HH:MM UTC] on [YYYY-MM-DD], [Service Name] experienced an outage impacting [X%] of active users. The root cause was identified as [one-sentence summary of trigger and underlying vulnerability]. Total user-facing downtime was [X minutes].
## 2. Customer Impact
* **Impacted Services:** [e.g., API Gateway, Payment Checkout, User Authentication]
* **User Experience:** [e.g., Users received HTTP 504 Gateway Timeout errors when submitting forms]
* **Support Ticket Volume:** [X] incoming customer inquiries
* **Financial / SLA Impact:** [e.g., SLA breached by 0.05%, refund credits issued]
---
## 3. Incident Timeline (UTC)
* **HH:MM** - Change / trigger deployed: [commit hash or configuration change].
* **HH:MM** - Automated Datadog / Prometheus alert fires: `API Error Rate > 5%`.
* **HH:MM** - On-call engineer [Name] acknowledges alert and initiates incident response.
* **HH:MM** - Incident triage: Rollback of release [v1.2.3] initiated.
* **HH:MM** - Traffic redirected to secondary standby cluster.
* **HH:MM** - Error rates return to baseline 0.01%. Incident resolved.
---
## 4. Root Cause Analysis (The 5 Whys)
1. **Why did the service fail?** The database connection pool exhausted all available sockets.
2. **Why was the pool exhausted?** A new unindexed query on the `invoices` table performed full table scans.
3. **Why was the query slow?** Volume on this table grew 10x following marketing campaign launch.
4. **Why was the query deployed without an index?** The migration script failed silently in staging due to timeout.
5. **Why was staging timeout unmonitored?** Staging CI pipeline lacked strict exit-code assertions for migration steps.
---
## 5. Preventative Action Items (Corrective & Preventive)
| ID | Action Item | Type | Owner | Target Date |
|---|---|---|---|---|
| ACT-01 | Add composite index on `invoices(user_id, status)` | Fix | @dev | YYYY-MM-DD |
| ACT-02 | Enforce strict non-zero exit codes in CI migrations | Prevent | @devops | YYYY-MM-DD |
| ACT-03 | Add connection pool saturation alerts at 80% capacity | Detect | @sre | YYYY-MM-DD |
| ACT-04 | Conduct load testing at 5x peak production traffic | Verify | @qa | YYYY-MM-DD |
---
## 6. Lessons Learned
* **What went well:** Automated rollback script completed in under 4 minutes.
* **Where we got lucky:** The secondary standby database cluster picked up traffic seamlessly.
* **Where we struggled:** Discovering which microservice opened the orphaned connections took 15 minutes of manual log grepping.