# Runbook — Incident Response

When something is broken and users notice (or are about to).

## Audience

On-call rotation. Every engineer rotates through this.

## Severity (declare within 5 min)

| Sev      | Criteria                                                             | Response                                        |
| -------- | -------------------------------------------------------------------- | ----------------------------------------------- |
| **SEV1** | Service down OR data loss OR security breach OR > 50% users affected | Page everyone, 15-min cadence updates, war room |
| **SEV2** | Degraded for a subset OR a critical feature broken                   | On-call + Eng manager, 30-min updates           |
| **SEV3** | Minor — annoying but workable                                        | Track in JIRA/Linear; fix in normal cadence     |

## Roles (assign in the first 5 min)

- **Commander** — runs the response. Often the on-call.
- **Scribe** — keeps the timeline with timestamps + correlation IDs.
- **Comms** — talks to stakeholders. Internal + external.
- **Subject matter expert(s)** — pulled in as needed.

## Steps

### 0–5 min: Detect + declare

1. Acknowledge the page.
2. Confirm via dashboards (`platform/observability/grafana/dashboards/`).
3. Declare severity in the incident channel (`#incident-<id>`).
4. Assign roles.

### 5–30 min: Triage + mitigate

1. Form a hypothesis from recent changes (`git log --oneline --since=2h`).
2. Cheapest mitigation first:
   - Disable the feature flag (if applicable).
   - Roll back the last deploy ([`rollback.md`](rollback.md)).
   - Scale up.
   - Route around (DNS).
3. Communicate ETA to stakeholders.

### 30 min – resolution: Diagnose + fix

1. Use correlation IDs to trace requests through `audit_logs` + traces.
2. Fix the cause (not symptom). Test the fix.
3. Ship via the deploy runbook; preferred to roll forward when possible.

### Verification

- Service back to baseline on the dashboards.
- Sample of failing requests now succeeds.
- No new errors for 15+ min.

### Post-incident (≤ 5 business days)

1. **Blameless postmortem** using `.claude/skills/incident-postmortem/SKILL.md`.
2. Timeline, root cause, impact, action items.
3. Each action item has an owner + a date.
4. If user data was affected → engage [`breach-response-playbook.md`](../compliance/breach-response-playbook.md).
5. If architecture changed → ADR in `docs/decisions/`.

## Communications

| Audience                  | When                           | Channel                                                                        |
| ------------------------- | ------------------------------ | ------------------------------------------------------------------------------ |
| Internal #incident-X      | Every 15–30 min until resolved | Slack/LINE/Teams                                                               |
| Engineering all-hands     | SEV1                           | Email + chat                                                                   |
| Customers                 | Updates per status page        | `https://status.loma-portal.example`                                           |
| Regulator (if PII breach) | Within 72h                     | Per [`breach-response-playbook.md`](../compliance/breach-response-playbook.md) |

## Tools we use

- Logs: Loki (correlation-id filter).
- Traces: Tempo via OTel.
- Errors: Sentry.
- Status page: Atlassian Statuspage or equivalent.
- War room: video bridge — long incidents live on text; bring video when paging.

## Forbidden during an incident

- Doing risky things to "save time" (`--force`, skipping `--atomic`).
- Going dark — over-communicate.
- Blaming individuals — the system let it happen.
