# Runbook — Disaster Recovery

Restore service from a major incident: full region outage, irreversible data corruption, ransomware.

## Audience

Platform lead + DPO + on-call.

## Objectives

| Metric                    | Target             |
| ------------------------- | ------------------ |
| RPO (data loss tolerance) | ≤ 1 hour           |
| RTO (time to restore)     | ≤ 4 hours for SEV1 |

## Backups

| Source                             | Frequency                                   | Retention | Where                                             |
| ---------------------------------- | ------------------------------------------- | --------- | ------------------------------------------------- |
| Postgres (`pg_dump` + WAL archive) | Continuous (WAL) + nightly base             | 30 days   | Object storage in a different region              |
| SQL Server (BACPAC)                | Nightly                                     | 30 days   | Object storage                                    |
| S3-compatible blob                 | Versioning enabled + lifecycle to cold tier | 90 days   | Same bucket, replication to a 2nd region for prod |
| Redis                              | None (ephemeral)                            | —         | Rebuild on restart                                |
| `.mcp/.memory`                     | Not backed up                               | —         | Local-only                                        |

## Quarterly drill (mandatory)

Once a quarter:

1. Spin up a clean staging environment.
2. Restore the latest backup.
3. Run the smoke + DSAR + login flows.
4. Record `time to restore`. If it exceeds RTO, write an ADR.

## Procedures

### A — Region outage (no data loss)

1. DNS failover to the standby region.
2. K8s in standby region is warm (pods at minReplicas). Scale up.
3. Re-point object storage clients if not multi-region transparent.
4. Verify health + announce.

### B — Database corruption

1. Stop writes — scale `apiDotnet` + `apiNest` + `workerNest` to 0.
2. Identify the last good point (incident time → WAL position).
3. Restore Postgres to point-in-time using the base backup + WAL replay.
4. Verify `audit_logs` continuity and a sample of `User` + `Document` rows.
5. Bring services back up gradually (1 replica → confirm → scale up).
6. Audit emit: `dr.restore` with metadata `{ rpoSeconds, targetPit }`.

### C — Ransomware / encryption attack

1. Isolate affected nodes (network policy `deny-all`).
2. Do NOT pay.
3. Restore from immutable backups in the parallel region.
4. Engage the breach response playbook ([`docs/compliance/breach-response-playbook.md`](../compliance/breach-response-playbook.md)).
5. After restore, rotate every credential the attacker could have touched.

### D — Accidental destructive operation (truncate / mass delete)

1. Stop writes immediately.
2. Determine scope from `audit_logs`.
3. Restore the affected table(s) to a temp database from PITR backup; copy back the missing rows.
4. Reconcile + audit.
5. Postmortem with an action item to prevent the class (e.g. require admin escalation for mass mutations).

## Communications template

Subject: `[SEV1] LOMA-PORTAL outage — investigating`

Body:

```
We're experiencing a service outage starting <time UTC>.
Investigating; we'll update every 15 minutes.
Status page: <link>.
```

Cadence: every 15 min until stable.

## Contacts (replace with real)

- Platform lead — pager
- DPO — `dpo@loma-portal.example` for breach concerns
- Cloud provider escalation — saved with billing account
- Backup vendor support — saved separately
