# Incident-Response Runbook This runbook is the canonical incident-response (IR) playbook for the platform. It covers severity classification, the six-phase IR workflow, the GDPR 72-hour breach-notification workflow, and the communication templates used during and after an incident. It complements (does not replace) the existing [`runbook.md`](./runbook.md), which documents the **security-header and CORS contract**. When `runbook.md` references "the IR runbook" or "the security-team", this document is what it means. > Cross-references: > - [`threat-model.md`](./threat-model.md) — STRIDE matrix referenced from "Detection" > - [`vulnerability-disclosure.md`](./vulnerability-disclosure.md) — researcher-side flow > - [`pen-test-scope.md`](./pen-test-scope.md) — engagement contract > - [`sbom.md`](./sbom.md) — dependency-CVE forensics ## 1. IR-team roster and contacts The operator maintains the live roster in their secure password manager and on the operator status page. The roles below are required for every incident: | Role | Responsibility | Channel | | --- | --- | --- | | **IR Lead** | Owns the incident end-to-end; approves all external communication | `security@`, on-call paging | | **Engineering Lead** | Drives containment + eradication; owns the technical fix | `oncall@` | | **Communications Lead** | Owns customer + regulator + press messaging | `comms@` | | **Data-Protection Officer (DPO)** | Owns GDPR Article 33 / 34 decisions and filings | `dpo@` | | **Legal Counsel** | Reviews customer + regulator + press communication; signs the breach-decision record | `legal@` | | **Scribe** | Captures timeline, decisions and evidence into the incident channel | rotated per incident | For **P0 / Sev-1** incidents the IR Lead and Engineering Lead must both be paged within 15 minutes of detection. For P1 / Sev-2 within 1 hour. P2 within 4 hours. P3 next business day. ## 2. Severity classification | Severity | Examples | Response SLA | Pages | | --- | --- | --- | --- | | **P0 / Sev-1** | Confirmed data breach with PII exfiltration. Production-down across all tenants. Active exploit of a known vulnerability. Cryptographic-key compromise. Backup-restore failure during a recovery operation. | Acknowledge ≤ 15 min, IR-team assembled ≤ 30 min | IR Lead + Engineering Lead + DPO + Comms | | **P1 / Sev-2** | Confirmed credential compromise scoped to one tenant. Strong suspicion of breach with intact-but-elevated-risk evidence. Service degradation affecting > 25 % of tenants. Audit-verifier chain failure on a recent window. Outbound webhook leaking to an unintended URL. | Acknowledge ≤ 1 h, IR-team assembled ≤ 2 h | IR Lead + Engineering Lead | | **P2 / Sev-3** | Suspicious activity that has not yet escalated to "compromise". Failed audit check on an isolated record. Permission-matrix regression caught in staging. Single-tenant performance regression without data risk. | Acknowledge ≤ 4 h, IR-team assembled within 1 business day | IR Lead | | **P3 / Sev-4** | Low-risk finding from a security researcher (no exploit path). Documentation gap. Operator-procedural drift. | Acknowledge ≤ 1 business day | IR Lead (async) | Severity is set by the IR Lead based on the evidence at detection time and **may be raised** during the incident; downgrades require Engineering Lead + DPO concurrence. ## 3. Six-phase IR workflow Every incident, regardless of severity, follows the same six phases. P2/P3 incidents may compress steps but must produce the same record. ### Phase 1 — Detection Sources of detection (non-exhaustive): - Sentry alert (especially `severity >= error` with `pii.scrubbed=false` tags) - Audit-verifier nightly job (`compliance-audit-export` chain-validation) - Metrics alert (login failure spike, queue depth spike, AI cost spike, webhook-delivery dead-letter spike) - User / customer report via support or status-page - Researcher report via [`vulnerability-disclosure.md`](./vulnerability-disclosure.md) - Provider notification (Stripe / Mailjet / OpenAI / Bunny / Sentry / Authentik) - IR team review of the [`threat-model.md`](./threat-model.md) residual-risk list **Action**: open an incident in the operator's incident-tracker, page the IR Lead, set provisional severity, and start the timeline log. ### Phase 2 — Triage The IR Lead, within the severity SLA: 1. Confirms or revises the severity 2. Confirms the affected surface (use [`threat-model.md`](./threat-model.md) component list) 3. Identifies whether GDPR Article 33 may apply (see [§5](#5-gdpr-72-hour-breach-notification-workflow)) 4. Identifies whether customer notification is required (Article 34) 5. Identifies the on-call IR-team members and pages them 6. Starts the dedicated incident Slack channel (`#incident--`) 7. Locks down evidence: snapshot Sentry events, AuditEvent rows, metrics windows, relevant logs **Output**: triage record with severity, scope, GDPR-relevance, IR-team roster, evidence pointers. ### Phase 3 — Containment Goal: stop the bleeding without destroying evidence. Tactical levers (in roughly increasing impact): - Disable affected **feature flag** (see `packages/feature-flags`) — narrowest blast radius - Disable affected **route** at API or Web layer via temporary middleware (operator change) - Disable affected **worker queue group** (`docs/ops/worker-queues.md`) - Disable **outbound webhook delivery** (operator killswitch documented in `docs/architecture/outbox-pattern.md`) - **Invalidate sessions** for affected users (mass-revoke via DB script; documented in `docs/operations/database-rollback.md`) - **Rotate API keys / JWT secret / Stripe webhook secret** (rotation runbook in operator password manager) - **Block IP ranges** at WAF / Coolify edge - **Take service offline** behind status-page maintenance — last resort For each lever applied: record actor, timestamp, lever, scope, expected impact, observed impact in the incident channel. ### Phase 4 — Eradication Goal: remove the root cause. - Root-cause analysis (RCA) using AuditEvent hash-chain forensics — when [`compliance-audit-export`](../improvement-plan/10-compliance-polish/compliance-audit-export.md) lands, the verifier is the canonical forensics source; until then the operator uses Sentry traces + DB audit rows + metrics - Code fix authored on a feature branch, reviewed by Engineering Lead - Tests added (regression + non-regression) — coverage gate enforced - Patch deployed (Coolify auto-deploy or manual `prisma:migrate:deploy`) - If the root cause is a leaked secret: rotate, force re-issue, document in the rotation log ### Phase 5 — Recovery Goal: return to normal operation, verified. - Re-enable feature flags / routes / queue groups disabled in containment - Validate via metrics + Sentry that the error signal is gone - Validate via synthetic / E2E (where the operator has explicitly approved E2E for this session) that the customer-facing path works - Update the status page to "resolved" with the timeline summary - For credential rotations: confirm 0 active sessions still bound to the rotated secret ### Phase 6 — Lessons learned Goal: institutional memory. Within **5 business days** of recovery, the IR Lead publishes a post-mortem at: ``` docs/security/incidents/-.md ``` (See [§9](#9-post-incident-review-template) for the template. The `incidents/` directory is created lazily — first incident is responsible for creating it with a `.gitkeep`.) The post-mortem is reviewed at the next IR-team weekly. Action items are tracked in the improvement plan with phase + commit category. The [`threat-model.md`](./threat-model.md) is updated if a new control class was added or a residual risk needs to be re-classified. ## 4. Forensics Forensics relies on three sources of truth, in order of preference: 1. **AuditEvent hash-chain** (canonical; tamper-evident once [`compliance-audit-export`](../improvement-plan/10-compliance-polish/compliance-audit-export.md) lands its verifier) 2. **Sentry events + breadcrumbs** (PII-scrubbed; subject to retention) 3. **Metrics + logs** (high-volume; subject to retention + sampling) **Evidence rules**: - Snapshot **before** changing anything. The triage step creates the evidence pointer record. - Do not paste raw secrets / customer PII / tenant identifiers / production tokens into the incident channel. Use generic labels per the rules in [`SECURITY.md`](../../SECURITY.md) "Safe evidence". - Preserve operator audit-trail (admin actions during the incident are themselves auditable). - For DB-level evidence: prefer `pg_dump --table` of a single table + WHERE clause over `SELECT` into the channel. Encrypt and attach to the incident record. ## 5. GDPR 72-hour breach-notification workflow This workflow applies whenever a confirmed or suspected breach involves personal data of EU / UK / EEA data subjects. ### 5.1 Decision flow ```mermaid flowchart TD Detection["Confirmed breach
(Article 4(12))"] Q1{"Is risk to rights and
freedoms unlikely?"} Q2{"Is risk high?"} NoNotify["Document decision;
no notification required"] Notify33["Notify supervisory authority
within 72 h (Art. 33)"] Notify34["Notify data subjects
without undue delay (Art. 34)"] Detection --> Q1 Q1 -- "yes" --> NoNotify Q1 -- "no" --> Notify33 Notify33 --> Q2 Q2 -- "yes" --> Notify34 Q2 -- "no" --> Done["Done — keep record"] Notify34 --> Done ``` The **DPO owns the decision** at each branch and records the rationale in writing. The Legal Counsel signs off before any external notification leaves the operator. ### 5.2 72-hour clock The 72-hour clock starts when the operator becomes **aware** of a breach (Article 33(1)). "Aware" means reasonable certainty that a breach affecting personal data has occurred — not certainty about scope or attribution. | Hour | Required state | | --- | --- | | H+0 | Incident opened, IR-team paged, DPO notified | | H+4 | Initial scope assessment — affected categories of data subjects, categories of data, approximate volume | | H+24 | Containment in place or documented infeasibility | | H+48 | Article 33 notification drafted and Legal-Counsel reviewed | | H+72 | Article 33 notification filed with the competent supervisory authority | If the operator cannot meet H+72 with complete information, file an initial notification with the available facts and state the reason for the delay (Article 33(4)). ### 5.3 Supervisory-authority filing For German tenants the default supervisory authority is the **competent state DPA** (LDI) of the tenant's registered location, or the **Federal Commissioner for Data Protection and Freedom of Information (BfDI)** when federal jurisdiction applies. The DPO maintains the current authority mapping in the operator password manager. The filing must include (Article 33(3)): - Nature of the breach, categories and approximate numbers of data subjects and records concerned - Name and contact details of the DPO or other contact point - Likely consequences - Measures taken or proposed to address the breach and mitigate adverse effects ### 5.4 Article 34 data-subject notification Required when the breach is likely to result in a **high risk** to data subjects (Article 34(1)). Not required if: - Encryption / pseudonymisation renders the data unintelligible (Article 34(3)(a)) - Subsequent measures ensure the high risk is no longer likely (Article 34(3)(b)) - It would involve disproportionate effort — in which case a public communication is made instead (Article 34(3)(c)) Notification template: see [§7.2](#72-customer-data-subject-breach-notification). ### 5.5 Breach-decision record (BDR) Every breach — notified or not — produces a BDR stored at: ``` docs/security/incidents/-/breach-decision-record.md ``` The BDR captures: scope assessment, Article 33 decision + rationale, Article 34 decision + rationale, DPO signature, Legal-Counsel signature, IR-Lead signature, supervisory-authority filing reference number (if filed), customer-notification dispatch reference (if dispatched). The BDR is **not** published; it is retained for the operator's GDPR-compliance retention period. ## 6. Mass customer notification workflow When Article 34 (or operator policy) requires mass notification: 1. The Comms Lead drafts the message from the template ([§7.2](#72-customer-data-subject-breach-notification)) 2. Legal Counsel reviews 3. IR Lead approves 4. The Engineering Lead enqueues a **dedicated mail batch** through the worker mail queue with the **CRITICAL** priority (see `docs/architecture/job-priority.md`) 5. The mail batch is sent to the **affected data-subject cohort** only — selection script reviewed by Engineering Lead, peer-reviewed by IR Lead, stored as evidence 6. The send is monitored: per-second send rate kept within Mailjet reputation thresholds; bounce + complaint webhooks watched live 7. Status-page notice published before send-start For breaches affecting > 10 000 data subjects, the operator may instead publish a public communication on the operator's security page (Article 34(3)(c)) — Legal Counsel signs off. ## 7. Communication templates Templates are intentionally generic — they are filled in by the Comms Lead at incident time. **Do not** include real customer names, tenant identifiers, or production credentials in the live message. ### 7.1 Internal team — incident open ``` Subject: [INCIDENT][P] Severity: P Incident channel: #incident-- IR Lead: Engineering Lead: DPO involved: Status: triage Summary - What we know: - What we don't know: - What we're doing now: - Next checkpoint: