HiveGrid-Wireframes/hivegrid-cloud/docs/security/incident-response.md

# Incident-Response Runbook

This runbook is the canonical incident-response (IR) playbook for the platform. It covers severity classification, the six-phase IR workflow, the GDPR 72-hour breach-notification workflow, and the communication templates used during and after an incident.

It complements (does not replace) the existing [`runbook.md`](./runbook.md), which documents the **security-header and CORS contract**. When `runbook.md` references "the IR runbook" or "the security-team", this document is what it means.

> Cross-references:
> - [`threat-model.md`](./threat-model.md) — STRIDE matrix referenced from "Detection"
> - [`vulnerability-disclosure.md`](./vulnerability-disclosure.md) — researcher-side flow
> - [`pen-test-scope.md`](./pen-test-scope.md) — engagement contract
> - [`sbom.md`](./sbom.md) — dependency-CVE forensics

## 1. IR-team roster and contacts

The operator maintains the live roster in their secure password manager and on the operator status page. The roles below are required for every incident:

| Role | Responsibility | Channel |
| --- | --- | --- |
| **IR Lead** | Owns the incident end-to-end; approves all external communication | `security@<your-domain>`, on-call paging |
| **Engineering Lead** | Drives containment + eradication; owns the technical fix | `oncall@<your-domain>` |
| **Communications Lead** | Owns customer + regulator + press messaging | `comms@<your-domain>` |
| **Data-Protection Officer (DPO)** | Owns GDPR Article 33 / 34 decisions and filings | `dpo@<your-domain>` |
| **Legal Counsel** | Reviews customer + regulator + press communication; signs the breach-decision record | `legal@<your-domain>` |
| **Scribe** | Captures timeline, decisions and evidence into the incident channel | rotated per incident |

For **P0 / Sev-1** incidents the IR Lead and Engineering Lead must both be paged within 15 minutes of detection. For P1 / Sev-2 within 1 hour. P2 within 4 hours. P3 next business day.

## 2. Severity classification

| Severity | Examples | Response SLA | Pages |
| --- | --- | --- | --- |
| **P0 / Sev-1** | Confirmed data breach with PII exfiltration. Production-down across all tenants. Active exploit of a known vulnerability. Cryptographic-key compromise. Backup-restore failure during a recovery operation. | Acknowledge ≤ 15 min, IR-team assembled ≤ 30 min | IR Lead + Engineering Lead + DPO + Comms |
| **P1 / Sev-2** | Confirmed credential compromise scoped to one tenant. Strong suspicion of breach with intact-but-elevated-risk evidence. Service degradation affecting > 25 % of tenants. Audit-verifier chain failure on a recent window. Outbound webhook leaking to an unintended URL. | Acknowledge ≤ 1 h, IR-team assembled ≤ 2 h | IR Lead + Engineering Lead |
| **P2 / Sev-3** | Suspicious activity that has not yet escalated to "compromise". Failed audit check on an isolated record. Permission-matrix regression caught in staging. Single-tenant performance regression without data risk. | Acknowledge ≤ 4 h, IR-team assembled within 1 business day | IR Lead |
| **P3 / Sev-4** | Low-risk finding from a security researcher (no exploit path). Documentation gap. Operator-procedural drift. | Acknowledge ≤ 1 business day | IR Lead (async) |

Severity is set by the IR Lead based on the evidence at detection time and **may be raised** during the incident; downgrades require Engineering Lead + DPO concurrence.

## 3. Six-phase IR workflow

Every incident, regardless of severity, follows the same six phases. P2/P3 incidents may compress steps but must produce the same record.

### Phase 1 — Detection

Sources of detection (non-exhaustive):

- Sentry alert (especially `severity >= error` with `pii.scrubbed=false` tags)
- Audit-verifier nightly job (`compliance-audit-export` chain-validation)
- Metrics alert (login failure spike, queue depth spike, AI cost spike, webhook-delivery dead-letter spike)
- User / customer report via support or status-page
- Researcher report via [`vulnerability-disclosure.md`](./vulnerability-disclosure.md)
- Provider notification (Stripe / Mailjet / OpenAI / Bunny / Sentry / Authentik)
- IR team review of the [`threat-model.md`](./threat-model.md) residual-risk list

**Action**: open an incident in the operator's incident-tracker, page the IR Lead, set provisional severity, and start the timeline log.

### Phase 2 — Triage

The IR Lead, within the severity SLA:

1. Confirms or revises the severity
2. Confirms the affected surface (use [`threat-model.md`](./threat-model.md) component list)
3. Identifies whether GDPR Article 33 may apply (see [§5](#5-gdpr-72-hour-breach-notification-workflow))
4. Identifies whether customer notification is required (Article 34)
5. Identifies the on-call IR-team members and pages them
6. Starts the dedicated incident Slack channel (`#incident-<YYYY-MM-DD>-<short-id>`)
7. Locks down evidence: snapshot Sentry events, AuditEvent rows, metrics windows, relevant logs

**Output**: triage record with severity, scope, GDPR-relevance, IR-team roster, evidence pointers.

### Phase 3 — Containment

Goal: stop the bleeding without destroying evidence.

Tactical levers (in roughly increasing impact):

- Disable affected **feature flag** (see `packages/feature-flags`) — narrowest blast radius
- Disable affected **route** at API or Web layer via temporary middleware (operator change)
- Disable affected **worker queue group** (`docs/ops/worker-queues.md`)
- Disable **outbound webhook delivery** (operator killswitch documented in `docs/architecture/outbox-pattern.md`)
- **Invalidate sessions** for affected users (mass-revoke via DB script; documented in `docs/operations/database-rollback.md`)
- **Rotate API keys / JWT secret / Stripe webhook secret** (rotation runbook in operator password manager)
- **Block IP ranges** at WAF / Coolify edge
- **Take service offline** behind status-page maintenance — last resort

For each lever applied: record actor, timestamp, lever, scope, expected impact, observed impact in the incident channel.

### Phase 4 — Eradication

Goal: remove the root cause.

- Root-cause analysis (RCA) using AuditEvent hash-chain forensics — when [`compliance-audit-export`](../improvement-plan/10-compliance-polish/compliance-audit-export.md) lands, the verifier is the canonical forensics source; until then the operator uses Sentry traces + DB audit rows + metrics
- Code fix authored on a feature branch, reviewed by Engineering Lead
- Tests added (regression + non-regression) — coverage gate enforced
- Patch deployed (Coolify auto-deploy or manual `prisma:migrate:deploy`)
- If the root cause is a leaked secret: rotate, force re-issue, document in the rotation log

### Phase 5 — Recovery

Goal: return to normal operation, verified.

- Re-enable feature flags / routes / queue groups disabled in containment
- Validate via metrics + Sentry that the error signal is gone
- Validate via synthetic / E2E (where the operator has explicitly approved E2E for this session) that the customer-facing path works
- Update the status page to "resolved" with the timeline summary
- For credential rotations: confirm 0 active sessions still bound to the rotated secret

### Phase 6 — Lessons learned

Goal: institutional memory.

Within **5 business days** of recovery, the IR Lead publishes a post-mortem at:

```
docs/security/incidents/<YYYY-MM-DD>-<short-id>.md
```

(See [§9](#9-post-incident-review-template) for the template. The `incidents/` directory is created lazily — first incident is responsible for creating it with a `.gitkeep`.)

The post-mortem is reviewed at the next IR-team weekly. Action items are tracked in the improvement plan with phase + commit category. The [`threat-model.md`](./threat-model.md) is updated if a new control class was added or a residual risk needs to be re-classified.

## 4. Forensics

Forensics relies on three sources of truth, in order of preference:

1. **AuditEvent hash-chain** (canonical; tamper-evident once [`compliance-audit-export`](../improvement-plan/10-compliance-polish/compliance-audit-export.md) lands its verifier)
2. **Sentry events + breadcrumbs** (PII-scrubbed; subject to retention)
3. **Metrics + logs** (high-volume; subject to retention + sampling)

**Evidence rules**:

- Snapshot **before** changing anything. The triage step creates the evidence pointer record.
- Do not paste raw secrets / customer PII / tenant identifiers / production tokens into the incident channel. Use generic labels per the rules in [`SECURITY.md`](../../SECURITY.md) "Safe evidence".
- Preserve operator audit-trail (admin actions during the incident are themselves auditable).
- For DB-level evidence: prefer `pg_dump --table` of a single table + WHERE clause over `SELECT` into the channel. Encrypt and attach to the incident record.

## 5. GDPR 72-hour breach-notification workflow

This workflow applies whenever a confirmed or suspected breach involves personal data of EU / UK / EEA data subjects.

### 5.1 Decision flow

```mermaid
flowchart TD
  Detection["Confirmed breach<br/>(Article 4(12))"]
  Q1{"Is risk to rights and<br/>freedoms unlikely?"}
  Q2{"Is risk high?"}
  NoNotify["Document decision;<br/>no notification required"]
  Notify33["Notify supervisory authority<br/>within 72 h (Art. 33)"]
  Notify34["Notify data subjects<br/>without undue delay (Art. 34)"]
  Detection --> Q1
  Q1 -- "yes" --> NoNotify
  Q1 -- "no" --> Notify33
  Notify33 --> Q2
  Q2 -- "yes" --> Notify34
  Q2 -- "no" --> Done["Done — keep record"]
  Notify34 --> Done
```

The **DPO owns the decision** at each branch and records the rationale in writing. The Legal Counsel signs off before any external notification leaves the operator.

### 5.2 72-hour clock

The 72-hour clock starts when the operator becomes **aware** of a breach (Article 33(1)). "Aware" means reasonable certainty that a breach affecting personal data has occurred — not certainty about scope or attribution.

| Hour | Required state |
| --- | --- |
| H+0 | Incident opened, IR-team paged, DPO notified |
| H+4 | Initial scope assessment — affected categories of data subjects, categories of data, approximate volume |
| H+24 | Containment in place or documented infeasibility |
| H+48 | Article 33 notification drafted and Legal-Counsel reviewed |
| H+72 | Article 33 notification filed with the competent supervisory authority |

If the operator cannot meet H+72 with complete information, file an initial notification with the available facts and state the reason for the delay (Article 33(4)).

### 5.3 Supervisory-authority filing

For German tenants the default supervisory authority is the **competent state DPA** (LDI) of the tenant's registered location, or the **Federal Commissioner for Data Protection and Freedom of Information (BfDI)** when federal jurisdiction applies. The DPO maintains the current authority mapping in the operator password manager.

The filing must include (Article 33(3)):

- Nature of the breach, categories and approximate numbers of data subjects and records concerned
- Name and contact details of the DPO or other contact point
- Likely consequences
- Measures taken or proposed to address the breach and mitigate adverse effects

### 5.4 Article 34 data-subject notification

Required when the breach is likely to result in a **high risk** to data subjects (Article 34(1)). Not required if:

- Encryption / pseudonymisation renders the data unintelligible (Article 34(3)(a))
- Subsequent measures ensure the high risk is no longer likely (Article 34(3)(b))
- It would involve disproportionate effort — in which case a public communication is made instead (Article 34(3)(c))

Notification template: see [§7.2](#72-customer-data-subject-breach-notification).

### 5.5 Breach-decision record (BDR)

Every breach — notified or not — produces a BDR stored at:

```
docs/security/incidents/<YYYY-MM-DD>-<short-id>/breach-decision-record.md
```

The BDR captures: scope assessment, Article 33 decision + rationale, Article 34 decision + rationale, DPO signature, Legal-Counsel signature, IR-Lead signature, supervisory-authority filing reference number (if filed), customer-notification dispatch reference (if dispatched). The BDR is **not** published; it is retained for the operator's GDPR-compliance retention period.

## 6. Mass customer notification workflow

When Article 34 (or operator policy) requires mass notification:

1. The Comms Lead drafts the message from the template ([§7.2](#72-customer-data-subject-breach-notification))
2. Legal Counsel reviews
3. IR Lead approves
4. The Engineering Lead enqueues a **dedicated mail batch** through the worker mail queue with the **CRITICAL** priority (see `docs/architecture/job-priority.md`)
5. The mail batch is sent to the **affected data-subject cohort** only — selection script reviewed by Engineering Lead, peer-reviewed by IR Lead, stored as evidence
6. The send is monitored: per-second send rate kept within Mailjet reputation thresholds; bounce + complaint webhooks watched live
7. Status-page notice published before send-start

For breaches affecting > 10 000 data subjects, the operator may instead publish a public communication on the operator's security page (Article 34(3)(c)) — Legal Counsel signs off.

## 7. Communication templates

Templates are intentionally generic — they are filled in by the Comms Lead at incident time. **Do not** include real customer names, tenant identifiers, or production credentials in the live message.

### 7.1 Internal team — incident open

```
Subject: [INCIDENT][P<severity>] <short-headline>

Severity: P<severity>
Incident channel: #incident-<YYYY-MM-DD>-<short-id>
IR Lead: <name>
Engineering Lead: <name>
DPO involved: <yes/no>
Status: triage

Summary
- What we know:
- What we don't know:
- What we're doing now:
- Next checkpoint: <time>

If you are on the IR roster, join the channel. If you are not on the roster but
have data relevant to this incident, post in the channel.

Do not paste customer data, tokens, or credentials. Use the generic-label rules
in docs/security/runbook.md.
```

### 7.2 Customer / data-subject breach notification

```
Subject: Important security notice regarding your <product> account

Dear <Customer / Data subject>,

We are writing to inform you of a security incident that may have affected
information associated with your <product> account.

What happened
On <date>, our security team identified <short factual description>. We took
the following steps to contain the incident: <bullet list>.

What information was involved
The incident may have involved the following categories of information about
your account: <bullet list of categories — e.g. "email address", "encrypted
password hash", "tenant configuration metadata">. We have no evidence that
<categories that were not involved, if known>.

What we are doing
- <containment step>
- <eradication step>
- <recovery step>
- We have <notified / not notified> the competent supervisory authority under
  Article 33 GDPR.

What you can do
- <action 1 — e.g. "rotate any API keys associated with your account">
- <action 2 — e.g. "review your audit log at <URL>">
- <action 3 — e.g. "contact us with any questions at security@<your-domain>">

For more information, please see our security advisory at
https://app.<your-domain>/security/advisories/<advisory-id>.

We are sorry for this incident and for the disruption it may cause. If you have
questions, please reply to this email or write to security@<your-domain>.

The <product> security team
```

### 7.3 Status-page post

```
[INVESTIGATING] <short-headline>
We are currently investigating <generic description>. Customer impact is
<scope description>. We will post an update by <time>.

[IDENTIFIED] <short-headline>
We have identified the cause as <generic description>. We are <action>.
Estimated time to resolution: <time>.

[MONITORING] <short-headline>
A fix has been deployed. We are monitoring metrics. We will post a final
update when we are confident the incident is resolved.

[RESOLVED] <short-headline>
The incident is resolved. A post-mortem will be published at
https://app.<your-domain>/security/advisories/<advisory-id> within 5 business
days.
```

### 7.4 Press statement (P0 only, CEO-approved)

```
On <date>, <company> identified a security incident affecting <generic
description of scope>. We took immediate steps to contain the incident,
notified affected customers, and reported the incident to the competent
supervisory authority as required by GDPR.

We are committed to the security of our customers' data. We are conducting
a full post-incident review and will publish a public advisory at
https://app.<your-domain>/security/advisories/<advisory-id> within five
business days.

For media enquiries: press@<your-domain>.
For customer enquiries: security@<your-domain>.
```

### 7.5 Supervisory-authority filing (Article 33)

The DPO uses the supervisory authority's own form (e.g. BfDI online portal). The form maps to the Article 33(3) requirements listed in [§5.3](#53-supervisory-authority-filing). The filing reference number is recorded in the BDR.

## 8. Drills

The operator runs IR drills **at least every six months**. A drill consists of:

- A synthetic detection event (e.g. a planted Sentry alert against a `pentest:*` tag)
- Real-time response with the full IR-team roster
- Tabletop walk-through of the six-phase workflow
- Article 33 / 34 decision exercise (using a synthetic data-subject cohort)
- Communication-template fill-in (no actual send)
- Drill post-mortem at `docs/security/incidents/drills/<YYYY-MM-DD>-drill.md`

A drill is **not** a substitute for a real incident's post-mortem; it is preparation.

## 9. Post-incident review template

A post-mortem is published at `docs/security/incidents/<YYYY-MM-DD>-<short-id>.md` within 5 business days of recovery. The template:

```markdown
# Incident: <short-headline>

- **Date:** <YYYY-MM-DD>
- **Severity:** P<n>
- **Duration (detection → resolved):** <h:mm>
- **IR Lead:** <name>
- **Engineering Lead:** <name>
- **DPO involved:** <yes/no>
- **Customer impact:** <scope>
- **GDPR Article 33 filed:** <yes/no — reference if filed>
- **GDPR Article 34 sent:** <yes/no — count if sent>

## Summary
<2–3 sentences>

## Timeline
| Time (UTC) | Event |
| --- | --- |

## Detection
How did we find out? Which signal fired? Could we have found out earlier?

## Root cause
Technical root cause + contributing factors. Reference STRIDE component from
threat-model.md if applicable.

## Containment
What did we do, in what order? What worked, what did not?

## Eradication
Code/config change(s) that closed the root cause. Link to commits + PRs.

## Recovery
How did we verify normal operation?

## What went well
- …

## What went badly
- …

## Action items
| # | Action | Owner | Due |
| --- | --- | --- | --- |

## Threat-model updates
Did this incident reveal a new control class or move a residual risk into
"accepted" or "mitigated"? Link to the threat-model.md edit.

## Customer communication
- Article 34 notification: <yes/no, dispatch reference>
- Status-page post: <link>
- Public advisory: <link>
```

## 10. Break-glass procedures

Some operations require break-glass access (e.g. direct DB write, secret rotation outside the rotation cadence, IdP fallback). These are **always P1+** by definition and trigger:

1. Page IR Lead + Engineering Lead + DPO simultaneously
2. Open incident channel before the break-glass action
3. Two-person approval recorded in the channel
4. AuditEvent emitted from the operator-tooling executing the action
5. Post-action review within 24 hours
6. Break-glass-credential rotation within 24 hours

Documented break-glass procedures:

- **Direct DB write** — see `docs/operations/database-rollback.md`
- **Stripe webhook-secret rotation** — operator password-manager runbook
- **JWT-secret rotation** — operator password-manager runbook (mass-revoke required)
- **IdP fallback (Authentik down)** — fall back to local admin credential; documented in `infra/authentik/README.md`
- **Backup restore in production** — `docs/ops/backup-cli.md` `restore` section
- **Tenant-snapshot restore** — `docs/ops/tenant-snapshot.md` `restore` section