# Storage AV Scanning

Defense-in-depth model for user-uploaded files. Two layers:

- **Layer 1 — Magic-Bytes Pre-Upload Scan**. Synchronous, runs in the upload
  request handler **before** the bytes are committed to the storage backend
  (S3 / local). Validates the actual content type via signature detection and
  rejects mismatches or disallowed types.
- **Layer 2 — ClamAV Post-Upload Scan**. Asynchronous BullMQ job that runs
  after the object is in the bucket. Hits trigger server-side quarantine
  (CopyObject + DeleteObject into a `quarantine/` prefix).

## How it works

The `MagicBytesScanAdapter` in `@toolctrl/storage` reads the first ~4100 bytes
of an upload buffer and resolves its MIME type via the `file-type` library's
signature database. It then validates the detected type against:

1. The client-declared `Content-Type` (rejects mismatches — common attack
   vector: an `.exe` smuggled as `image/png`).
2. A per-context allow-list (rejects e.g. a PDF uploaded into the avatar
   context — see allow-list table below).
3. The filename — control characters (`\x00-\x1f`, `\x7f`) and path traversal
   sequences (`../`, `..\\`) are rejected; filenames over 255 chars are
   rejected.

## Allow-list defaults

See `packages/storage/src/scan/allow-list.ts` for the canonical list.

| Context | Allowed MIME types |
|---|---|
| `avatar` | image/png, image/jpeg, image/webp |
| `mediaLibrary` | + image/gif, image/svg+xml, image/x-icon, image/vnd.microsoft.icon |
| `emailAttachment` | image/{png,jpeg,webp,gif} + application/pdf, text/plain, text/csv, application/zip, Office .docx/.xlsx/.pptx |
| `formUpload` | image/{png,jpeg,webp} + application/pdf |
| `kbAsset` | image/{png,jpeg,webp} + application/pdf |

Per-context override via env: `STORAGE_SCAN_ALLOWLIST_<CONTEXT>` accepts a
comma-separated list of MIME types and **replaces** the default for that
context (not merged). Contexts in env-var form: `AVATAR`, `MEDIA_LIBRARY`,
`EMAIL_ATTACHMENT`, `FORM_UPLOAD`, `KB_ASSET`.

Example: allow only PNG and JPEG for tenant avatars in production:

```
STORAGE_SCAN_ALLOWLIST_AVATAR=image/png,image/jpeg
```

## Notfall-Bypass

Set `STORAGE_SCAN_LAYER1_ENABLED=false` to disable scanning entirely (the
adapter returns `status: "skipped"` for every call). Intended for emergency
incident response only — do not run with scanning disabled in production
for any extended period.

## How callers wire it in

`storeFile` from `apps/web/app/utils/integrations/storageService.ts` accepts
an optional `scan` parameter:

```ts
import { storeFile } from "~/utils/integrations/storageService";
import { auditRejectedUpload } from "~/utils/integrations/storageAudit.server";

await storeFile({
  bucket: "users-icons",
  content: avatarDataUrl,
  id: userId,
  scan: {
    context: "avatar",
    filename: `${userId}.avatar`,
    onReject: auditRejectedUpload(request, "tenant.profile.avatar"),
  },
});
```

A rejected scan throws `StorageScanRejectedError` from `storageService.ts`.
The error carries the bucket, id, filename and full `PreUploadScanResult`
(status, reason, detail, detectedMimeType). Callers can catch it and map
to a 415 response with a user-friendly message.

The `onReject` callback fires before the error throws and is awaited, so the
audit record is persisted even when the route swallows the error.

## Audit events

`auditRejectedUpload(request, context)` writes a `createAdminLog`-style row
with action `upload.rejected.scan`. The serialised detail field contains:

```json
{
  "context": "tenant.profile.avatar",
  "reason": "mime_mismatch",
  "detail": "declared=\"image/png\", detected=\"application/x-msdownload\"",
  "detectedMimeType": "application/x-msdownload"
}
```

The buffer content is never written to the audit log. Only metadata
(bounded reason enum + adapter-level diagnostic string + detected MIME).

## Currently-scanned upload paths

| Route | Context | Buckets |
|---|---|---|
| `apps/web/app/routes/admin/settings/general.tsx` | `mediaLibrary` | branding (logo, dark logo, icon, dark icon, favicon) |
| `apps/web/app/routes/admin/settings/profile.tsx` | `avatar` | users-icons |
| `apps/web/app/routes/app.$tenant/settings/profile.tsx` | `avatar` | users-icons |
| `apps/web/app/routes/app.$tenant/settings/account.tsx` | `avatar` | accounts-icons |
| `apps/web/app/routes/app.$tenant/__portals/portals.$portal/settings.tsx` | `mediaLibrary` | branding (5), seo (2) |
| `apps/web/app/routes/app.$tenant/__widgets/widgets/$id.tsx` | `mediaLibrary` | widgets |

Any new upload route that takes user-provided file bytes **must** opt into
the scan by passing a `scan` parameter. Server-internal uploads (e.g. seed
data, automated reports) may skip it because the bytes never came from a
user — but this is the only legitimate reason to omit `scan`.

## Limits & known gaps

- **Polyglot files** (a valid JPEG with an embedded JS payload exploiting an
  ImageMagick CVE) pass Layer 1 because the magic-bytes signature is still
  a valid JPEG. Layer 2 (`#17`) closes this with content scanning.
- **SVG XSS**: SVG is permitted in the `mediaLibrary` context. Files
  containing `<script>` tags can execute when rendered in a permissive
  context. CSP headers + DOMPurify on the render path are the right
  mitigation; sanitisation is out of scope for this task — track separately.
- **Text/plain and CSV** cannot be detected from magic bytes (no signature).
  The scanner accepts them when the declared MIME matches `text/plain` or
  `text/csv` AND the allow-list permits — this is a known weak point.
- **Rate limiting** on upload endpoints is the responsibility of the route
  itself. The scanner does not gate request rates.

---

# Layer 2 — ClamAV Post-Upload Scan

## Components

- **ClamAV container**: `clamav/clamav:stable` in `docker-compose.yml` with a
  named volume for the signature DB. First boot pulls ~250 MB of signatures
  via freshclam — the health check's `start_period` is set to 1200 s to
  cover that. Subsequent boots pass in under 2 s.
- **`createClamAVStorageScanner(...)`** in `@toolctrl/storage` —
  StorageScanner implementation. Talks to clamd over TCP via the `clamscan`
  npm package. Pings before every scan; throws `ClamAVUnreachableError`
  when clamd is unreachable.
- **`createS3StorageQuarantineAdapter(...)`** — StorageQuarantineAdapter.
  Server-side `CopyObjectCommand` into `quarantine/<key>` (or a dedicated
  `STORAGE_QUARANTINE_BUCKET`), then `DeleteObjectCommand` on the source.
  Writes `scan-signature`, `scan-quarantined-at`, `scan-source-bucket` and
  `scan-source-key` as S3 metadata so an admin restore action has a full
  back-pointer.
- **`createWorkerScanService(env)`** in `apps/worker/src/scanService.ts` —
  wires the scanner + quarantine adapter together via
  `createStorageScanService` from `@toolctrl/storage`. Returns `undefined`
  when STORAGE credentials are missing; the worker handler then short-
  circuits the scan-file job rather than running blind.
- **`enqueueFileScan(...)`** in `apps/web/app/utils/integrations/
  fileScanQueue.server.ts` — producer-side helper that opens a
  `TypedQueue<typeof queueNames.fileScan>`, enqueues a `scan-file` job
  built from `buildFileScanJob`, and closes the queue.

## Enabling Layer 2

Layer 2 is **opt-in** via two env knobs:

- `SCAN_QUEUE_ENABLED=true` on both the web and worker processes. On the
  web side the producer helper short-circuits without it; on the worker
  side `getWorkerQueueNames` appends `toolctrl.file-scan` only when set.
- `CLAMAV_HOST` (default `clamav`), `CLAMAV_PORT` (default `3310`),
  `CLAMAV_TIMEOUT_MS` (default `30000`). Defaults match the compose
  service. Override for staging/prod when ClamAV lives on a separate host.
- `STORAGE_QUARANTINE_BUCKET` (optional) — dedicated bucket for infected
  objects. Defaults to the source bucket with the `quarantine/` prefix.

Existing storage env (`STORAGE_REGION`, `STORAGE_ACCESS_KEY_ID`,
`STORAGE_SECRET_ACCESS_KEY`, etc.) is reused for both the scanner's GET
path and the quarantine adapter.

## How callers wire it in

`storeFile`'s `scan` option already learnt about Layer 1; for Layer 2 set
`enqueuePostScan: true`:

```ts
await storeFile({
  bucket: "users-icons",
  content: avatarDataUrl,
  id: userId,
  scan: {
    context: "avatar",
    filename: `${userId}.avatar`,
    onReject: auditRejectedUpload(request, "tenant.profile.avatar"),
    enqueuePostScan: true,
    tenantId,
  },
});
```

After a successful PUT, `storeFile` dynamically imports
`fileScanQueue.server` and enqueues. Failures to enqueue are swallowed (a
queue write hiccup must not block the synchronous user-visible upload);
they should be picked up by Sentry once #29 lands.

## When ClamAV is down

The worker's scan-file handler routes through `createStorageScanService`
which wraps the scanner's `ClamAVUnreachableError` into a `StorageScanResult`
with `verdict: "error"`. The job is retried per BullMQ's default policy; if
the outage persists, the job ends up in the failed-jobs DB row written by
the existing `recordQueueFailedJob` path. The synchronous upload remains
unaffected — fail-open by design.

When clamd is reachable but the signature DB is stale (freshclam hasn't
run for >24 h), the operator should restart the container or trigger
freshclam manually. The next task (`#29` Sentry self-host) gives us
alerting on signature staleness.

## EICAR smoke test

The standard EICAR test file (`X5O!P%@AP[4\PZX54(P^)7CC)7}$EICAR-...`) is
recognised by every AV engine including ClamAV. Use it for staging
verification:

1. Upload an `eicar.txt` (renamed to `.png` and re-typed to `image/png`
   so it passes layer 1) into the media library.
2. Within ~5 s the worker should run the scan-file job, get an "infected"
   verdict with signature `Win.Test.EICAR_HDB-1`, and quarantine the
   object.
3. Verify: `s3://<source-bucket>/<key>` no longer exists; the corresponding
   `quarantine/<key>` does, with the `scan-signature` metadata header set.

## Known limits

- **DB tracking is deferred.** The `media.scanStatus` / `media.scanResult`
  columns from the original detail-MD are not added yet. Today the only
  durable signal that a file was quarantined is the absence of the
  source-key + the presence of the quarantine copy + the S3 metadata. A
  follow-up commit will add the columns + the UI badge.
- **User notification on quarantine is deferred.** No email is sent today.
  Operator can watch the worker logs / S3 access logs.
- **Concurrency**: BullMQ defaults to a per-worker concurrency of 1.
  Bursty uploads will queue. Raise via the standard worker config knobs
  once production traffic justifies it.