Track HiveGrid Cloud planning reference
This commit is contained in:
david committed
1 parent
b3d5ea6764
commit
e341c5938a
5784 files changed
+1104929
-3
No files matched your search
@@ -0,0 +1,245 @@
|
||||
# Storage AV Scanning
|
||||
|
||||
Defense-in-depth model for user-uploaded files. Two layers:
|
||||
|
||||
- **Layer 1 — Magic-Bytes Pre-Upload Scan**. Synchronous, runs in the upload
|
||||
request handler **before** the bytes are committed to the storage backend
|
||||
(S3 / local). Validates the actual content type via signature detection and
|
||||
rejects mismatches or disallowed types.
|
||||
- **Layer 2 — ClamAV Post-Upload Scan**. Asynchronous BullMQ job that runs
|
||||
after the object is in the bucket. Hits trigger server-side quarantine
|
||||
(CopyObject + DeleteObject into a `quarantine/` prefix).
|
||||
|
||||
## How it works
|
||||
|
||||
The `MagicBytesScanAdapter` in `@toolctrl/storage` reads the first ~4100 bytes
|
||||
of an upload buffer and resolves its MIME type via the `file-type` library's
|
||||
signature database. It then validates the detected type against:
|
||||
|
||||
1. The client-declared `Content-Type` (rejects mismatches — common attack
|
||||
vector: an `.exe` smuggled as `image/png`).
|
||||
2. A per-context allow-list (rejects e.g. a PDF uploaded into the avatar
|
||||
context — see allow-list table below).
|
||||
3. The filename — control characters (`\x00-\x1f`, `\x7f`) and path traversal
|
||||
sequences (`../`, `..\\`) are rejected; filenames over 255 chars are
|
||||
rejected.
|
||||
|
||||
## Allow-list defaults
|
||||
|
||||
See `packages/storage/src/scan/allow-list.ts` for the canonical list.
|
||||
|
||||
| Context | Allowed MIME types |
|
||||
|---|---|
|
||||
| `avatar` | image/png, image/jpeg, image/webp |
|
||||
| `mediaLibrary` | + image/gif, image/svg+xml, image/x-icon, image/vnd.microsoft.icon |
|
||||
| `emailAttachment` | image/{png,jpeg,webp,gif} + application/pdf, text/plain, text/csv, application/zip, Office .docx/.xlsx/.pptx |
|
||||
| `formUpload` | image/{png,jpeg,webp} + application/pdf |
|
||||
| `kbAsset` | image/{png,jpeg,webp} + application/pdf |
|
||||
|
||||
Per-context override via env: `STORAGE_SCAN_ALLOWLIST_<CONTEXT>` accepts a
|
||||
comma-separated list of MIME types and **replaces** the default for that
|
||||
context (not merged). Contexts in env-var form: `AVATAR`, `MEDIA_LIBRARY`,
|
||||
`EMAIL_ATTACHMENT`, `FORM_UPLOAD`, `KB_ASSET`.
|
||||
|
||||
Example: allow only PNG and JPEG for tenant avatars in production:
|
||||
|
||||
```
|
||||
STORAGE_SCAN_ALLOWLIST_AVATAR=image/png,image/jpeg
|
||||
```
|
||||
|
||||
## Notfall-Bypass
|
||||
|
||||
Set `STORAGE_SCAN_LAYER1_ENABLED=false` to disable scanning entirely (the
|
||||
adapter returns `status: "skipped"` for every call). Intended for emergency
|
||||
incident response only — do not run with scanning disabled in production
|
||||
for any extended period.
|
||||
|
||||
## How callers wire it in
|
||||
|
||||
`storeFile` from `apps/web/app/utils/integrations/storageService.ts` accepts
|
||||
an optional `scan` parameter:
|
||||
|
||||
```ts
|
||||
import { storeFile } from "~/utils/integrations/storageService";
|
||||
import { auditRejectedUpload } from "~/utils/integrations/storageAudit.server";
|
||||
|
||||
await storeFile({
|
||||
bucket: "users-icons",
|
||||
content: avatarDataUrl,
|
||||
id: userId,
|
||||
scan: {
|
||||
context: "avatar",
|
||||
filename: `${userId}.avatar`,
|
||||
onReject: auditRejectedUpload(request, "tenant.profile.avatar"),
|
||||
},
|
||||
});
|
||||
```
|
||||
|
||||
A rejected scan throws `StorageScanRejectedError` from `storageService.ts`.
|
||||
The error carries the bucket, id, filename and full `PreUploadScanResult`
|
||||
(status, reason, detail, detectedMimeType). Callers can catch it and map
|
||||
to a 415 response with a user-friendly message.
|
||||
|
||||
The `onReject` callback fires before the error throws and is awaited, so the
|
||||
audit record is persisted even when the route swallows the error.
|
||||
|
||||
## Audit events
|
||||
|
||||
`auditRejectedUpload(request, context)` writes a `createAdminLog`-style row
|
||||
with action `upload.rejected.scan`. The serialised detail field contains:
|
||||
|
||||
```json
|
||||
{
|
||||
"context": "tenant.profile.avatar",
|
||||
"reason": "mime_mismatch",
|
||||
"detail": "declared=\"image/png\", detected=\"application/x-msdownload\"",
|
||||
"detectedMimeType": "application/x-msdownload"
|
||||
}
|
||||
```
|
||||
|
||||
The buffer content is never written to the audit log. Only metadata
|
||||
(bounded reason enum + adapter-level diagnostic string + detected MIME).
|
||||
|
||||
## Currently-scanned upload paths
|
||||
|
||||
| Route | Context | Buckets |
|
||||
|---|---|---|
|
||||
| `apps/web/app/routes/admin/settings/general.tsx` | `mediaLibrary` | branding (logo, dark logo, icon, dark icon, favicon) |
|
||||
| `apps/web/app/routes/admin/settings/profile.tsx` | `avatar` | users-icons |
|
||||
| `apps/web/app/routes/app.$tenant/settings/profile.tsx` | `avatar` | users-icons |
|
||||
| `apps/web/app/routes/app.$tenant/settings/account.tsx` | `avatar` | accounts-icons |
|
||||
| `apps/web/app/routes/app.$tenant/__portals/portals.$portal/settings.tsx` | `mediaLibrary` | branding (5), seo (2) |
|
||||
| `apps/web/app/routes/app.$tenant/__widgets/widgets/$id.tsx` | `mediaLibrary` | widgets |
|
||||
|
||||
Any new upload route that takes user-provided file bytes **must** opt into
|
||||
the scan by passing a `scan` parameter. Server-internal uploads (e.g. seed
|
||||
data, automated reports) may skip it because the bytes never came from a
|
||||
user — but this is the only legitimate reason to omit `scan`.
|
||||
|
||||
## Limits & known gaps
|
||||
|
||||
- **Polyglot files** (a valid JPEG with an embedded JS payload exploiting an
|
||||
ImageMagick CVE) pass Layer 1 because the magic-bytes signature is still
|
||||
a valid JPEG. Layer 2 (`#17`) closes this with content scanning.
|
||||
- **SVG XSS**: SVG is permitted in the `mediaLibrary` context. Files
|
||||
containing `<script>` tags can execute when rendered in a permissive
|
||||
context. CSP headers + DOMPurify on the render path are the right
|
||||
mitigation; sanitisation is out of scope for this task — track separately.
|
||||
- **Text/plain and CSV** cannot be detected from magic bytes (no signature).
|
||||
The scanner accepts them when the declared MIME matches `text/plain` or
|
||||
`text/csv` AND the allow-list permits — this is a known weak point.
|
||||
- **Rate limiting** on upload endpoints is the responsibility of the route
|
||||
itself. The scanner does not gate request rates.
|
||||
|
||||
---
|
||||
|
||||
# Layer 2 — ClamAV Post-Upload Scan
|
||||
|
||||
## Components
|
||||
|
||||
- **ClamAV container**: `clamav/clamav:stable` in `docker-compose.yml` with a
|
||||
named volume for the signature DB. First boot pulls ~250 MB of signatures
|
||||
via freshclam — the health check's `start_period` is set to 1200 s to
|
||||
cover that. Subsequent boots pass in under 2 s.
|
||||
- **`createClamAVStorageScanner(...)`** in `@toolctrl/storage` —
|
||||
StorageScanner implementation. Talks to clamd over TCP via the `clamscan`
|
||||
npm package. Pings before every scan; throws `ClamAVUnreachableError`
|
||||
when clamd is unreachable.
|
||||
- **`createS3StorageQuarantineAdapter(...)`** — StorageQuarantineAdapter.
|
||||
Server-side `CopyObjectCommand` into `quarantine/<key>` (or a dedicated
|
||||
`STORAGE_QUARANTINE_BUCKET`), then `DeleteObjectCommand` on the source.
|
||||
Writes `scan-signature`, `scan-quarantined-at`, `scan-source-bucket` and
|
||||
`scan-source-key` as S3 metadata so an admin restore action has a full
|
||||
back-pointer.
|
||||
- **`createWorkerScanService(env)`** in `apps/worker/src/scanService.ts` —
|
||||
wires the scanner + quarantine adapter together via
|
||||
`createStorageScanService` from `@toolctrl/storage`. Returns `undefined`
|
||||
when STORAGE credentials are missing; the worker handler then short-
|
||||
circuits the scan-file job rather than running blind.
|
||||
- **`enqueueFileScan(...)`** in `apps/web/app/utils/integrations/
|
||||
fileScanQueue.server.ts` — producer-side helper that opens a
|
||||
`TypedQueue<typeof queueNames.fileScan>`, enqueues a `scan-file` job
|
||||
built from `buildFileScanJob`, and closes the queue.
|
||||
|
||||
## Enabling Layer 2
|
||||
|
||||
Layer 2 is **opt-in** via two env knobs:
|
||||
|
||||
- `SCAN_QUEUE_ENABLED=true` on both the web and worker processes. On the
|
||||
web side the producer helper short-circuits without it; on the worker
|
||||
side `getWorkerQueueNames` appends `toolctrl.file-scan` only when set.
|
||||
- `CLAMAV_HOST` (default `clamav`), `CLAMAV_PORT` (default `3310`),
|
||||
`CLAMAV_TIMEOUT_MS` (default `30000`). Defaults match the compose
|
||||
service. Override for staging/prod when ClamAV lives on a separate host.
|
||||
- `STORAGE_QUARANTINE_BUCKET` (optional) — dedicated bucket for infected
|
||||
objects. Defaults to the source bucket with the `quarantine/` prefix.
|
||||
|
||||
Existing storage env (`STORAGE_REGION`, `STORAGE_ACCESS_KEY_ID`,
|
||||
`STORAGE_SECRET_ACCESS_KEY`, etc.) is reused for both the scanner's GET
|
||||
path and the quarantine adapter.
|
||||
|
||||
## How callers wire it in
|
||||
|
||||
`storeFile`'s `scan` option already learnt about Layer 1; for Layer 2 set
|
||||
`enqueuePostScan: true`:
|
||||
|
||||
```ts
|
||||
await storeFile({
|
||||
bucket: "users-icons",
|
||||
content: avatarDataUrl,
|
||||
id: userId,
|
||||
scan: {
|
||||
context: "avatar",
|
||||
filename: `${userId}.avatar`,
|
||||
onReject: auditRejectedUpload(request, "tenant.profile.avatar"),
|
||||
enqueuePostScan: true,
|
||||
tenantId,
|
||||
},
|
||||
});
|
||||
```
|
||||
|
||||
After a successful PUT, `storeFile` dynamically imports
|
||||
`fileScanQueue.server` and enqueues. Failures to enqueue are swallowed (a
|
||||
queue write hiccup must not block the synchronous user-visible upload);
|
||||
they should be picked up by Sentry once #29 lands.
|
||||
|
||||
## When ClamAV is down
|
||||
|
||||
The worker's scan-file handler routes through `createStorageScanService`
|
||||
which wraps the scanner's `ClamAVUnreachableError` into a `StorageScanResult`
|
||||
with `verdict: "error"`. The job is retried per BullMQ's default policy; if
|
||||
the outage persists, the job ends up in the failed-jobs DB row written by
|
||||
the existing `recordQueueFailedJob` path. The synchronous upload remains
|
||||
unaffected — fail-open by design.
|
||||
|
||||
When clamd is reachable but the signature DB is stale (freshclam hasn't
|
||||
run for >24 h), the operator should restart the container or trigger
|
||||
freshclam manually. The next task (`#29` Sentry self-host) gives us
|
||||
alerting on signature staleness.
|
||||
|
||||
## EICAR smoke test
|
||||
|
||||
The standard EICAR test file (`X5O!P%@AP[4\PZX54(P^)7CC)7}$EICAR-...`) is
|
||||
recognised by every AV engine including ClamAV. Use it for staging
|
||||
verification:
|
||||
|
||||
1. Upload an `eicar.txt` (renamed to `.png` and re-typed to `image/png`
|
||||
so it passes layer 1) into the media library.
|
||||
2. Within ~5 s the worker should run the scan-file job, get an "infected"
|
||||
verdict with signature `Win.Test.EICAR_HDB-1`, and quarantine the
|
||||
object.
|
||||
3. Verify: `s3://<source-bucket>/<key>` no longer exists; the corresponding
|
||||
`quarantine/<key>` does, with the `scan-signature` metadata header set.
|
||||
|
||||
## Known limits
|
||||
|
||||
- **DB tracking is deferred.** The `media.scanStatus` / `media.scanResult`
|
||||
columns from the original detail-MD are not added yet. Today the only
|
||||
durable signal that a file was quarantined is the absence of the
|
||||
source-key + the presence of the quarantine copy + the S3 metadata. A
|
||||
follow-up commit will add the columns + the UI badge.
|
||||
- **User notification on quarantine is deferred.** No email is sent today.
|
||||
Operator can watch the worker logs / S3 access logs.
|
||||
- **Concurrency**: BullMQ defaults to a per-worker concurrency of 1.
|
||||
Bursty uploads will queue. Raise via the standard worker config knobs
|
||||
once production traffic justifies it.
|
||||
Reference in new issue
Block a user