Refresh HiveGrid Cloud reference snapshot

This commit is contained in:
david committed 2026-08-03 15:31:14 +02:00
1 parent 5e99e14772
commit 7f88026f79
5955 files changed
+1177325 -188362

No files matched your search

@@ -6,9 +6,10 @@ Defense-in-depth model for user-uploaded files. Two layers:
request handler **before** the bytes are committed to the storage backend
(S3 / local). Validates the actual content type via signature detection and
rejects mismatches or disallowed types.
- **Layer 2 — ClamAV Post-Upload Scan**. Asynchronous BullMQ job that runs
after the object is in the bucket. Hits trigger server-side quarantine
(CopyObject + DeleteObject into a `quarantine/` prefix).
- **Layer 2 — ClamAV malware scan**. Directly served assets such as avatars
and branding are scanned synchronously before the storage PUT. File-module
uploads remain private and non-deliverable while an asynchronous BullMQ job
scans them; hits trigger server-side quarantine.
## How it works
@@ -70,6 +71,7 @@ await storeFile({
scan: {
context: "avatar",
filename: `${userId}.avatar`,
requireMalwareScan: true,
onReject: auditRejectedUpload(request, "tenant.profile.avatar"),
},
});
@@ -133,11 +135,11 @@ user — but this is the only legitimate reason to omit `scan`.
---
# Layer 2 — ClamAV Post-Upload Scan
# Layer 2 — ClamAV Malware Scan
## Components
- **ClamAV container**: `clamav/clamav:stable` in `docker-compose.yml` with a
- **ClamAV container**: `clamav/clamav:stable@sha256:…` in `docker-compose.yml` with a
named volume for the signature DB. First boot pulls ~250 MB of signatures
via freshclam — the health check's `start_period` is set to 1200 s to
cover that. Subsequent boots pass in under 2 s.
@@ -168,9 +170,10 @@ Layer 2 is **opt-in** via two env knobs:
- `SCAN_QUEUE_ENABLED=true` on both the web and worker processes. On the
web side the producer helper short-circuits without it; on the worker
side `getWorkerQueueNames` appends `hivegrid.file-scan` only when set.
- `CLAMAV_HOST` (default `clamav`), `CLAMAV_PORT` (default `3310`),
`CLAMAV_TIMEOUT_MS` (default `30000`). Defaults match the compose
service. Override for staging/prod when ClamAV lives on a separate host.
- `CLAMD_HOST`, `CLAMD_PORT` (default `3310`) and `CLAMD_TIMEOUT_MS`
(default `30000`) configure synchronous pre-PUT scans. The legacy
`CLAMAV_*` names remain accepted for the worker/compose path. Production
uses the mesh-only central ClamAV endpoint from `infra/STANDARDS.md`.
- `STORAGE_QUARANTINE_BUCKET` (optional) — dedicated bucket for infected
objects. Defaults to the source bucket with the `quarantine/` prefix.
@@ -180,8 +183,8 @@ path and the quarantine adapter.
## How callers wire it in
`storeFile`'s `scan` option already learnt about Layer 1; for Layer 2 set
`enqueuePostScan: true`:
`storeFile`'s `scan` option already handles Layer 1. Directly served uploads
must additionally set `requireMalwareScan: true`:
```ts
await storeFile({
@@ -192,25 +195,25 @@ await storeFile({
context: "avatar",
filename: `${userId}.avatar`,
onReject: auditRejectedUpload(request, "tenant.profile.avatar"),
enqueuePostScan: true,
tenantId,
requireMalwareScan: true,
},
});
```
After a successful PUT, `storeFile` dynamically imports
`fileScanQueue.server` and enqueues. Failures to enqueue are swallowed (a
queue write hiccup must not block the synchronous user-visible upload);
they should be picked up by Sentry once #29 lands.
`storeFile` sends the in-memory buffer to clamd before the storage PUT. An
infected, unavailable, malformed or indeterminate scanner response rejects
the upload. File-module objects use the asynchronous queue instead and stay
at `scanStatus=pending`; media delivery and bulk ZIPs accept only `clean`.
## When ClamAV is down
The worker's scan-file handler routes through `createStorageScanService`
Synchronous direct-asset uploads fail closed before storage. The worker's
scan-file handler routes through `createStorageScanService`
which wraps the scanner's `ClamAVUnreachableError` into a `StorageScanResult`
with `verdict: "error"`. The job is retried per BullMQ's default policy; if
the outage persists, the job ends up in the failed-jobs DB row written by
the existing `recordQueueFailedJob` path. The synchronous upload remains
unaffected — fail-open by design.
the existing `recordQueueFailedJob` path. The object remains pending and
cannot be downloaded while the scanner is unavailable.
When clamd is reachable but the signature DB is stale (freshclam hasn't
run for >24 h), the operator should restart the container or trigger