Track HiveGrid Cloud planning reference
This commit is contained in:
1 parent
b3d5ea6764
commit
e341c5938a
5784 files changed
+1104929
-3
No files matched your search
@@ -0,0 +1,97 @@
|
||||
# Streaming Exports — Operator Runbook
|
||||
|
||||
This document describes the streaming export pipeline shipped in Phase 03
|
||||
task #46. It is the operator-facing complement to the detail plan
|
||||
(`docs/improvement-plan/03-scaling-infrastructure/streaming-exports.md`).
|
||||
|
||||
## Why streaming
|
||||
|
||||
The previous export pipeline buffered every row of the target dataset in
|
||||
worker memory, serialised it into a single string or `Uint8Array`, wrote
|
||||
the result to a temporary file on disk, and only then uploaded the file
|
||||
to S3. For large exports (>50k rows) this caused three operational
|
||||
problems:
|
||||
|
||||
- worker OOM crashes on 200k+ row XLSX exports;
|
||||
- user-visible latency in the multi-minute range because nothing
|
||||
reached S3 until the very end;
|
||||
- doubled I/O on the worker host (local write → S3 PUT).
|
||||
|
||||
The streaming pipeline replaces both the buffering and the tempfile with
|
||||
per-format streamers and an `@aws-sdk/lib-storage` multipart upload.
|
||||
|
||||
## Per-format strategy
|
||||
|
||||
| Format | Streaming strategy | Hard cap | Notes |
|
||||
|---|---|---|---|
|
||||
| CSV | Async iterator → Node writable | – | `serializeExportTableToCsvStream` |
|
||||
| JSON (NDJSON) | Async iterator → one JSON object per line | – | Default JSON mode |
|
||||
| JSON (Pretty) | In-memory buffer of normalised rows | 10_000 | `ExportPrettyJsonTooLargeError` above the cap |
|
||||
| XLSX | ExcelJS `WorkbookWriter` when `estimatedRowCount > 50_000`, in-memory workbook otherwise | 500_000 | `serializeExportTableToXlsxStream` |
|
||||
| PDF | `@react-pdf/renderer` → `renderToStream` | 5_000 | `serializeExportTableToPdfStream` |
|
||||
|
||||
The format hard caps protect against format-specific failure modes (XLSX
|
||||
stream-writer is fragile above 500k rows; PDF cost grows super-linearly).
|
||||
They apply even to Enterprise tenants.
|
||||
|
||||
## Plan-based row caps
|
||||
|
||||
`packages/exports/src/limits.ts` exposes `resolveExportRowCap(plan, format)`
|
||||
and `assertExportRowCap(rowCount, plan, format)`. The plan caps are:
|
||||
|
||||
| Plan | Cap (CSV/JSON) | XLSX cap | PDF cap |
|
||||
|---|---|---|---|
|
||||
| free | 10_000 | 10_000 | 5_000 |
|
||||
| pro | 100_000 | 100_000 | 5_000 |
|
||||
| enterprise | unbounded | 500_000 | 5_000 |
|
||||
|
||||
Throws `ExportRowCapExceededError` (code `EXPORT_ROW_LIMIT_PLAN`) when
|
||||
the requested row count exceeds the resolved cap. Both the up-front
|
||||
estimate and the streaming mid-pass enforce the cap so callers cannot
|
||||
overshoot by under-reporting the estimate.
|
||||
|
||||
## S3 multipart upload sink
|
||||
|
||||
`createS3MultipartUploadSink({ client, bucket, key, contentType })`
|
||||
returns `{ writeStream, completionPromise }`. The factory:
|
||||
|
||||
- uses `@aws-sdk/lib-storage`'s `Upload` class with `partSize: 8 MB`
|
||||
and `queueSize: 4` (overridable);
|
||||
- destroys the write stream when the upload rejects so producers see
|
||||
the failure via the normal stream `error` event;
|
||||
- relies on the AWS SDK to call `AbortMultipartUpload` on failure —
|
||||
there is no orphaned multipart cleanup task to schedule.
|
||||
|
||||
## Progress tracking
|
||||
|
||||
The worker job (`apps/worker/src/jobs/export.ts`) writes
|
||||
`export:progress:<exportJobId>` into the shared store every
|
||||
`progressFlushInterval` rows (default 5_000) and again on success. The
|
||||
payload is `{ processedRows, totalEstimate, updatedAt }` and the TTL is
|
||||
1 hour.
|
||||
|
||||
The API exposes `GET /v1/exports/:exportJobId/progress` which simply
|
||||
proxies the snapshot back. Front-ends poll every 2-5 s until the
|
||||
artefact download URL becomes available.
|
||||
|
||||
## Backward compatibility
|
||||
|
||||
The in-memory façades (`serializeExportTableToCsv`,
|
||||
`serializeExportTableToJson`, `serializeExportTableToXlsx`,
|
||||
`serializeExportTableToPdf`) are intentionally preserved and continue to
|
||||
work without code changes. They are still suitable for small synchronous
|
||||
exports (e.g. the existing `Rows_Export.server.ts` page-bounded CSV
|
||||
download in the web app).
|
||||
|
||||
New code that touches the worker pipeline should prefer the streaming
|
||||
variants and the row-cap helper.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
| Symptom | Likely cause | Action |
|
||||
|---|---|---|
|
||||
| Job stuck at 0 processed rows | Worker cannot reach the shared store | Check `REDIS_URL` and the shared-store warning emitted at boot; the worker continues but progress is invisible |
|
||||
| `EXPORT_ROW_LIMIT_PLAN` returned | Tenant requested more rows than their plan allows | Upgrade plan or split the export |
|
||||
| `EXPORT_ROW_LIMIT_EXCEEDED` returned | Tenant requested more than the format hard cap | Switch to CSV/NDJSON for the bulk export |
|
||||
| Multipart upload visible in S3 console as `incomplete` | Job failed mid-stream; SDK abort raced with check | AWS abandons the upload after `LifecycleRule` for `AbortIncompleteMultipartUpload` (recommended: 7 days). Make sure the bucket has one configured |
|
||||
| 200k-row CSV export takes minutes | Row source `iterateRows` is not actually cursored | Verify the row source uses `prisma.findMany({ cursor, take: 1000 })` rather than a single unbounded query |
|
||||
Reference in new issue
Block a user