98 lines
4.7 KiB
Markdown
98 lines
4.7 KiB
Markdown
# Streaming Exports — Operator Runbook
|
||
|
||
This document describes the streaming export pipeline shipped in Phase 03
|
||
task #46. It is the operator-facing complement to the detail plan
|
||
(`docs/improvement-plan/03-scaling-infrastructure/streaming-exports.md`).
|
||
|
||
## Why streaming
|
||
|
||
The previous export pipeline buffered every row of the target dataset in
|
||
worker memory, serialised it into a single string or `Uint8Array`, wrote
|
||
the result to a temporary file on disk, and only then uploaded the file
|
||
to S3. For large exports (>50k rows) this caused three operational
|
||
problems:
|
||
|
||
- worker OOM crashes on 200k+ row XLSX exports;
|
||
- user-visible latency in the multi-minute range because nothing
|
||
reached S3 until the very end;
|
||
- doubled I/O on the worker host (local write → S3 PUT).
|
||
|
||
The streaming pipeline replaces both the buffering and the tempfile with
|
||
per-format streamers and an `@aws-sdk/lib-storage` multipart upload.
|
||
|
||
## Per-format strategy
|
||
|
||
| Format | Streaming strategy | Hard cap | Notes |
|
||
|---|---|---|---|
|
||
| CSV | Async iterator → Node writable | – | `serializeExportTableToCsvStream` |
|
||
| JSON (NDJSON) | Async iterator → one JSON object per line | – | Default JSON mode |
|
||
| JSON (Pretty) | In-memory buffer of normalised rows | 10_000 | `ExportPrettyJsonTooLargeError` above the cap |
|
||
| XLSX | ExcelJS `WorkbookWriter` when `estimatedRowCount > 50_000`, in-memory workbook otherwise | 500_000 | `serializeExportTableToXlsxStream` |
|
||
| PDF | `@react-pdf/renderer` → `renderToStream` | 5_000 | `serializeExportTableToPdfStream` |
|
||
|
||
The format hard caps protect against format-specific failure modes (XLSX
|
||
stream-writer is fragile above 500k rows; PDF cost grows super-linearly).
|
||
They apply even to Enterprise tenants.
|
||
|
||
## Plan-based row caps
|
||
|
||
`packages/exports/src/limits.ts` exposes `resolveExportRowCap(plan, format)`
|
||
and `assertExportRowCap(rowCount, plan, format)`. The plan caps are:
|
||
|
||
| Plan | Cap (CSV/JSON) | XLSX cap | PDF cap |
|
||
|---|---|---|---|
|
||
| free | 10_000 | 10_000 | 5_000 |
|
||
| pro | 100_000 | 100_000 | 5_000 |
|
||
| enterprise | unbounded | 500_000 | 5_000 |
|
||
|
||
Throws `ExportRowCapExceededError` (code `EXPORT_ROW_LIMIT_PLAN`) when
|
||
the requested row count exceeds the resolved cap. Both the up-front
|
||
estimate and the streaming mid-pass enforce the cap so callers cannot
|
||
overshoot by under-reporting the estimate.
|
||
|
||
## S3 multipart upload sink
|
||
|
||
`createS3MultipartUploadSink({ client, bucket, key, contentType })`
|
||
returns `{ writeStream, completionPromise }`. The factory:
|
||
|
||
- uses `@aws-sdk/lib-storage`'s `Upload` class with `partSize: 8 MB`
|
||
and `queueSize: 4` (overridable);
|
||
- destroys the write stream when the upload rejects so producers see
|
||
the failure via the normal stream `error` event;
|
||
- relies on the AWS SDK to call `AbortMultipartUpload` on failure —
|
||
there is no orphaned multipart cleanup task to schedule.
|
||
|
||
## Progress tracking
|
||
|
||
The worker job (`apps/worker/src/jobs/export.ts`) writes
|
||
`export:progress:<exportJobId>` into the shared store every
|
||
`progressFlushInterval` rows (default 5_000) and again on success. The
|
||
payload is `{ processedRows, totalEstimate, updatedAt }` and the TTL is
|
||
1 hour.
|
||
|
||
The API exposes `GET /v1/exports/:exportJobId/progress` which simply
|
||
proxies the snapshot back. Front-ends poll every 2-5 s until the
|
||
artefact download URL becomes available.
|
||
|
||
## Backward compatibility
|
||
|
||
The in-memory façades (`serializeExportTableToCsv`,
|
||
`serializeExportTableToJson`, `serializeExportTableToXlsx`,
|
||
`serializeExportTableToPdf`) are intentionally preserved and continue to
|
||
work without code changes. They are still suitable for small synchronous
|
||
exports (e.g. the existing `Rows_Export.server.ts` page-bounded CSV
|
||
download in the web app).
|
||
|
||
New code that touches the worker pipeline should prefer the streaming
|
||
variants and the row-cap helper.
|
||
|
||
## Troubleshooting
|
||
|
||
| Symptom | Likely cause | Action |
|
||
|---|---|---|
|
||
| Job stuck at 0 processed rows | Worker cannot reach the shared store | Check `REDIS_URL` and the shared-store warning emitted at boot; the worker continues but progress is invisible |
|
||
| `EXPORT_ROW_LIMIT_PLAN` returned | Tenant requested more rows than their plan allows | Upgrade plan or split the export |
|
||
| `EXPORT_ROW_LIMIT_EXCEEDED` returned | Tenant requested more than the format hard cap | Switch to CSV/NDJSON for the bulk export |
|
||
| Multipart upload visible in S3 console as `incomplete` | Job failed mid-stream; SDK abort raced with check | AWS abandons the upload after `LifecycleRule` for `AbortIncompleteMultipartUpload` (recommended: 7 days). Make sure the bucket has one configured |
|
||
| 200k-row CSV export takes minutes | Row source `iterateRows` is not actually cursored | Verify the row source uses `prisma.findMany({ cursor, take: 1000 })` rather than a single unbounded query |
|