4.7 KiB
Streaming Exports — Operator Runbook
This document describes the streaming export pipeline shipped in Phase 03
task #46. It is the operator-facing complement to the detail plan
(docs/improvement-plan/03-scaling-infrastructure/streaming-exports.md).
Why streaming
The previous export pipeline buffered every row of the target dataset in
worker memory, serialised it into a single string or Uint8Array, wrote
the result to a temporary file on disk, and only then uploaded the file
to S3. For large exports (>50k rows) this caused three operational
problems:
- worker OOM crashes on 200k+ row XLSX exports;
- user-visible latency in the multi-minute range because nothing reached S3 until the very end;
- doubled I/O on the worker host (local write → S3 PUT).
The streaming pipeline replaces both the buffering and the tempfile with
per-format streamers and an @aws-sdk/lib-storage multipart upload.
Per-format strategy
| Format | Streaming strategy | Hard cap | Notes |
|---|---|---|---|
| CSV | Async iterator → Node writable | – | serializeExportTableToCsvStream |
| JSON (NDJSON) | Async iterator → one JSON object per line | – | Default JSON mode |
| JSON (Pretty) | In-memory buffer of normalised rows | 10_000 | ExportPrettyJsonTooLargeError above the cap |
| XLSX | ExcelJS WorkbookWriter when estimatedRowCount > 50_000, in-memory workbook otherwise |
500_000 | serializeExportTableToXlsxStream |
@react-pdf/renderer → renderToStream |
5_000 | serializeExportTableToPdfStream |
The format hard caps protect against format-specific failure modes (XLSX stream-writer is fragile above 500k rows; PDF cost grows super-linearly). They apply even to Enterprise tenants.
Plan-based row caps
packages/exports/src/limits.ts exposes resolveExportRowCap(plan, format)
and assertExportRowCap(rowCount, plan, format). The plan caps are:
| Plan | Cap (CSV/JSON) | XLSX cap | PDF cap |
|---|---|---|---|
| free | 10_000 | 10_000 | 5_000 |
| pro | 100_000 | 100_000 | 5_000 |
| enterprise | unbounded | 500_000 | 5_000 |
Throws ExportRowCapExceededError (code EXPORT_ROW_LIMIT_PLAN) when
the requested row count exceeds the resolved cap. Both the up-front
estimate and the streaming mid-pass enforce the cap so callers cannot
overshoot by under-reporting the estimate.
S3 multipart upload sink
createS3MultipartUploadSink({ client, bucket, key, contentType })
returns { writeStream, completionPromise }. The factory:
- uses
@aws-sdk/lib-storage'sUploadclass withpartSize: 8 MBandqueueSize: 4(overridable); - destroys the write stream when the upload rejects so producers see
the failure via the normal stream
errorevent; - relies on the AWS SDK to call
AbortMultipartUploadon failure — there is no orphaned multipart cleanup task to schedule.
Progress tracking
The worker job (apps/worker/src/jobs/export.ts) writes
export:progress:<exportJobId> into the shared store every
progressFlushInterval rows (default 5_000) and again on success. The
payload is { processedRows, totalEstimate, updatedAt } and the TTL is
1 hour.
The API exposes GET /v1/exports/:exportJobId/progress which simply
proxies the snapshot back. Front-ends poll every 2-5 s until the
artefact download URL becomes available.
Backward compatibility
The in-memory façades (serializeExportTableToCsv,
serializeExportTableToJson, serializeExportTableToXlsx,
serializeExportTableToPdf) are intentionally preserved and continue to
work without code changes. They are still suitable for small synchronous
exports (e.g. the existing Rows_Export.server.ts page-bounded CSV
download in the web app).
New code that touches the worker pipeline should prefer the streaming variants and the row-cap helper.
Troubleshooting
| Symptom | Likely cause | Action |
|---|---|---|
| Job stuck at 0 processed rows | Worker cannot reach the shared store | Check REDIS_URL and the shared-store warning emitted at boot; the worker continues but progress is invisible |
EXPORT_ROW_LIMIT_PLAN returned |
Tenant requested more rows than their plan allows | Upgrade plan or split the export |
EXPORT_ROW_LIMIT_EXCEEDED returned |
Tenant requested more than the format hard cap | Switch to CSV/NDJSON for the bulk export |
Multipart upload visible in S3 console as incomplete |
Job failed mid-stream; SDK abort raced with check | AWS abandons the upload after LifecycleRule for AbortIncompleteMultipartUpload (recommended: 7 days). Make sure the bucket has one configured |
| 200k-row CSV export takes minutes | Row source iterateRows is not actually cursored |
Verify the row source uses prisma.findMany({ cursor, take: 1000 }) rather than a single unbounded query |