# Streaming Exports — Operator Runbook

This document describes the streaming export pipeline shipped in Phase 03
task #46. It is the operator-facing complement to the detail plan
(`docs/improvement-plan/03-scaling-infrastructure/streaming-exports.md`).

## Why streaming

The previous export pipeline buffered every row of the target dataset in
worker memory, serialised it into a single string or `Uint8Array`, wrote
the result to a temporary file on disk, and only then uploaded the file
to S3. For large exports (>50k rows) this caused three operational
problems:

- worker OOM crashes on 200k+ row XLSX exports;
- user-visible latency in the multi-minute range because nothing
  reached S3 until the very end;
- doubled I/O on the worker host (local write → S3 PUT).

The streaming pipeline replaces both the buffering and the tempfile with
per-format streamers and an `@aws-sdk/lib-storage` multipart upload.

## Per-format strategy

| Format | Streaming strategy | Hard cap | Notes |
|---|---|---|---|
| CSV | Async iterator → Node writable | – | `serializeExportTableToCsvStream` |
| JSON (NDJSON) | Async iterator → one JSON object per line | – | Default JSON mode |
| JSON (Pretty) | In-memory buffer of normalised rows | 10_000 | `ExportPrettyJsonTooLargeError` above the cap |
| XLSX | ExcelJS `WorkbookWriter` when `estimatedRowCount > 50_000`, in-memory workbook otherwise | 500_000 | `serializeExportTableToXlsxStream` |
| PDF | `@react-pdf/renderer` → `renderToStream` | 5_000 | `serializeExportTableToPdfStream` |

The format hard caps protect against format-specific failure modes (XLSX
stream-writer is fragile above 500k rows; PDF cost grows super-linearly).
They apply even to Enterprise tenants.

## Plan-based row caps

`packages/exports/src/limits.ts` exposes `resolveExportRowCap(plan, format)`
and `assertExportRowCap(rowCount, plan, format)`. The plan caps are:

| Plan | Cap (CSV/JSON) | XLSX cap | PDF cap |
|---|---|---|---|
| free | 10_000 | 10_000 | 5_000 |
| pro | 100_000 | 100_000 | 5_000 |
| enterprise | unbounded | 500_000 | 5_000 |

Throws `ExportRowCapExceededError` (code `EXPORT_ROW_LIMIT_PLAN`) when
the requested row count exceeds the resolved cap. Both the up-front
estimate and the streaming mid-pass enforce the cap so callers cannot
overshoot by under-reporting the estimate.

## S3 multipart upload sink

`createS3MultipartUploadSink({ client, bucket, key, contentType })`
returns `{ writeStream, completionPromise }`. The factory:

- uses `@aws-sdk/lib-storage`'s `Upload` class with `partSize: 8 MB`
  and `queueSize: 4` (overridable);
- destroys the write stream when the upload rejects so producers see
  the failure via the normal stream `error` event;
- relies on the AWS SDK to call `AbortMultipartUpload` on failure —
  there is no orphaned multipart cleanup task to schedule.

## Progress tracking

The worker job (`apps/worker/src/jobs/export.ts`) writes
`export:progress:<exportJobId>` into the shared store every
`progressFlushInterval` rows (default 5_000) and again on success. The
payload is `{ processedRows, totalEstimate, updatedAt }` and the TTL is
1 hour.

The API exposes `GET /v1/exports/:exportJobId/progress` which simply
proxies the snapshot back. Front-ends poll every 2-5 s until the
artefact download URL becomes available.

## Backward compatibility

The in-memory façades (`serializeExportTableToCsv`,
`serializeExportTableToJson`, `serializeExportTableToXlsx`,
`serializeExportTableToPdf`) are intentionally preserved and continue to
work without code changes. They are still suitable for small synchronous
exports (e.g. the existing `Rows_Export.server.ts` page-bounded CSV
download in the web app).

New code that touches the worker pipeline should prefer the streaming
variants and the row-cap helper.

## Troubleshooting

| Symptom | Likely cause | Action |
|---|---|---|
| Job stuck at 0 processed rows | Worker cannot reach the shared store | Check `REDIS_URL` and the shared-store warning emitted at boot; the worker continues but progress is invisible |
| `EXPORT_ROW_LIMIT_PLAN` returned | Tenant requested more rows than their plan allows | Upgrade plan or split the export |
| `EXPORT_ROW_LIMIT_EXCEEDED` returned | Tenant requested more than the format hard cap | Switch to CSV/NDJSON for the bulk export |
| Multipart upload visible in S3 console as `incomplete` | Job failed mid-stream; SDK abort raced with check | AWS abandons the upload after `LifecycleRule` for `AbortIncompleteMultipartUpload` (recommended: 7 days). Make sure the bucket has one configured |
| 200k-row CSV export takes minutes | Row source `iterateRows` is not actually cursored | Verify the row source uses `prisma.findMany({ cursor, take: 1000 })` rather than a single unbounded query |