# Plan-basierte Job-Priorities in BullMQ **Status:** completed **Task-ID:** #47 **Phase:** 03 **Estimated Effort:** 1-2 Tage **Owner:** unassigned ## Problem Alle Jobs werden aktuell mit identischer Priorität enqueued (BullMQ-Default: 0 = highest). Folgen: - **Enterprise-Tenants warten in der Queue hinter Free-Tier-Tenants.** Ein Free-Tenant der 10k Mail-Jobs schedulet blockt Enterprise-Tenants die zahlende Kunden haben. - **Interne Jobs (cron-retention, audit-cleanup) konkurrieren mit User-Jobs** — sie sollten höchste Priorität haben damit Wartung nicht verschleppt wird, tun es aber nicht. - **Keine Differenzierung zwischen Real-Time-Notification (User wartet im UI) und Batch-Mail (asynchron) möglich.** - **Kein Hebel für QoS-Commitments** in Verträgen („Enterprise SLA: alle Jobs < 30s"). Betroffene Codestellen: - `packages/queue/src/producers.ts` — `enqueueJob(queueName, payload, opts?)` ignoriert Priority - Alle Producer-Call-Sites: - `packages/notifications/src/producer.ts` - `packages/mail/src/queue.ts` - `packages/webhooks/src/producer.ts` - `packages/imports/src/producer.ts` - `packages/exports/src/producer.ts` - `packages/ai/src/jobs.ts` - `packages/workflows/src/executor.ts` - `apps/api/src/cron/*.ts` ## Depends on - `03-scaling-infrastructure/shared-store-package.md` — Tenant-Plan-Lookup cached im SharedStore (TTL 5min) damit `resolveJobPriorityForTenant` nicht jeden enqueue-Call zur DB schickt. - `02-observability/metrics-grafana.md` — Counter `queue_jobs_by_priority` braucht das `/metrics`-Endpoint. ## Blocks - Keine harten Blocker, aber: - `03-scaling-infrastructure/worker-queues.md` — Worker-Trennung amplifiziert den Wert (Priority innerhalb einer Queue + Trennung der Queues = bessere QoS). - `05-business-features/ai-cost-phase-2.md` — Quota-Enforcement kann Job-Priority-Boost als Pro-Plan-Feature vermarkten. ## Approach **Priority-Resolver-Helper + Producer-API-Erweiterung + Override für Internal-Jobs.** **BullMQ-Priority-Mapping:** - BullMQ akzeptiert `priority: 1-2_097_152` (1 = höchste). Wir nutzen nur 1, 5, 10 für Klarheit. - Tenant-Plan → Priority: - **Enterprise:** `priority: 1` (höchste) - **Pro:** `priority: 5` - **Free:** `priority: 10` - **Internal-/Admin-/Cron-Jobs:** immer `priority: 1` — Wartungs-Tasks dürfen nicht von User-Workload überholt werden. - **Override-Hook:** Caller kann explizit `priority: 'critical' | 'high' | 'normal' | 'low'` setzen wenn semantisch klar (z.B. Password-Reset-Mail → critical, unabhängig vom Plan). **Resolver-Logik:** ```ts async function resolveJobPriorityForTenant(tenantId: string | null): Promise { if (!tenantId) return 1; // internal job const plan = await getCachedTenantPlan(tenantId); // SharedStore-cached switch (plan) { case 'enterprise': return 1; case 'pro': return 5; case 'free': return 10; default: return 10; } } ``` **Producer-API:** ```ts enqueueJob(queueName, payload, { tenantId, // → automatischer Plan-Lookup priorityOverride?: 'critical' | 'high' | 'normal' | 'low', ...otherBullMqOpts }); ``` **Begründung gegen Alternativen:** - *Separate Queues pro Plan:* Queue-Explosion (4 Queues × 3 Plans = 12 Queues), unklar wo Internal-Jobs hingehören, Worker-Config wird komplex. BullMQ-Priority ist die idiomatische Lösung. - *Globale Priority-Quotas (z.B. „Free-Jobs max 30% Throughput"):* überengineert für unsere Größe; Standard-Priority-Order reicht. - *Caller setzt Priority manuell überall:* fehleranfällig, vergessbar; lieber zentralisiert. ## Implementation Steps 1. **Priority-Resolver-Helper** — Plan → BullMQ-Priority - Files: - `packages/queue/src/priority.ts` (neu) — exportiert `resolveJobPriorityForTenant(tenantId)`, `priorityFromOverride(override)`, Konstanten `JOB_PRIORITY_CRITICAL=1, HIGH=3, NORMAL=5, LOW=10`. - Plan-Lookup über `getCachedTenantPlan(tenantId)` (entweder hier neu oder aus `packages/billing/src/cache.ts` — Subagent prüft was existiert). - Tests: `packages/queue/src/priority.test.ts` — alle Plans, null-Case (internal), Override-Cases, Cache-Hit/Miss. - Commit: `03-scaling/job-priority: add priority resolver helper` 2. **Producer-API erweitern** — `enqueueJob` akzeptiert `tenantId` + `priorityOverride` - Files: `packages/queue/src/producers.ts` (Edit) — Signatur: ```ts enqueueJob( queueName: string, payload: T, opts?: { tenantId?: string | null; priorityOverride?: 'critical' | 'high' | 'normal' | 'low'; delay?: number; attempts?: number; backoff?: BackoffOptions; } ): Promise> ``` - Wenn `priorityOverride` gesetzt → den nehmen; sonst `resolveJobPriorityForTenant(tenantId)`; sonst Default `NORMAL`. - Tests: `packages/queue/src/producers.test.ts` (Edit) — assert dass BullMQ-Add-Call die richtige `priority` bekommt für: Enterprise-Tenant, Pro, Free, internal (null), Override. - Commit: `03-scaling/job-priority: producer accepts tenantId and priority override` 3. **Call-Site-Migration: Notifications** — `packages/notifications/src/producer.ts` - Files: alle `enqueueJob`-Calls → mit `tenantId` ergänzen. Wo `tenantId` aus Context kommt: direkt. Wo nicht: aus Payload. - Password-Reset/Magic-Link-Notifications: `priorityOverride: 'critical'`. - Commit: `03-scaling/job-priority: wire notifications producer to priorities` 4. **Call-Site-Migration: Mail** — `packages/mail/src/queue.ts` - Files: Transactional Mails → `tenantId`. Marketing-Batch-Mails → `priorityOverride: 'low'` (großvolumige Batches sollten Transactional nicht überholen). - Commit: `03-scaling/job-priority: wire mail producer to priorities` 5. **Call-Site-Migration: Webhooks** — `packages/webhooks/src/producer.ts` - Files: outbound Webhook-Delivery → `tenantId`. Retry-Jobs erben Priority vom Original-Job (BullMQ tut das automatisch). - Commit: `03-scaling/job-priority: wire webhook producer to priorities` 6. **Call-Site-Migration: Imports/Exports** — `packages/imports/src/producer.ts`, `packages/exports/src/producer.ts` - Files: `tenantId` ergänzen. Imports sind meist long-running, keine Override. - Commit: `03-scaling/job-priority: wire imports/exports producers to priorities` 7. **Call-Site-Migration: AI-Generation** — `packages/ai/src/jobs.ts` - Files: `tenantId`. Pro-/Enterprise-Tenants bekommen automatisch bessere AI-Job-Priority. - Commit: `03-scaling/job-priority: wire ai job producer to priorities` 8. **Call-Site-Migration: Workflow-Execution** — `packages/workflows/src/executor.ts` - Files: `tenantId` aus Workflow-Context. User-getriggert: Plan-Priority. System-Trigger (z.B. Schedule): `priorityOverride: 'high'` (nicht critical, das ist für Auth-Flows). - Commit: `03-scaling/job-priority: wire workflow executor to priorities` 9. **Internal/Cron-Jobs auf Critical pinnen** — `apps/api/src/cron/*.ts` - Files: alle Cron-Producer → `priorityOverride: 'critical'` (oder `tenantId: null`, was den gleichen Effekt hat). - Cleanup, Retention, Audit-Compaction, Backup-Trigger. - Commit: `03-scaling/job-priority: pin internal cron jobs to critical priority` 10. **Metrics-Gauge `queue_jobs_by_priority`** — `/metrics`-Endpoint exposed Gauge pro Queue × Priority - Files: `apps/worker/src/metrics.ts` (Edit) — periodischer Sampler (alle 10s) liest BullMQ's `getJobCounts` pro Queue + Iterate über erste 100 jobs für Priority-Verteilung. Gauge `queue_jobs_by_priority{queue, priority}`. - Tests: `apps/worker/src/metrics.test.ts` — Gauge wird gesetzt. - Commit: `03-scaling/job-priority: queue_jobs_by_priority metric gauge` 11. **Doku** — `docs/operations/job-priority.md` - Files: neue Doc. Inhalt: - Plan → Priority-Mapping-Tabelle - Override-API-Beispiele - Wann Override sinnvoll (Auth, Password-Reset, System-Critical) - BullMQ-Priority-Internals (Performance-Cost von Priority-Sorting; Hinweis dass Priority Jobs in `prioritized`-Sorted-Set landen statt `wait`-List) - Sizing-Empfehlung: pro 10k jobs/min ist Priority-Sorting noch performant; ab 100k jobs/min Separate-Queues evaluieren. - Commit: `03-scaling/job-priority: add operations doc` ## Test Requirements - **Unit-Tests:** - `packages/queue/src/priority.test.ts` — Resolver für alle Plans + null + Override-Cases. - `packages/queue/src/producers.test.ts` — assert BullMQ-Add-Call-Args (Priority-Field). - Pro Producer-Package mind. ein Test der bestätigt dass `tenantId` durchgereicht wird. - **Integration-Tests:** - `packages/queue/test/integration/priority-ordering.test.ts` — gegen Test-Redis: enqueue 3 Jobs (Enterprise/Pro/Free) in der falschen Reihenfolge, assert dass Worker sie in Priority-Reihenfolge picks. - Test mit Internal-Job + User-Jobs: Internal kommt zuerst dran. - **E2E:** kein UI-Impact → kein E2E. - **Coverage-Target:** ≥ 90% für `packages/queue/src/priority.ts` und `producers.ts`. - **Test-Files:** wie aufgelistet. **NICHT als completed markieren bevor alle Tests grün sind.** ## Acceptance Criteria - [ ] `resolveJobPriorityForTenant` und Producer-API-Erweiterung implementiert - [ ] Alle 7 Producer-Sites migriert (notifications, mail, webhook, import, export, ai, workflow) - [ ] Internal/Cron-Jobs auf `priority: 1` - [ ] Override-API funktioniert (Critical für Password-Reset-Mail verifiziert) - [ ] Integration-Test beweist Priority-Order-Enforcement durch Worker - [ ] Metric `queue_jobs_by_priority` sichtbar in `/metrics` - [ ] `docs/operations/job-priority.md` vollständig - [ ] Tests grün (lint, typecheck, unit, integration) - [ ] Staging-Verifikation: Free-Tenant-Burst von 1k Mails verzögert Enterprise-Mail nicht (gemessen via Job-Wait-Time-Histogram) - [ ] Detail-MD Progress Log aktualisiert - [ ] PROGRESS.md Status auf `completed` ## Rollback Strategy - **Code-Rollback:** Producer-Call-Site-Commits einzeln revertierbar (Steps 3-9 sind unabhängig). Resolver-Helper kann bleiben, wird einfach nicht genutzt. - **Feature-Flag:** Env-Switch `JOB_PRIORITY_ENABLED=false` → Resolver returnt immer `JOB_PRIORITY_NORMAL=5`. Implementiert in `priority.ts`. Standard `true`. - **DB:** keine Schema-Changes. - **In-Flight-Jobs:** existierende Jobs in Queue behalten ihre Priority. Nach Rollback laufen sie normal weiter. ## Security Notes - **Plan-Lookup-Cache-Poisoning:** SharedStore-Cache liest Plan aus DB; ein kompromittierter Cache könnte Free-User zu Enterprise-Priority hochstufen. Mitigation: TTL kurz (5min), kein Direct-Write-Path; falls SharedStore-Compromise → größeres Sicherheitsproblem als nur Priority. - **Priority-Inflation-Risk:** Nichts hält Caller-Code davon ab `priorityOverride: 'critical'` für alles zu setzen → Effekt-Nullifizierung. Mitigation via Code-Review + Linting-Rule die `priorityOverride: 'critical'` außerhalb von `notifications/` und `auth/` flagged. - **Internal-Job-Spoofing:** `tenantId: null` triggert Critical-Priority. Nur Server-Code kann das setzen (keine User-Input-Pfade). Trotzdem: Producer-Signatur sollte `priorityOverride: 'critical'` nicht aus untrusted Input akzeptieren. ## Subagent Recommendations - **Planning:** `Plan`-Subagent mit Detail-MD + Audit der 7 Producer-Files. - **Research:** `Explore`-Subagent listet alle `enqueueJob`-Aufrufe im Codebase (rg-Search) → exhaustive Migration-Checkliste. - **Implementation:** `general-purpose`-Subagent pro Step. Steps 3-9 (Call-Site-Migration) können parallelisiert werden (verschiedene Packages). - **Tests:** separater `general-purpose`-Subagent — Integration-Test mit Test-Redis ist non-trivial, eigener Subagent. - **Verification:** `gsd-verifier`-Subagent gegen Acceptance-Criteria. - **Code-Review:** `gsd-code-reviewer` nach Phasen-Abschluss. ## Staging-Deployment-Steps 1. Branch `agent/03-job-priority` pushen. 2. Coolify-Auto-Deploy auf Staging. 3. Smoke-Test: ```bash # Setup: 3 Test-Tenants (Free, Pro, Enterprise) # Burst: 100 Jobs/Tenant in eine Queue (z.B. mail-transactional) pnpm tsx scripts/load-test-priority.ts \ --queue mail-transactional \ --tenants free:100,pro:100,enterprise:100 # Assert: Worker processed Enterprise-Jobs first (in Logs ablesbar oder via BullMQ-Dashboard) # Assert: Wait-Time-Histogram zeigt Enterprise p95 < Pro p95 < Free p95 ``` 4. Grafana-Dashboard: `queue_jobs_by_priority`-Gauge zeigt 3 verschiedene Priority-Levels gleichzeitig. 5. Manual-Test: Password-Reset-Trigger aus UI → Mail-Job in Queue mit `priority: 1` (BullMQ-Inspector verifiziert). ## Deviations from Plan - **Spec assumed dedicated packages (`packages/notifications`, `packages/webhooks`, `packages/imports`, `packages/exports`, `packages/ai`, `packages/workflows`) and an `apps/api/src/cron/` directory.** None of those exist in this monorepo yet. The actual producer call-sites are in `apps/web/app/modules/notifications/services/.server/NotificationService.ts`, `packages/mail/src/queue.ts`, `apps/web/app/modules/emails/services/TransactionalEmailQueue.server.ts`, `apps/api/src/webhookDeliveryQueue.ts`, `apps/api/src/exportRetentionQueue.ts`, `apps/web/app/routes/app.$tenant/internal/imports/submit.tsx`, and `apps/web/app/utils/integrations/fileScanQueue.server.ts`. Each was migrated. - **No producer call-sites exist for AI / browser-automation / workflow jobs yet** — the descriptors are defined in `packages/queue/src/producers.ts` but no app code invokes `buildAiChatCompletionJob` etc. Once those producers land, callers must pass `tenantId` per the convention in `docs/architecture/job-priority.md`. - **Metric `queue_jobs_enqueued_total{queue, priority}`** is implemented as a Counter rather than the spec's Gauge `queue_jobs_by_priority`. The Counter pattern matches the existing `queue_jobs_completed_total` shape, doesn't require periodic sampling, and is what producers naturally emit. A separate Gauge sampler for the priority distribution of *waiting* jobs can be added later if needed. - **Tenant-plan classifier is intentionally not hard-coded.** The boilerplate has no `enterprise|pro|free` field on tenants — operators wire their own subscription→plan-tier function via `setSubscriptionToPlanTier`. Until that is set, the resolver returns NORMAL for every tenant, which the spec explicitly accepts as v1. ## Progress Log - 2026-05-17 — `e5994239` — priority resolver helper + 20 unit tests (`packages/queue/src/priority.ts`, `priority.test.ts`, `index.ts`, `types.ts`). - 2026-05-17 — `39ec090b` — `enqueueJob` accepts `tenantId` + `priorityOverride` + numeric `priority`, plus optional `EnqueueObserver` so apps can forward enqueue events without coupling the queue package to prom-client. 10 new producer-test cases. - 2026-05-17 — `d26752b3` — `queue_jobs_enqueued_total{queue, priority}` Counter added to the metrics package; api/web/worker registries register the observer during boot. - 2026-05-17 — `bf806d46` — NotificationService.send passes targetTenantId to enqueueJob; regression test added. - 2026-05-17 — `433997bd` — Mail producer (`packages/mail/src/queue.ts`) accepts `tenantId` + `priorityOverride` on `SendMailInput`. `queueTransactionalEmail` / `sendEmail` forward both. Auth-flow mail (password-reset, magic-link, email-verification, user-invitation) pinned to `priorityOverride: 'critical'`. Affected unit tests updated. - 2026-05-17 — `c9e8ce38` — `queueLegacyEventWebhookAttemptDelivery` passes `attempt.tenantId`; test updated. - 2026-05-17 — `75715de4` — Imports submit route passes `dryRunJob.tenantId`. - 2026-05-17 — `2a177942` — `enqueueFileScan` passes `input.tenantId`. - 2026-05-17 — `4dc1b784` — `queueExportArtifactRetention` pinned to CRITICAL via `tenantId: null`; test asserts priority=1. - 2026-05-17 — `2596c746` — `tenantPlanLookup.server.ts` wires the subscription-cache as the queue resolver's plan lookup; registered during cache bootstrap. 5 new unit tests. - 2026-05-17 — `dfeb4204` — `docs/architecture/job-priority.md` covers resolver order, plan-tier mapping, producer guidelines, feature-flag, and Grafana queries. ## Lessons Learned (Optional am Task-Ende.)