16 KiB
Plan-basierte Job-Priorities in BullMQ
Status: completed Task-ID: #47 Phase: 03 Estimated Effort: 1-2 Tage Owner: unassigned
Problem
Alle Jobs werden aktuell mit identischer Priorität enqueued (BullMQ-Default: 0 = highest). Folgen:
- Enterprise-Tenants warten in der Queue hinter Free-Tier-Tenants. Ein Free-Tenant der 10k Mail-Jobs schedulet blockt Enterprise-Tenants die zahlende Kunden haben.
- Interne Jobs (cron-retention, audit-cleanup) konkurrieren mit User-Jobs — sie sollten höchste Priorität haben damit Wartung nicht verschleppt wird, tun es aber nicht.
- Keine Differenzierung zwischen Real-Time-Notification (User wartet im UI) und Batch-Mail (asynchron) möglich.
- Kein Hebel für QoS-Commitments in Verträgen („Enterprise SLA: alle Jobs < 30s").
Betroffene Codestellen:
packages/queue/src/producers.ts—enqueueJob(queueName, payload, opts?)ignoriert Priority- Alle Producer-Call-Sites:
packages/notifications/src/producer.tspackages/mail/src/queue.tspackages/webhooks/src/producer.tspackages/imports/src/producer.tspackages/exports/src/producer.tspackages/ai/src/jobs.tspackages/workflows/src/executor.tsapps/api/src/cron/*.ts
Depends on
03-scaling-infrastructure/shared-store-package.md— Tenant-Plan-Lookup cached im SharedStore (TTL 5min) damitresolveJobPriorityForTenantnicht jeden enqueue-Call zur DB schickt.02-observability/metrics-grafana.md— Counterqueue_jobs_by_prioritybraucht das/metrics-Endpoint.
Blocks
- Keine harten Blocker, aber:
03-scaling-infrastructure/worker-queues.md— Worker-Trennung amplifiziert den Wert (Priority innerhalb einer Queue + Trennung der Queues = bessere QoS).05-business-features/ai-cost-phase-2.md— Quota-Enforcement kann Job-Priority-Boost als Pro-Plan-Feature vermarkten.
Approach
Priority-Resolver-Helper + Producer-API-Erweiterung + Override für Internal-Jobs.
BullMQ-Priority-Mapping:
- BullMQ akzeptiert
priority: 1-2_097_152(1 = höchste). Wir nutzen nur 1, 5, 10 für Klarheit. - Tenant-Plan → Priority:
- Enterprise:
priority: 1(höchste) - Pro:
priority: 5 - Free:
priority: 10
- Enterprise:
- Internal-/Admin-/Cron-Jobs: immer
priority: 1— Wartungs-Tasks dürfen nicht von User-Workload überholt werden. - Override-Hook: Caller kann explizit
priority: 'critical' | 'high' | 'normal' | 'low'setzen wenn semantisch klar (z.B. Password-Reset-Mail → critical, unabhängig vom Plan).
Resolver-Logik:
async function resolveJobPriorityForTenant(tenantId: string | null): Promise<number> {
if (!tenantId) return 1; // internal job
const plan = await getCachedTenantPlan(tenantId); // SharedStore-cached
switch (plan) {
case 'enterprise': return 1;
case 'pro': return 5;
case 'free': return 10;
default: return 10;
}
}
Producer-API:
enqueueJob(queueName, payload, {
tenantId, // → automatischer Plan-Lookup
priorityOverride?: 'critical' | 'high' | 'normal' | 'low',
...otherBullMqOpts
});
Begründung gegen Alternativen:
- Separate Queues pro Plan: Queue-Explosion (4 Queues × 3 Plans = 12 Queues), unklar wo Internal-Jobs hingehören, Worker-Config wird komplex. BullMQ-Priority ist die idiomatische Lösung.
- Globale Priority-Quotas (z.B. „Free-Jobs max 30% Throughput"): überengineert für unsere Größe; Standard-Priority-Order reicht.
- Caller setzt Priority manuell überall: fehleranfällig, vergessbar; lieber zentralisiert.
Implementation Steps
-
Priority-Resolver-Helper — Plan → BullMQ-Priority
- Files:
packages/queue/src/priority.ts(neu) — exportiertresolveJobPriorityForTenant(tenantId),priorityFromOverride(override), KonstantenJOB_PRIORITY_CRITICAL=1, HIGH=3, NORMAL=5, LOW=10.- Plan-Lookup über
getCachedTenantPlan(tenantId)(entweder hier neu oder auspackages/billing/src/cache.ts— Subagent prüft was existiert).
- Tests:
packages/queue/src/priority.test.ts— alle Plans, null-Case (internal), Override-Cases, Cache-Hit/Miss. - Commit:
03-scaling/job-priority: add priority resolver helper
- Files:
-
Producer-API erweitern —
enqueueJobakzeptierttenantId+priorityOverride- Files:
packages/queue/src/producers.ts(Edit) — Signatur:enqueueJob<T>( queueName: string, payload: T, opts?: { tenantId?: string | null; priorityOverride?: 'critical' | 'high' | 'normal' | 'low'; delay?: number; attempts?: number; backoff?: BackoffOptions; } ): Promise<Job<T>> - Wenn
priorityOverridegesetzt → den nehmen; sonstresolveJobPriorityForTenant(tenantId); sonst DefaultNORMAL. - Tests:
packages/queue/src/producers.test.ts(Edit) — assert dass BullMQ-Add-Call die richtigeprioritybekommt für: Enterprise-Tenant, Pro, Free, internal (null), Override. - Commit:
03-scaling/job-priority: producer accepts tenantId and priority override
- Files:
-
Call-Site-Migration: Notifications —
packages/notifications/src/producer.ts- Files: alle
enqueueJob-Calls → mittenantIdergänzen. WotenantIdaus Context kommt: direkt. Wo nicht: aus Payload. - Password-Reset/Magic-Link-Notifications:
priorityOverride: 'critical'. - Commit:
03-scaling/job-priority: wire notifications producer to priorities
- Files: alle
-
Call-Site-Migration: Mail —
packages/mail/src/queue.ts- Files: Transactional Mails →
tenantId. Marketing-Batch-Mails →priorityOverride: 'low'(großvolumige Batches sollten Transactional nicht überholen). - Commit:
03-scaling/job-priority: wire mail producer to priorities
- Files: Transactional Mails →
-
Call-Site-Migration: Webhooks —
packages/webhooks/src/producer.ts- Files: outbound Webhook-Delivery →
tenantId. Retry-Jobs erben Priority vom Original-Job (BullMQ tut das automatisch). - Commit:
03-scaling/job-priority: wire webhook producer to priorities
- Files: outbound Webhook-Delivery →
-
Call-Site-Migration: Imports/Exports —
packages/imports/src/producer.ts,packages/exports/src/producer.ts- Files:
tenantIdergänzen. Imports sind meist long-running, keine Override. - Commit:
03-scaling/job-priority: wire imports/exports producers to priorities
- Files:
-
Call-Site-Migration: AI-Generation —
packages/ai/src/jobs.ts- Files:
tenantId. Pro-/Enterprise-Tenants bekommen automatisch bessere AI-Job-Priority. - Commit:
03-scaling/job-priority: wire ai job producer to priorities
- Files:
-
Call-Site-Migration: Workflow-Execution —
packages/workflows/src/executor.ts- Files:
tenantIdaus Workflow-Context. User-getriggert: Plan-Priority. System-Trigger (z.B. Schedule):priorityOverride: 'high'(nicht critical, das ist für Auth-Flows). - Commit:
03-scaling/job-priority: wire workflow executor to priorities
- Files:
-
Internal/Cron-Jobs auf Critical pinnen —
apps/api/src/cron/*.ts- Files: alle Cron-Producer →
priorityOverride: 'critical'(odertenantId: null, was den gleichen Effekt hat). - Cleanup, Retention, Audit-Compaction, Backup-Trigger.
- Commit:
03-scaling/job-priority: pin internal cron jobs to critical priority
- Files: alle Cron-Producer →
-
Metrics-Gauge
queue_jobs_by_priority—/metrics-Endpoint exposed Gauge pro Queue × Priority- Files:
apps/worker/src/metrics.ts(Edit) — periodischer Sampler (alle 10s) liest BullMQ'sgetJobCountspro Queue + Iterate über erste 100 jobs für Priority-Verteilung. Gaugequeue_jobs_by_priority{queue, priority}. - Tests:
apps/worker/src/metrics.test.ts— Gauge wird gesetzt. - Commit:
03-scaling/job-priority: queue_jobs_by_priority metric gauge
- Files:
-
Doku —
docs/operations/job-priority.md- Files: neue Doc. Inhalt:
- Plan → Priority-Mapping-Tabelle
- Override-API-Beispiele
- Wann Override sinnvoll (Auth, Password-Reset, System-Critical)
- BullMQ-Priority-Internals (Performance-Cost von Priority-Sorting; Hinweis dass Priority Jobs in
prioritized-Sorted-Set landen stattwait-List) - Sizing-Empfehlung: pro 10k jobs/min ist Priority-Sorting noch performant; ab 100k jobs/min Separate-Queues evaluieren.
- Commit:
03-scaling/job-priority: add operations doc
- Files: neue Doc. Inhalt:
Test Requirements
- Unit-Tests:
packages/queue/src/priority.test.ts— Resolver für alle Plans + null + Override-Cases.packages/queue/src/producers.test.ts— assert BullMQ-Add-Call-Args (Priority-Field).- Pro Producer-Package mind. ein Test der bestätigt dass
tenantIddurchgereicht wird.
- Integration-Tests:
packages/queue/test/integration/priority-ordering.test.ts— gegen Test-Redis: enqueue 3 Jobs (Enterprise/Pro/Free) in der falschen Reihenfolge, assert dass Worker sie in Priority-Reihenfolge picks.- Test mit Internal-Job + User-Jobs: Internal kommt zuerst dran.
- E2E: kein UI-Impact → kein E2E.
- Coverage-Target: ≥ 90% für
packages/queue/src/priority.tsundproducers.ts. - Test-Files: wie aufgelistet.
NICHT als completed markieren bevor alle Tests grün sind.
Acceptance Criteria
resolveJobPriorityForTenantund Producer-API-Erweiterung implementiert- Alle 7 Producer-Sites migriert (notifications, mail, webhook, import, export, ai, workflow)
- Internal/Cron-Jobs auf
priority: 1 - Override-API funktioniert (Critical für Password-Reset-Mail verifiziert)
- Integration-Test beweist Priority-Order-Enforcement durch Worker
- Metric
queue_jobs_by_prioritysichtbar in/metrics docs/operations/job-priority.mdvollständig- Tests grün (lint, typecheck, unit, integration)
- Staging-Verifikation: Free-Tenant-Burst von 1k Mails verzögert Enterprise-Mail nicht (gemessen via Job-Wait-Time-Histogram)
- Detail-MD Progress Log aktualisiert
- PROGRESS.md Status auf
completed
Rollback Strategy
- Code-Rollback: Producer-Call-Site-Commits einzeln revertierbar (Steps 3-9 sind unabhängig). Resolver-Helper kann bleiben, wird einfach nicht genutzt.
- Feature-Flag: Env-Switch
JOB_PRIORITY_ENABLED=false→ Resolver returnt immerJOB_PRIORITY_NORMAL=5. Implementiert inpriority.ts. Standardtrue. - DB: keine Schema-Changes.
- In-Flight-Jobs: existierende Jobs in Queue behalten ihre Priority. Nach Rollback laufen sie normal weiter.
Security Notes
- Plan-Lookup-Cache-Poisoning: SharedStore-Cache liest Plan aus DB; ein kompromittierter Cache könnte Free-User zu Enterprise-Priority hochstufen. Mitigation: TTL kurz (5min), kein Direct-Write-Path; falls SharedStore-Compromise → größeres Sicherheitsproblem als nur Priority.
- Priority-Inflation-Risk: Nichts hält Caller-Code davon ab
priorityOverride: 'critical'für alles zu setzen → Effekt-Nullifizierung. Mitigation via Code-Review + Linting-Rule diepriorityOverride: 'critical'außerhalb vonnotifications/undauth/flagged. - Internal-Job-Spoofing:
tenantId: nulltriggert Critical-Priority. Nur Server-Code kann das setzen (keine User-Input-Pfade). Trotzdem: Producer-Signatur solltepriorityOverride: 'critical'nicht aus untrusted Input akzeptieren.
Subagent Recommendations
- Planning:
Plan-Subagent mit Detail-MD + Audit der 7 Producer-Files. - Research:
Explore-Subagent listet alleenqueueJob-Aufrufe im Codebase (rg-Search) → exhaustive Migration-Checkliste. - Implementation:
general-purpose-Subagent pro Step. Steps 3-9 (Call-Site-Migration) können parallelisiert werden (verschiedene Packages). - Tests: separater
general-purpose-Subagent — Integration-Test mit Test-Redis ist non-trivial, eigener Subagent. - Verification:
gsd-verifier-Subagent gegen Acceptance-Criteria. - Code-Review:
gsd-code-reviewernach Phasen-Abschluss.
Staging-Deployment-Steps
- Branch
agent/03-job-prioritypushen. - Coolify-Auto-Deploy auf Staging.
- Smoke-Test:
# Setup: 3 Test-Tenants (Free, Pro, Enterprise) # Burst: 100 Jobs/Tenant in eine Queue (z.B. mail-transactional) pnpm tsx scripts/load-test-priority.ts \ --queue mail-transactional \ --tenants free:100,pro:100,enterprise:100 # Assert: Worker processed Enterprise-Jobs first (in Logs ablesbar oder via BullMQ-Dashboard) # Assert: Wait-Time-Histogram zeigt Enterprise p95 < Pro p95 < Free p95 - Grafana-Dashboard:
queue_jobs_by_priority-Gauge zeigt 3 verschiedene Priority-Levels gleichzeitig. - Manual-Test: Password-Reset-Trigger aus UI → Mail-Job in Queue mit
priority: 1(BullMQ-Inspector verifiziert).
Deviations from Plan
- Spec assumed dedicated packages (
packages/notifications,packages/webhooks,packages/imports,packages/exports,packages/ai,packages/workflows) and anapps/api/src/cron/directory. None of those exist in this monorepo yet. The actual producer call-sites are inapps/web/app/modules/notifications/services/.server/NotificationService.ts,packages/mail/src/queue.ts,apps/web/app/modules/emails/services/TransactionalEmailQueue.server.ts,apps/api/src/webhookDeliveryQueue.ts,apps/api/src/exportRetentionQueue.ts,apps/web/app/routes/app.$tenant/internal/imports/submit.tsx, andapps/web/app/utils/integrations/fileScanQueue.server.ts. Each was migrated. - No producer call-sites exist for AI / browser-automation / workflow jobs yet — the descriptors are defined in
packages/queue/src/producers.tsbut no app code invokesbuildAiChatCompletionJobetc. Once those producers land, callers must passtenantIdper the convention indocs/architecture/job-priority.md. - Metric
queue_jobs_enqueued_total{queue, priority}is implemented as a Counter rather than the spec's Gaugequeue_jobs_by_priority. The Counter pattern matches the existingqueue_jobs_completed_totalshape, doesn't require periodic sampling, and is what producers naturally emit. A separate Gauge sampler for the priority distribution of waiting jobs can be added later if needed. - Tenant-plan classifier is intentionally not hard-coded. The boilerplate has no
enterprise|pro|freefield on tenants — operators wire their own subscription→plan-tier function viasetSubscriptionToPlanTier. Until that is set, the resolver returns NORMAL for every tenant, which the spec explicitly accepts as v1.
Progress Log
- 2026-05-17 —
e5994239— priority resolver helper + 20 unit tests (packages/queue/src/priority.ts,priority.test.ts,index.ts,types.ts). - 2026-05-17 —
39ec090b—enqueueJobacceptstenantId+priorityOverride+ numericpriority, plus optionalEnqueueObserverso apps can forward enqueue events without coupling the queue package to prom-client. 10 new producer-test cases. - 2026-05-17 —
d26752b3—queue_jobs_enqueued_total{queue, priority}Counter added to the metrics package; api/web/worker registries register the observer during boot. - 2026-05-17 —
bf806d46— NotificationService.send passes targetTenantId to enqueueJob; regression test added. - 2026-05-17 —
433997bd— Mail producer (packages/mail/src/queue.ts) acceptstenantId+priorityOverrideonSendMailInput.queueTransactionalEmail/sendEmailforward both. Auth-flow mail (password-reset, magic-link, email-verification, user-invitation) pinned topriorityOverride: 'critical'. Affected unit tests updated. - 2026-05-17 —
c9e8ce38—queueLegacyEventWebhookAttemptDeliverypassesattempt.tenantId; test updated. - 2026-05-17 —
75715de4— Imports submit route passesdryRunJob.tenantId. - 2026-05-17 —
2a177942—enqueueFileScanpassesinput.tenantId. - 2026-05-17 —
4dc1b784—queueExportArtifactRetentionpinned to CRITICAL viatenantId: null; test asserts priority=1. - 2026-05-17 —
2596c746—tenantPlanLookup.server.tswires the subscription-cache as the queue resolver's plan lookup; registered during cache bootstrap. 5 new unit tests. - 2026-05-17 —
dfeb4204—docs/architecture/job-priority.mdcovers resolver order, plan-tier mapping, producer guidelines, feature-flag, and Grafana queries.
Lessons Learned
(Optional am Task-Ende.)