Files
HiveGrid-Wireframes/hivegrid-cloud/docs/improvement-plan/03-scaling-infrastructure/job-priority.md
T

16 KiB
Raw Blame History

Plan-basierte Job-Priorities in BullMQ

Status: completed Task-ID: #47 Phase: 03 Estimated Effort: 1-2 Tage Owner: unassigned

Problem

Alle Jobs werden aktuell mit identischer Priorität enqueued (BullMQ-Default: 0 = highest). Folgen:

  • Enterprise-Tenants warten in der Queue hinter Free-Tier-Tenants. Ein Free-Tenant der 10k Mail-Jobs schedulet blockt Enterprise-Tenants die zahlende Kunden haben.
  • Interne Jobs (cron-retention, audit-cleanup) konkurrieren mit User-Jobs — sie sollten höchste Priorität haben damit Wartung nicht verschleppt wird, tun es aber nicht.
  • Keine Differenzierung zwischen Real-Time-Notification (User wartet im UI) und Batch-Mail (asynchron) möglich.
  • Kein Hebel für QoS-Commitments in Verträgen („Enterprise SLA: alle Jobs < 30s").

Betroffene Codestellen:

  • packages/queue/src/producers.ts — enqueueJob(queueName, payload, opts?) ignoriert Priority
  • Alle Producer-Call-Sites:
    • packages/notifications/src/producer.ts
    • packages/mail/src/queue.ts
    • packages/webhooks/src/producer.ts
    • packages/imports/src/producer.ts
    • packages/exports/src/producer.ts
    • packages/ai/src/jobs.ts
    • packages/workflows/src/executor.ts
    • apps/api/src/cron/*.ts

Depends on

  • 03-scaling-infrastructure/shared-store-package.md — Tenant-Plan-Lookup cached im SharedStore (TTL 5min) damit resolveJobPriorityForTenant nicht jeden enqueue-Call zur DB schickt.
  • 02-observability/metrics-grafana.md — Counter queue_jobs_by_priority braucht das /metrics-Endpoint.

Blocks

  • Keine harten Blocker, aber:
    • 03-scaling-infrastructure/worker-queues.md — Worker-Trennung amplifiziert den Wert (Priority innerhalb einer Queue + Trennung der Queues = bessere QoS).
    • 05-business-features/ai-cost-phase-2.md — Quota-Enforcement kann Job-Priority-Boost als Pro-Plan-Feature vermarkten.

Approach

Priority-Resolver-Helper + Producer-API-Erweiterung + Override für Internal-Jobs.

BullMQ-Priority-Mapping:

  • BullMQ akzeptiert priority: 1-2_097_152 (1 = höchste). Wir nutzen nur 1, 5, 10 für Klarheit.
  • Tenant-Plan → Priority:
    • Enterprise: priority: 1 (höchste)
    • Pro: priority: 5
    • Free: priority: 10
  • Internal-/Admin-/Cron-Jobs: immer priority: 1 — Wartungs-Tasks dürfen nicht von User-Workload überholt werden.
  • Override-Hook: Caller kann explizit priority: 'critical' | 'high' | 'normal' | 'low' setzen wenn semantisch klar (z.B. Password-Reset-Mail → critical, unabhängig vom Plan).

Resolver-Logik:

async function resolveJobPriorityForTenant(tenantId: string | null): Promise<number> {
  if (!tenantId) return 1; // internal job
  const plan = await getCachedTenantPlan(tenantId); // SharedStore-cached
  switch (plan) {
    case 'enterprise': return 1;
    case 'pro': return 5;
    case 'free': return 10;
    default: return 10;
  }
}

Producer-API:

enqueueJob(queueName, payload, {
  tenantId,                 // → automatischer Plan-Lookup
  priorityOverride?: 'critical' | 'high' | 'normal' | 'low',
  ...otherBullMqOpts
});

Begründung gegen Alternativen:

  • Separate Queues pro Plan: Queue-Explosion (4 Queues × 3 Plans = 12 Queues), unklar wo Internal-Jobs hingehören, Worker-Config wird komplex. BullMQ-Priority ist die idiomatische Lösung.
  • Globale Priority-Quotas (z.B. „Free-Jobs max 30% Throughput"): überengineert für unsere Größe; Standard-Priority-Order reicht.
  • Caller setzt Priority manuell überall: fehleranfällig, vergessbar; lieber zentralisiert.

Implementation Steps

  1. Priority-Resolver-Helper — Plan → BullMQ-Priority

    • Files:
      • packages/queue/src/priority.ts (neu) — exportiert resolveJobPriorityForTenant(tenantId), priorityFromOverride(override), Konstanten JOB_PRIORITY_CRITICAL=1, HIGH=3, NORMAL=5, LOW=10.
      • Plan-Lookup über getCachedTenantPlan(tenantId) (entweder hier neu oder aus packages/billing/src/cache.ts — Subagent prüft was existiert).
    • Tests: packages/queue/src/priority.test.ts — alle Plans, null-Case (internal), Override-Cases, Cache-Hit/Miss.
    • Commit: 03-scaling/job-priority: add priority resolver helper
  2. Producer-API erweitern — enqueueJob akzeptiert tenantId + priorityOverride

    • Files: packages/queue/src/producers.ts (Edit) — Signatur:
      enqueueJob<T>(
        queueName: string,
        payload: T,
        opts?: {
          tenantId?: string | null;
          priorityOverride?: 'critical' | 'high' | 'normal' | 'low';
          delay?: number;
          attempts?: number;
          backoff?: BackoffOptions;
        }
      ): Promise<Job<T>>
      
    • Wenn priorityOverride gesetzt → den nehmen; sonst resolveJobPriorityForTenant(tenantId); sonst Default NORMAL.
    • Tests: packages/queue/src/producers.test.ts (Edit) — assert dass BullMQ-Add-Call die richtige priority bekommt für: Enterprise-Tenant, Pro, Free, internal (null), Override.
    • Commit: 03-scaling/job-priority: producer accepts tenantId and priority override
  3. Call-Site-Migration: Notifications — packages/notifications/src/producer.ts

    • Files: alle enqueueJob-Calls → mit tenantId ergänzen. Wo tenantId aus Context kommt: direkt. Wo nicht: aus Payload.
    • Password-Reset/Magic-Link-Notifications: priorityOverride: 'critical'.
    • Commit: 03-scaling/job-priority: wire notifications producer to priorities
  4. Call-Site-Migration: Mail — packages/mail/src/queue.ts

    • Files: Transactional Mails → tenantId. Marketing-Batch-Mails → priorityOverride: 'low' (großvolumige Batches sollten Transactional nicht überholen).
    • Commit: 03-scaling/job-priority: wire mail producer to priorities
  5. Call-Site-Migration: Webhooks — packages/webhooks/src/producer.ts

    • Files: outbound Webhook-Delivery → tenantId. Retry-Jobs erben Priority vom Original-Job (BullMQ tut das automatisch).
    • Commit: 03-scaling/job-priority: wire webhook producer to priorities
  6. Call-Site-Migration: Imports/Exports — packages/imports/src/producer.ts, packages/exports/src/producer.ts

    • Files: tenantId ergänzen. Imports sind meist long-running, keine Override.
    • Commit: 03-scaling/job-priority: wire imports/exports producers to priorities
  7. Call-Site-Migration: AI-Generation — packages/ai/src/jobs.ts

    • Files: tenantId. Pro-/Enterprise-Tenants bekommen automatisch bessere AI-Job-Priority.
    • Commit: 03-scaling/job-priority: wire ai job producer to priorities
  8. Call-Site-Migration: Workflow-Execution — packages/workflows/src/executor.ts

    • Files: tenantId aus Workflow-Context. User-getriggert: Plan-Priority. System-Trigger (z.B. Schedule): priorityOverride: 'high' (nicht critical, das ist für Auth-Flows).
    • Commit: 03-scaling/job-priority: wire workflow executor to priorities
  9. Internal/Cron-Jobs auf Critical pinnen — apps/api/src/cron/*.ts

    • Files: alle Cron-Producer → priorityOverride: 'critical' (oder tenantId: null, was den gleichen Effekt hat).
    • Cleanup, Retention, Audit-Compaction, Backup-Trigger.
    • Commit: 03-scaling/job-priority: pin internal cron jobs to critical priority
  10. Metrics-Gauge queue_jobs_by_priority — /metrics-Endpoint exposed Gauge pro Queue × Priority

    • Files: apps/worker/src/metrics.ts (Edit) — periodischer Sampler (alle 10s) liest BullMQ's getJobCounts pro Queue + Iterate über erste 100 jobs für Priority-Verteilung. Gauge queue_jobs_by_priority{queue, priority}.
    • Tests: apps/worker/src/metrics.test.ts — Gauge wird gesetzt.
    • Commit: 03-scaling/job-priority: queue_jobs_by_priority metric gauge
  11. Doku — docs/operations/job-priority.md

    • Files: neue Doc. Inhalt:
      • Plan → Priority-Mapping-Tabelle
      • Override-API-Beispiele
      • Wann Override sinnvoll (Auth, Password-Reset, System-Critical)
      • BullMQ-Priority-Internals (Performance-Cost von Priority-Sorting; Hinweis dass Priority Jobs in prioritized-Sorted-Set landen statt wait-List)
      • Sizing-Empfehlung: pro 10k jobs/min ist Priority-Sorting noch performant; ab 100k jobs/min Separate-Queues evaluieren.
    • Commit: 03-scaling/job-priority: add operations doc

Test Requirements

  • Unit-Tests:
    • packages/queue/src/priority.test.ts — Resolver für alle Plans + null + Override-Cases.
    • packages/queue/src/producers.test.ts — assert BullMQ-Add-Call-Args (Priority-Field).
    • Pro Producer-Package mind. ein Test der bestätigt dass tenantId durchgereicht wird.
  • Integration-Tests:
    • packages/queue/test/integration/priority-ordering.test.ts — gegen Test-Redis: enqueue 3 Jobs (Enterprise/Pro/Free) in der falschen Reihenfolge, assert dass Worker sie in Priority-Reihenfolge picks.
    • Test mit Internal-Job + User-Jobs: Internal kommt zuerst dran.
  • E2E: kein UI-Impact → kein E2E.
  • Coverage-Target: ≥ 90% für packages/queue/src/priority.ts und producers.ts.
  • Test-Files: wie aufgelistet.

NICHT als completed markieren bevor alle Tests grün sind.

Acceptance Criteria

  • resolveJobPriorityForTenant und Producer-API-Erweiterung implementiert
  • Alle 7 Producer-Sites migriert (notifications, mail, webhook, import, export, ai, workflow)
  • Internal/Cron-Jobs auf priority: 1
  • Override-API funktioniert (Critical für Password-Reset-Mail verifiziert)
  • Integration-Test beweist Priority-Order-Enforcement durch Worker
  • Metric queue_jobs_by_priority sichtbar in /metrics
  • docs/operations/job-priority.md vollständig
  • Tests grün (lint, typecheck, unit, integration)
  • Staging-Verifikation: Free-Tenant-Burst von 1k Mails verzögert Enterprise-Mail nicht (gemessen via Job-Wait-Time-Histogram)
  • Detail-MD Progress Log aktualisiert
  • PROGRESS.md Status auf completed

Rollback Strategy

  • Code-Rollback: Producer-Call-Site-Commits einzeln revertierbar (Steps 3-9 sind unabhängig). Resolver-Helper kann bleiben, wird einfach nicht genutzt.
  • Feature-Flag: Env-Switch JOB_PRIORITY_ENABLED=false → Resolver returnt immer JOB_PRIORITY_NORMAL=5. Implementiert in priority.ts. Standard true.
  • DB: keine Schema-Changes.
  • In-Flight-Jobs: existierende Jobs in Queue behalten ihre Priority. Nach Rollback laufen sie normal weiter.

Security Notes

  • Plan-Lookup-Cache-Poisoning: SharedStore-Cache liest Plan aus DB; ein kompromittierter Cache könnte Free-User zu Enterprise-Priority hochstufen. Mitigation: TTL kurz (5min), kein Direct-Write-Path; falls SharedStore-Compromise → größeres Sicherheitsproblem als nur Priority.
  • Priority-Inflation-Risk: Nichts hält Caller-Code davon ab priorityOverride: 'critical' für alles zu setzen → Effekt-Nullifizierung. Mitigation via Code-Review + Linting-Rule die priorityOverride: 'critical' außerhalb von notifications/ und auth/ flagged.
  • Internal-Job-Spoofing: tenantId: null triggert Critical-Priority. Nur Server-Code kann das setzen (keine User-Input-Pfade). Trotzdem: Producer-Signatur sollte priorityOverride: 'critical' nicht aus untrusted Input akzeptieren.

Subagent Recommendations

  • Planning: Plan-Subagent mit Detail-MD + Audit der 7 Producer-Files.
  • Research: Explore-Subagent listet alle enqueueJob-Aufrufe im Codebase (rg-Search) → exhaustive Migration-Checkliste.
  • Implementation: general-purpose-Subagent pro Step. Steps 3-9 (Call-Site-Migration) können parallelisiert werden (verschiedene Packages).
  • Tests: separater general-purpose-Subagent — Integration-Test mit Test-Redis ist non-trivial, eigener Subagent.
  • Verification: gsd-verifier-Subagent gegen Acceptance-Criteria.
  • Code-Review: gsd-code-reviewer nach Phasen-Abschluss.

Staging-Deployment-Steps

  1. Branch agent/03-job-priority pushen.
  2. Coolify-Auto-Deploy auf Staging.
  3. Smoke-Test:
    # Setup: 3 Test-Tenants (Free, Pro, Enterprise)
    # Burst: 100 Jobs/Tenant in eine Queue (z.B. mail-transactional)
    pnpm tsx scripts/load-test-priority.ts \
      --queue mail-transactional \
      --tenants free:100,pro:100,enterprise:100
    
    # Assert: Worker processed Enterprise-Jobs first (in Logs ablesbar oder via BullMQ-Dashboard)
    # Assert: Wait-Time-Histogram zeigt Enterprise p95 < Pro p95 < Free p95
    
  4. Grafana-Dashboard: queue_jobs_by_priority-Gauge zeigt 3 verschiedene Priority-Levels gleichzeitig.
  5. Manual-Test: Password-Reset-Trigger aus UI → Mail-Job in Queue mit priority: 1 (BullMQ-Inspector verifiziert).

Deviations from Plan

  • Spec assumed dedicated packages (packages/notifications, packages/webhooks, packages/imports, packages/exports, packages/ai, packages/workflows) and an apps/api/src/cron/ directory. None of those exist in this monorepo yet. The actual producer call-sites are in apps/web/app/modules/notifications/services/.server/NotificationService.ts, packages/mail/src/queue.ts, apps/web/app/modules/emails/services/TransactionalEmailQueue.server.ts, apps/api/src/webhookDeliveryQueue.ts, apps/api/src/exportRetentionQueue.ts, apps/web/app/routes/app.$tenant/internal/imports/submit.tsx, and apps/web/app/utils/integrations/fileScanQueue.server.ts. Each was migrated.
  • No producer call-sites exist for AI / browser-automation / workflow jobs yet — the descriptors are defined in packages/queue/src/producers.ts but no app code invokes buildAiChatCompletionJob etc. Once those producers land, callers must pass tenantId per the convention in docs/architecture/job-priority.md.
  • Metric queue_jobs_enqueued_total{queue, priority} is implemented as a Counter rather than the spec's Gauge queue_jobs_by_priority. The Counter pattern matches the existing queue_jobs_completed_total shape, doesn't require periodic sampling, and is what producers naturally emit. A separate Gauge sampler for the priority distribution of waiting jobs can be added later if needed.
  • Tenant-plan classifier is intentionally not hard-coded. The boilerplate has no enterprise|pro|free field on tenants — operators wire their own subscription→plan-tier function via setSubscriptionToPlanTier. Until that is set, the resolver returns NORMAL for every tenant, which the spec explicitly accepts as v1.

Progress Log

  • 2026-05-17 — e5994239 — priority resolver helper + 20 unit tests (packages/queue/src/priority.ts, priority.test.ts, index.ts, types.ts).
  • 2026-05-17 — 39ec090b — enqueueJob accepts tenantId + priorityOverride + numeric priority, plus optional EnqueueObserver so apps can forward enqueue events without coupling the queue package to prom-client. 10 new producer-test cases.
  • 2026-05-17 — d26752b3 — queue_jobs_enqueued_total{queue, priority} Counter added to the metrics package; api/web/worker registries register the observer during boot.
  • 2026-05-17 — bf806d46 — NotificationService.send passes targetTenantId to enqueueJob; regression test added.
  • 2026-05-17 — 433997bd — Mail producer (packages/mail/src/queue.ts) accepts tenantId + priorityOverride on SendMailInput. queueTransactionalEmail / sendEmail forward both. Auth-flow mail (password-reset, magic-link, email-verification, user-invitation) pinned to priorityOverride: 'critical'. Affected unit tests updated.
  • 2026-05-17 — c9e8ce38 — queueLegacyEventWebhookAttemptDelivery passes attempt.tenantId; test updated.
  • 2026-05-17 — 75715de4 — Imports submit route passes dryRunJob.tenantId.
  • 2026-05-17 — 2a177942 — enqueueFileScan passes input.tenantId.
  • 2026-05-17 — 4dc1b784 — queueExportArtifactRetention pinned to CRITICAL via tenantId: null; test asserts priority=1.
  • 2026-05-17 — 2596c746 — tenantPlanLookup.server.ts wires the subscription-cache as the queue resolver's plan lookup; registered during cache bootstrap. 5 new unit tests.
  • 2026-05-17 — dfeb4204 — docs/architecture/job-priority.md covers resolver order, plan-tier mapping, producer guidelines, feature-flag, and Grafana queries.

Lessons Learned

(Optional am Task-Ende.)