Files
HiveGrid-Wireframes/hivegrid-cloud/docs/improvement-plan/03-scaling-infrastructure/job-priority.md
T

247 lines
16 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Plan-basierte Job-Priorities in BullMQ
**Status:** completed
**Task-ID:** #47
**Phase:** 03
**Estimated Effort:** 1-2 Tage
**Owner:** unassigned
## Problem
Alle Jobs werden aktuell mit identischer Priorität enqueued (BullMQ-Default: 0 = highest). Folgen:
- **Enterprise-Tenants warten in der Queue hinter Free-Tier-Tenants.** Ein Free-Tenant der 10k Mail-Jobs schedulet blockt Enterprise-Tenants die zahlende Kunden haben.
- **Interne Jobs (cron-retention, audit-cleanup) konkurrieren mit User-Jobs** — sie sollten höchste Priorität haben damit Wartung nicht verschleppt wird, tun es aber nicht.
- **Keine Differenzierung zwischen Real-Time-Notification (User wartet im UI) und Batch-Mail (asynchron) möglich.**
- **Kein Hebel für QoS-Commitments** in Verträgen („Enterprise SLA: alle Jobs < 30s").
Betroffene Codestellen:
- `packages/queue/src/producers.ts` — `enqueueJob(queueName, payload, opts?)` ignoriert Priority
- Alle Producer-Call-Sites:
- `packages/notifications/src/producer.ts`
- `packages/mail/src/queue.ts`
- `packages/webhooks/src/producer.ts`
- `packages/imports/src/producer.ts`
- `packages/exports/src/producer.ts`
- `packages/ai/src/jobs.ts`
- `packages/workflows/src/executor.ts`
- `apps/api/src/cron/*.ts`
## Depends on
- `03-scaling-infrastructure/shared-store-package.md` — Tenant-Plan-Lookup cached im SharedStore (TTL 5min) damit `resolveJobPriorityForTenant` nicht jeden enqueue-Call zur DB schickt.
- `02-observability/metrics-grafana.md` — Counter `queue_jobs_by_priority` braucht das `/metrics`-Endpoint.
## Blocks
- Keine harten Blocker, aber:
- `03-scaling-infrastructure/worker-queues.md` — Worker-Trennung amplifiziert den Wert (Priority innerhalb einer Queue + Trennung der Queues = bessere QoS).
- `05-business-features/ai-cost-phase-2.md` — Quota-Enforcement kann Job-Priority-Boost als Pro-Plan-Feature vermarkten.
## Approach
**Priority-Resolver-Helper + Producer-API-Erweiterung + Override für Internal-Jobs.**
**BullMQ-Priority-Mapping:**
- BullMQ akzeptiert `priority: 1-2_097_152` (1 = höchste). Wir nutzen nur 1, 5, 10 für Klarheit.
- Tenant-Plan → Priority:
- **Enterprise:** `priority: 1` (höchste)
- **Pro:** `priority: 5`
- **Free:** `priority: 10`
- **Internal-/Admin-/Cron-Jobs:** immer `priority: 1` — Wartungs-Tasks dürfen nicht von User-Workload überholt werden.
- **Override-Hook:** Caller kann explizit `priority: 'critical' | 'high' | 'normal' | 'low'` setzen wenn semantisch klar (z.B. Password-Reset-Mail → critical, unabhängig vom Plan).
**Resolver-Logik:**
```ts
async function resolveJobPriorityForTenant(tenantId: string | null): Promise<number> {
if (!tenantId) return 1; // internal job
const plan = await getCachedTenantPlan(tenantId); // SharedStore-cached
switch (plan) {
case 'enterprise': return 1;
case 'pro': return 5;
case 'free': return 10;
default: return 10;
}
}
```
**Producer-API:**
```ts
enqueueJob(queueName, payload, {
tenantId, // → automatischer Plan-Lookup
priorityOverride?: 'critical' | 'high' | 'normal' | 'low',
...otherBullMqOpts
});
```
**Begründung gegen Alternativen:**
- *Separate Queues pro Plan:* Queue-Explosion (4 Queues × 3 Plans = 12 Queues), unklar wo Internal-Jobs hingehören, Worker-Config wird komplex. BullMQ-Priority ist die idiomatische Lösung.
- *Globale Priority-Quotas (z.B. „Free-Jobs max 30% Throughput"):* überengineert für unsere Größe; Standard-Priority-Order reicht.
- *Caller setzt Priority manuell überall:* fehleranfällig, vergessbar; lieber zentralisiert.
## Implementation Steps
1. **Priority-Resolver-Helper** — Plan → BullMQ-Priority
- Files:
- `packages/queue/src/priority.ts` (neu) — exportiert `resolveJobPriorityForTenant(tenantId)`, `priorityFromOverride(override)`, Konstanten `JOB_PRIORITY_CRITICAL=1, HIGH=3, NORMAL=5, LOW=10`.
- Plan-Lookup über `getCachedTenantPlan(tenantId)` (entweder hier neu oder aus `packages/billing/src/cache.ts` — Subagent prüft was existiert).
- Tests: `packages/queue/src/priority.test.ts` — alle Plans, null-Case (internal), Override-Cases, Cache-Hit/Miss.
- Commit: `03-scaling/job-priority: add priority resolver helper`
2. **Producer-API erweitern** — `enqueueJob` akzeptiert `tenantId` + `priorityOverride`
- Files: `packages/queue/src/producers.ts` (Edit) — Signatur:
```ts
enqueueJob<T>(
queueName: string,
payload: T,
opts?: {
tenantId?: string | null;
priorityOverride?: 'critical' | 'high' | 'normal' | 'low';
delay?: number;
attempts?: number;
backoff?: BackoffOptions;
}
): Promise<Job<T>>
```
- Wenn `priorityOverride` gesetzt → den nehmen; sonst `resolveJobPriorityForTenant(tenantId)`; sonst Default `NORMAL`.
- Tests: `packages/queue/src/producers.test.ts` (Edit) — assert dass BullMQ-Add-Call die richtige `priority` bekommt für: Enterprise-Tenant, Pro, Free, internal (null), Override.
- Commit: `03-scaling/job-priority: producer accepts tenantId and priority override`
3. **Call-Site-Migration: Notifications** — `packages/notifications/src/producer.ts`
- Files: alle `enqueueJob`-Calls → mit `tenantId` ergänzen. Wo `tenantId` aus Context kommt: direkt. Wo nicht: aus Payload.
- Password-Reset/Magic-Link-Notifications: `priorityOverride: 'critical'`.
- Commit: `03-scaling/job-priority: wire notifications producer to priorities`
4. **Call-Site-Migration: Mail** — `packages/mail/src/queue.ts`
- Files: Transactional Mails → `tenantId`. Marketing-Batch-Mails → `priorityOverride: 'low'` (großvolumige Batches sollten Transactional nicht überholen).
- Commit: `03-scaling/job-priority: wire mail producer to priorities`
5. **Call-Site-Migration: Webhooks** — `packages/webhooks/src/producer.ts`
- Files: outbound Webhook-Delivery → `tenantId`. Retry-Jobs erben Priority vom Original-Job (BullMQ tut das automatisch).
- Commit: `03-scaling/job-priority: wire webhook producer to priorities`
6. **Call-Site-Migration: Imports/Exports** — `packages/imports/src/producer.ts`, `packages/exports/src/producer.ts`
- Files: `tenantId` ergänzen. Imports sind meist long-running, keine Override.
- Commit: `03-scaling/job-priority: wire imports/exports producers to priorities`
7. **Call-Site-Migration: AI-Generation** — `packages/ai/src/jobs.ts`
- Files: `tenantId`. Pro-/Enterprise-Tenants bekommen automatisch bessere AI-Job-Priority.
- Commit: `03-scaling/job-priority: wire ai job producer to priorities`
8. **Call-Site-Migration: Workflow-Execution** — `packages/workflows/src/executor.ts`
- Files: `tenantId` aus Workflow-Context. User-getriggert: Plan-Priority. System-Trigger (z.B. Schedule): `priorityOverride: 'high'` (nicht critical, das ist für Auth-Flows).
- Commit: `03-scaling/job-priority: wire workflow executor to priorities`
9. **Internal/Cron-Jobs auf Critical pinnen** — `apps/api/src/cron/*.ts`
- Files: alle Cron-Producer → `priorityOverride: 'critical'` (oder `tenantId: null`, was den gleichen Effekt hat).
- Cleanup, Retention, Audit-Compaction, Backup-Trigger.
- Commit: `03-scaling/job-priority: pin internal cron jobs to critical priority`
10. **Metrics-Gauge `queue_jobs_by_priority`** — `/metrics`-Endpoint exposed Gauge pro Queue × Priority
- Files: `apps/worker/src/metrics.ts` (Edit) — periodischer Sampler (alle 10s) liest BullMQ's `getJobCounts` pro Queue + Iterate über erste 100 jobs für Priority-Verteilung. Gauge `queue_jobs_by_priority{queue, priority}`.
- Tests: `apps/worker/src/metrics.test.ts` — Gauge wird gesetzt.
- Commit: `03-scaling/job-priority: queue_jobs_by_priority metric gauge`
11. **Doku** — `docs/operations/job-priority.md`
- Files: neue Doc. Inhalt:
- Plan → Priority-Mapping-Tabelle
- Override-API-Beispiele
- Wann Override sinnvoll (Auth, Password-Reset, System-Critical)
- BullMQ-Priority-Internals (Performance-Cost von Priority-Sorting; Hinweis dass Priority Jobs in `prioritized`-Sorted-Set landen statt `wait`-List)
- Sizing-Empfehlung: pro 10k jobs/min ist Priority-Sorting noch performant; ab 100k jobs/min Separate-Queues evaluieren.
- Commit: `03-scaling/job-priority: add operations doc`
## Test Requirements
- **Unit-Tests:**
- `packages/queue/src/priority.test.ts` — Resolver für alle Plans + null + Override-Cases.
- `packages/queue/src/producers.test.ts` — assert BullMQ-Add-Call-Args (Priority-Field).
- Pro Producer-Package mind. ein Test der bestätigt dass `tenantId` durchgereicht wird.
- **Integration-Tests:**
- `packages/queue/test/integration/priority-ordering.test.ts` — gegen Test-Redis: enqueue 3 Jobs (Enterprise/Pro/Free) in der falschen Reihenfolge, assert dass Worker sie in Priority-Reihenfolge picks.
- Test mit Internal-Job + User-Jobs: Internal kommt zuerst dran.
- **E2E:** kein UI-Impact → kein E2E.
- **Coverage-Target:** ≥ 90% für `packages/queue/src/priority.ts` und `producers.ts`.
- **Test-Files:** wie aufgelistet.
**NICHT als completed markieren bevor alle Tests grün sind.**
## Acceptance Criteria
- [ ] `resolveJobPriorityForTenant` und Producer-API-Erweiterung implementiert
- [ ] Alle 7 Producer-Sites migriert (notifications, mail, webhook, import, export, ai, workflow)
- [ ] Internal/Cron-Jobs auf `priority: 1`
- [ ] Override-API funktioniert (Critical für Password-Reset-Mail verifiziert)
- [ ] Integration-Test beweist Priority-Order-Enforcement durch Worker
- [ ] Metric `queue_jobs_by_priority` sichtbar in `/metrics`
- [ ] `docs/operations/job-priority.md` vollständig
- [ ] Tests grün (lint, typecheck, unit, integration)
- [ ] Staging-Verifikation: Free-Tenant-Burst von 1k Mails verzögert Enterprise-Mail nicht (gemessen via Job-Wait-Time-Histogram)
- [ ] Detail-MD Progress Log aktualisiert
- [ ] PROGRESS.md Status auf `completed`
## Rollback Strategy
- **Code-Rollback:** Producer-Call-Site-Commits einzeln revertierbar (Steps 3-9 sind unabhängig). Resolver-Helper kann bleiben, wird einfach nicht genutzt.
- **Feature-Flag:** Env-Switch `JOB_PRIORITY_ENABLED=false` → Resolver returnt immer `JOB_PRIORITY_NORMAL=5`. Implementiert in `priority.ts`. Standard `true`.
- **DB:** keine Schema-Changes.
- **In-Flight-Jobs:** existierende Jobs in Queue behalten ihre Priority. Nach Rollback laufen sie normal weiter.
## Security Notes
- **Plan-Lookup-Cache-Poisoning:** SharedStore-Cache liest Plan aus DB; ein kompromittierter Cache könnte Free-User zu Enterprise-Priority hochstufen. Mitigation: TTL kurz (5min), kein Direct-Write-Path; falls SharedStore-Compromise → größeres Sicherheitsproblem als nur Priority.
- **Priority-Inflation-Risk:** Nichts hält Caller-Code davon ab `priorityOverride: 'critical'` für alles zu setzen → Effekt-Nullifizierung. Mitigation via Code-Review + Linting-Rule die `priorityOverride: 'critical'` außerhalb von `notifications/` und `auth/` flagged.
- **Internal-Job-Spoofing:** `tenantId: null` triggert Critical-Priority. Nur Server-Code kann das setzen (keine User-Input-Pfade). Trotzdem: Producer-Signatur sollte `priorityOverride: 'critical'` nicht aus untrusted Input akzeptieren.
## Subagent Recommendations
- **Planning:** `Plan`-Subagent mit Detail-MD + Audit der 7 Producer-Files.
- **Research:** `Explore`-Subagent listet alle `enqueueJob`-Aufrufe im Codebase (rg-Search) → exhaustive Migration-Checkliste.
- **Implementation:** `general-purpose`-Subagent pro Step. Steps 3-9 (Call-Site-Migration) können parallelisiert werden (verschiedene Packages).
- **Tests:** separater `general-purpose`-Subagent — Integration-Test mit Test-Redis ist non-trivial, eigener Subagent.
- **Verification:** `gsd-verifier`-Subagent gegen Acceptance-Criteria.
- **Code-Review:** `gsd-code-reviewer` nach Phasen-Abschluss.
## Staging-Deployment-Steps
1. Branch `agent/03-job-priority` pushen.
2. Coolify-Auto-Deploy auf Staging.
3. Smoke-Test:
```bash
# Setup: 3 Test-Tenants (Free, Pro, Enterprise)
# Burst: 100 Jobs/Tenant in eine Queue (z.B. mail-transactional)
pnpm tsx scripts/load-test-priority.ts \
--queue mail-transactional \
--tenants free:100,pro:100,enterprise:100
# Assert: Worker processed Enterprise-Jobs first (in Logs ablesbar oder via BullMQ-Dashboard)
# Assert: Wait-Time-Histogram zeigt Enterprise p95 < Pro p95 < Free p95
```
4. Grafana-Dashboard: `queue_jobs_by_priority`-Gauge zeigt 3 verschiedene Priority-Levels gleichzeitig.
5. Manual-Test: Password-Reset-Trigger aus UI → Mail-Job in Queue mit `priority: 1` (BullMQ-Inspector verifiziert).
## Deviations from Plan
- **Spec assumed dedicated packages (`packages/notifications`, `packages/webhooks`, `packages/imports`, `packages/exports`, `packages/ai`, `packages/workflows`) and an `apps/api/src/cron/` directory.** None of those exist in this monorepo yet. The actual producer call-sites are in `apps/web/app/modules/notifications/services/.server/NotificationService.ts`, `packages/mail/src/queue.ts`, `apps/web/app/modules/emails/services/TransactionalEmailQueue.server.ts`, `apps/api/src/webhookDeliveryQueue.ts`, `apps/api/src/exportRetentionQueue.ts`, `apps/web/app/routes/app.$tenant/internal/imports/submit.tsx`, and `apps/web/app/utils/integrations/fileScanQueue.server.ts`. Each was migrated.
- **No producer call-sites exist for AI / browser-automation / workflow jobs yet** — the descriptors are defined in `packages/queue/src/producers.ts` but no app code invokes `buildAiChatCompletionJob` etc. Once those producers land, callers must pass `tenantId` per the convention in `docs/architecture/job-priority.md`.
- **Metric `queue_jobs_enqueued_total{queue, priority}`** is implemented as a Counter rather than the spec's Gauge `queue_jobs_by_priority`. The Counter pattern matches the existing `queue_jobs_completed_total` shape, doesn't require periodic sampling, and is what producers naturally emit. A separate Gauge sampler for the priority distribution of *waiting* jobs can be added later if needed.
- **Tenant-plan classifier is intentionally not hard-coded.** The boilerplate has no `enterprise|pro|free` field on tenants — operators wire their own subscription→plan-tier function via `setSubscriptionToPlanTier`. Until that is set, the resolver returns NORMAL for every tenant, which the spec explicitly accepts as v1.
## Progress Log
- 2026-05-17 — `e5994239` — priority resolver helper + 20 unit tests (`packages/queue/src/priority.ts`, `priority.test.ts`, `index.ts`, `types.ts`).
- 2026-05-17 — `39ec090b` — `enqueueJob` accepts `tenantId` + `priorityOverride` + numeric `priority`, plus optional `EnqueueObserver` so apps can forward enqueue events without coupling the queue package to prom-client. 10 new producer-test cases.
- 2026-05-17 — `d26752b3` — `queue_jobs_enqueued_total{queue, priority}` Counter added to the metrics package; api/web/worker registries register the observer during boot.
- 2026-05-17 — `bf806d46` — NotificationService.send passes targetTenantId to enqueueJob; regression test added.
- 2026-05-17 — `433997bd` — Mail producer (`packages/mail/src/queue.ts`) accepts `tenantId` + `priorityOverride` on `SendMailInput`. `queueTransactionalEmail` / `sendEmail` forward both. Auth-flow mail (password-reset, magic-link, email-verification, user-invitation) pinned to `priorityOverride: 'critical'`. Affected unit tests updated.
- 2026-05-17 — `c9e8ce38` — `queueLegacyEventWebhookAttemptDelivery` passes `attempt.tenantId`; test updated.
- 2026-05-17 — `75715de4` — Imports submit route passes `dryRunJob.tenantId`.
- 2026-05-17 — `2a177942` — `enqueueFileScan` passes `input.tenantId`.
- 2026-05-17 — `4dc1b784` — `queueExportArtifactRetention` pinned to CRITICAL via `tenantId: null`; test asserts priority=1.
- 2026-05-17 — `2596c746` — `tenantPlanLookup.server.ts` wires the subscription-cache as the queue resolver's plan lookup; registered during cache bootstrap. 5 new unit tests.
- 2026-05-17 — `dfeb4204` — `docs/architecture/job-priority.md` covers resolver order, plan-tier mapping, producer guidelines, feature-flag, and Grafana queries.
## Lessons Learned
(Optional am Task-Ende.)