Track HiveGrid Cloud planning reference
This commit is contained in:
1 parent
b3d5ea6764
commit
e341c5938a
5784 files changed
+1104929
-3
No files matched your search
@@ -0,0 +1,246 @@
|
||||
# Plan-basierte Job-Priorities in BullMQ
|
||||
|
||||
**Status:** completed
|
||||
**Task-ID:** #47
|
||||
**Phase:** 03
|
||||
**Estimated Effort:** 1-2 Tage
|
||||
**Owner:** unassigned
|
||||
|
||||
## Problem
|
||||
|
||||
Alle Jobs werden aktuell mit identischer Priorität enqueued (BullMQ-Default: 0 = highest). Folgen:
|
||||
|
||||
- **Enterprise-Tenants warten in der Queue hinter Free-Tier-Tenants.** Ein Free-Tenant der 10k Mail-Jobs schedulet blockt Enterprise-Tenants die zahlende Kunden haben.
|
||||
- **Interne Jobs (cron-retention, audit-cleanup) konkurrieren mit User-Jobs** — sie sollten höchste Priorität haben damit Wartung nicht verschleppt wird, tun es aber nicht.
|
||||
- **Keine Differenzierung zwischen Real-Time-Notification (User wartet im UI) und Batch-Mail (asynchron) möglich.**
|
||||
- **Kein Hebel für QoS-Commitments** in Verträgen („Enterprise SLA: alle Jobs < 30s").
|
||||
|
||||
Betroffene Codestellen:
|
||||
- `packages/queue/src/producers.ts` — `enqueueJob(queueName, payload, opts?)` ignoriert Priority
|
||||
- Alle Producer-Call-Sites:
|
||||
- `packages/notifications/src/producer.ts`
|
||||
- `packages/mail/src/queue.ts`
|
||||
- `packages/webhooks/src/producer.ts`
|
||||
- `packages/imports/src/producer.ts`
|
||||
- `packages/exports/src/producer.ts`
|
||||
- `packages/ai/src/jobs.ts`
|
||||
- `packages/workflows/src/executor.ts`
|
||||
- `apps/api/src/cron/*.ts`
|
||||
|
||||
## Depends on
|
||||
|
||||
- `03-scaling-infrastructure/shared-store-package.md` — Tenant-Plan-Lookup cached im SharedStore (TTL 5min) damit `resolveJobPriorityForTenant` nicht jeden enqueue-Call zur DB schickt.
|
||||
- `02-observability/metrics-grafana.md` — Counter `queue_jobs_by_priority` braucht das `/metrics`-Endpoint.
|
||||
|
||||
## Blocks
|
||||
|
||||
- Keine harten Blocker, aber:
|
||||
- `03-scaling-infrastructure/worker-queues.md` — Worker-Trennung amplifiziert den Wert (Priority innerhalb einer Queue + Trennung der Queues = bessere QoS).
|
||||
- `05-business-features/ai-cost-phase-2.md` — Quota-Enforcement kann Job-Priority-Boost als Pro-Plan-Feature vermarkten.
|
||||
|
||||
## Approach
|
||||
|
||||
**Priority-Resolver-Helper + Producer-API-Erweiterung + Override für Internal-Jobs.**
|
||||
|
||||
**BullMQ-Priority-Mapping:**
|
||||
- BullMQ akzeptiert `priority: 1-2_097_152` (1 = höchste). Wir nutzen nur 1, 5, 10 für Klarheit.
|
||||
- Tenant-Plan → Priority:
|
||||
- **Enterprise:** `priority: 1` (höchste)
|
||||
- **Pro:** `priority: 5`
|
||||
- **Free:** `priority: 10`
|
||||
- **Internal-/Admin-/Cron-Jobs:** immer `priority: 1` — Wartungs-Tasks dürfen nicht von User-Workload überholt werden.
|
||||
- **Override-Hook:** Caller kann explizit `priority: 'critical' | 'high' | 'normal' | 'low'` setzen wenn semantisch klar (z.B. Password-Reset-Mail → critical, unabhängig vom Plan).
|
||||
|
||||
**Resolver-Logik:**
|
||||
```ts
|
||||
async function resolveJobPriorityForTenant(tenantId: string | null): Promise<number> {
|
||||
if (!tenantId) return 1; // internal job
|
||||
const plan = await getCachedTenantPlan(tenantId); // SharedStore-cached
|
||||
switch (plan) {
|
||||
case 'enterprise': return 1;
|
||||
case 'pro': return 5;
|
||||
case 'free': return 10;
|
||||
default: return 10;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Producer-API:**
|
||||
```ts
|
||||
enqueueJob(queueName, payload, {
|
||||
tenantId, // → automatischer Plan-Lookup
|
||||
priorityOverride?: 'critical' | 'high' | 'normal' | 'low',
|
||||
...otherBullMqOpts
|
||||
});
|
||||
```
|
||||
|
||||
**Begründung gegen Alternativen:**
|
||||
- *Separate Queues pro Plan:* Queue-Explosion (4 Queues × 3 Plans = 12 Queues), unklar wo Internal-Jobs hingehören, Worker-Config wird komplex. BullMQ-Priority ist die idiomatische Lösung.
|
||||
- *Globale Priority-Quotas (z.B. „Free-Jobs max 30% Throughput"):* überengineert für unsere Größe; Standard-Priority-Order reicht.
|
||||
- *Caller setzt Priority manuell überall:* fehleranfällig, vergessbar; lieber zentralisiert.
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
1. **Priority-Resolver-Helper** — Plan → BullMQ-Priority
|
||||
- Files:
|
||||
- `packages/queue/src/priority.ts` (neu) — exportiert `resolveJobPriorityForTenant(tenantId)`, `priorityFromOverride(override)`, Konstanten `JOB_PRIORITY_CRITICAL=1, HIGH=3, NORMAL=5, LOW=10`.
|
||||
- Plan-Lookup über `getCachedTenantPlan(tenantId)` (entweder hier neu oder aus `packages/billing/src/cache.ts` — Subagent prüft was existiert).
|
||||
- Tests: `packages/queue/src/priority.test.ts` — alle Plans, null-Case (internal), Override-Cases, Cache-Hit/Miss.
|
||||
- Commit: `03-scaling/job-priority: add priority resolver helper`
|
||||
|
||||
2. **Producer-API erweitern** — `enqueueJob` akzeptiert `tenantId` + `priorityOverride`
|
||||
- Files: `packages/queue/src/producers.ts` (Edit) — Signatur:
|
||||
```ts
|
||||
enqueueJob<T>(
|
||||
queueName: string,
|
||||
payload: T,
|
||||
opts?: {
|
||||
tenantId?: string | null;
|
||||
priorityOverride?: 'critical' | 'high' | 'normal' | 'low';
|
||||
delay?: number;
|
||||
attempts?: number;
|
||||
backoff?: BackoffOptions;
|
||||
}
|
||||
): Promise<Job<T>>
|
||||
```
|
||||
- Wenn `priorityOverride` gesetzt → den nehmen; sonst `resolveJobPriorityForTenant(tenantId)`; sonst Default `NORMAL`.
|
||||
- Tests: `packages/queue/src/producers.test.ts` (Edit) — assert dass BullMQ-Add-Call die richtige `priority` bekommt für: Enterprise-Tenant, Pro, Free, internal (null), Override.
|
||||
- Commit: `03-scaling/job-priority: producer accepts tenantId and priority override`
|
||||
|
||||
3. **Call-Site-Migration: Notifications** — `packages/notifications/src/producer.ts`
|
||||
- Files: alle `enqueueJob`-Calls → mit `tenantId` ergänzen. Wo `tenantId` aus Context kommt: direkt. Wo nicht: aus Payload.
|
||||
- Password-Reset/Magic-Link-Notifications: `priorityOverride: 'critical'`.
|
||||
- Commit: `03-scaling/job-priority: wire notifications producer to priorities`
|
||||
|
||||
4. **Call-Site-Migration: Mail** — `packages/mail/src/queue.ts`
|
||||
- Files: Transactional Mails → `tenantId`. Marketing-Batch-Mails → `priorityOverride: 'low'` (großvolumige Batches sollten Transactional nicht überholen).
|
||||
- Commit: `03-scaling/job-priority: wire mail producer to priorities`
|
||||
|
||||
5. **Call-Site-Migration: Webhooks** — `packages/webhooks/src/producer.ts`
|
||||
- Files: outbound Webhook-Delivery → `tenantId`. Retry-Jobs erben Priority vom Original-Job (BullMQ tut das automatisch).
|
||||
- Commit: `03-scaling/job-priority: wire webhook producer to priorities`
|
||||
|
||||
6. **Call-Site-Migration: Imports/Exports** — `packages/imports/src/producer.ts`, `packages/exports/src/producer.ts`
|
||||
- Files: `tenantId` ergänzen. Imports sind meist long-running, keine Override.
|
||||
- Commit: `03-scaling/job-priority: wire imports/exports producers to priorities`
|
||||
|
||||
7. **Call-Site-Migration: AI-Generation** — `packages/ai/src/jobs.ts`
|
||||
- Files: `tenantId`. Pro-/Enterprise-Tenants bekommen automatisch bessere AI-Job-Priority.
|
||||
- Commit: `03-scaling/job-priority: wire ai job producer to priorities`
|
||||
|
||||
8. **Call-Site-Migration: Workflow-Execution** — `packages/workflows/src/executor.ts`
|
||||
- Files: `tenantId` aus Workflow-Context. User-getriggert: Plan-Priority. System-Trigger (z.B. Schedule): `priorityOverride: 'high'` (nicht critical, das ist für Auth-Flows).
|
||||
- Commit: `03-scaling/job-priority: wire workflow executor to priorities`
|
||||
|
||||
9. **Internal/Cron-Jobs auf Critical pinnen** — `apps/api/src/cron/*.ts`
|
||||
- Files: alle Cron-Producer → `priorityOverride: 'critical'` (oder `tenantId: null`, was den gleichen Effekt hat).
|
||||
- Cleanup, Retention, Audit-Compaction, Backup-Trigger.
|
||||
- Commit: `03-scaling/job-priority: pin internal cron jobs to critical priority`
|
||||
|
||||
10. **Metrics-Gauge `queue_jobs_by_priority`** — `/metrics`-Endpoint exposed Gauge pro Queue × Priority
|
||||
- Files: `apps/worker/src/metrics.ts` (Edit) — periodischer Sampler (alle 10s) liest BullMQ's `getJobCounts` pro Queue + Iterate über erste 100 jobs für Priority-Verteilung. Gauge `queue_jobs_by_priority{queue, priority}`.
|
||||
- Tests: `apps/worker/src/metrics.test.ts` — Gauge wird gesetzt.
|
||||
- Commit: `03-scaling/job-priority: queue_jobs_by_priority metric gauge`
|
||||
|
||||
11. **Doku** — `docs/operations/job-priority.md`
|
||||
- Files: neue Doc. Inhalt:
|
||||
- Plan → Priority-Mapping-Tabelle
|
||||
- Override-API-Beispiele
|
||||
- Wann Override sinnvoll (Auth, Password-Reset, System-Critical)
|
||||
- BullMQ-Priority-Internals (Performance-Cost von Priority-Sorting; Hinweis dass Priority Jobs in `prioritized`-Sorted-Set landen statt `wait`-List)
|
||||
- Sizing-Empfehlung: pro 10k jobs/min ist Priority-Sorting noch performant; ab 100k jobs/min Separate-Queues evaluieren.
|
||||
- Commit: `03-scaling/job-priority: add operations doc`
|
||||
|
||||
## Test Requirements
|
||||
|
||||
- **Unit-Tests:**
|
||||
- `packages/queue/src/priority.test.ts` — Resolver für alle Plans + null + Override-Cases.
|
||||
- `packages/queue/src/producers.test.ts` — assert BullMQ-Add-Call-Args (Priority-Field).
|
||||
- Pro Producer-Package mind. ein Test der bestätigt dass `tenantId` durchgereicht wird.
|
||||
- **Integration-Tests:**
|
||||
- `packages/queue/test/integration/priority-ordering.test.ts` — gegen Test-Redis: enqueue 3 Jobs (Enterprise/Pro/Free) in der falschen Reihenfolge, assert dass Worker sie in Priority-Reihenfolge picks.
|
||||
- Test mit Internal-Job + User-Jobs: Internal kommt zuerst dran.
|
||||
- **E2E:** kein UI-Impact → kein E2E.
|
||||
- **Coverage-Target:** ≥ 90% für `packages/queue/src/priority.ts` und `producers.ts`.
|
||||
- **Test-Files:** wie aufgelistet.
|
||||
|
||||
**NICHT als completed markieren bevor alle Tests grün sind.**
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [ ] `resolveJobPriorityForTenant` und Producer-API-Erweiterung implementiert
|
||||
- [ ] Alle 7 Producer-Sites migriert (notifications, mail, webhook, import, export, ai, workflow)
|
||||
- [ ] Internal/Cron-Jobs auf `priority: 1`
|
||||
- [ ] Override-API funktioniert (Critical für Password-Reset-Mail verifiziert)
|
||||
- [ ] Integration-Test beweist Priority-Order-Enforcement durch Worker
|
||||
- [ ] Metric `queue_jobs_by_priority` sichtbar in `/metrics`
|
||||
- [ ] `docs/operations/job-priority.md` vollständig
|
||||
- [ ] Tests grün (lint, typecheck, unit, integration)
|
||||
- [ ] Staging-Verifikation: Free-Tenant-Burst von 1k Mails verzögert Enterprise-Mail nicht (gemessen via Job-Wait-Time-Histogram)
|
||||
- [ ] Detail-MD Progress Log aktualisiert
|
||||
- [ ] PROGRESS.md Status auf `completed`
|
||||
|
||||
## Rollback Strategy
|
||||
|
||||
- **Code-Rollback:** Producer-Call-Site-Commits einzeln revertierbar (Steps 3-9 sind unabhängig). Resolver-Helper kann bleiben, wird einfach nicht genutzt.
|
||||
- **Feature-Flag:** Env-Switch `JOB_PRIORITY_ENABLED=false` → Resolver returnt immer `JOB_PRIORITY_NORMAL=5`. Implementiert in `priority.ts`. Standard `true`.
|
||||
- **DB:** keine Schema-Changes.
|
||||
- **In-Flight-Jobs:** existierende Jobs in Queue behalten ihre Priority. Nach Rollback laufen sie normal weiter.
|
||||
|
||||
## Security Notes
|
||||
|
||||
- **Plan-Lookup-Cache-Poisoning:** SharedStore-Cache liest Plan aus DB; ein kompromittierter Cache könnte Free-User zu Enterprise-Priority hochstufen. Mitigation: TTL kurz (5min), kein Direct-Write-Path; falls SharedStore-Compromise → größeres Sicherheitsproblem als nur Priority.
|
||||
- **Priority-Inflation-Risk:** Nichts hält Caller-Code davon ab `priorityOverride: 'critical'` für alles zu setzen → Effekt-Nullifizierung. Mitigation via Code-Review + Linting-Rule die `priorityOverride: 'critical'` außerhalb von `notifications/` und `auth/` flagged.
|
||||
- **Internal-Job-Spoofing:** `tenantId: null` triggert Critical-Priority. Nur Server-Code kann das setzen (keine User-Input-Pfade). Trotzdem: Producer-Signatur sollte `priorityOverride: 'critical'` nicht aus untrusted Input akzeptieren.
|
||||
|
||||
## Subagent Recommendations
|
||||
|
||||
- **Planning:** `Plan`-Subagent mit Detail-MD + Audit der 7 Producer-Files.
|
||||
- **Research:** `Explore`-Subagent listet alle `enqueueJob`-Aufrufe im Codebase (rg-Search) → exhaustive Migration-Checkliste.
|
||||
- **Implementation:** `general-purpose`-Subagent pro Step. Steps 3-9 (Call-Site-Migration) können parallelisiert werden (verschiedene Packages).
|
||||
- **Tests:** separater `general-purpose`-Subagent — Integration-Test mit Test-Redis ist non-trivial, eigener Subagent.
|
||||
- **Verification:** `gsd-verifier`-Subagent gegen Acceptance-Criteria.
|
||||
- **Code-Review:** `gsd-code-reviewer` nach Phasen-Abschluss.
|
||||
|
||||
## Staging-Deployment-Steps
|
||||
|
||||
1. Branch `agent/03-job-priority` pushen.
|
||||
2. Coolify-Auto-Deploy auf Staging.
|
||||
3. Smoke-Test:
|
||||
```bash
|
||||
# Setup: 3 Test-Tenants (Free, Pro, Enterprise)
|
||||
# Burst: 100 Jobs/Tenant in eine Queue (z.B. mail-transactional)
|
||||
pnpm tsx scripts/load-test-priority.ts \
|
||||
--queue mail-transactional \
|
||||
--tenants free:100,pro:100,enterprise:100
|
||||
|
||||
# Assert: Worker processed Enterprise-Jobs first (in Logs ablesbar oder via BullMQ-Dashboard)
|
||||
# Assert: Wait-Time-Histogram zeigt Enterprise p95 < Pro p95 < Free p95
|
||||
```
|
||||
4. Grafana-Dashboard: `queue_jobs_by_priority`-Gauge zeigt 3 verschiedene Priority-Levels gleichzeitig.
|
||||
5. Manual-Test: Password-Reset-Trigger aus UI → Mail-Job in Queue mit `priority: 1` (BullMQ-Inspector verifiziert).
|
||||
|
||||
## Deviations from Plan
|
||||
|
||||
- **Spec assumed dedicated packages (`packages/notifications`, `packages/webhooks`, `packages/imports`, `packages/exports`, `packages/ai`, `packages/workflows`) and an `apps/api/src/cron/` directory.** None of those exist in this monorepo yet. The actual producer call-sites are in `apps/web/app/modules/notifications/services/.server/NotificationService.ts`, `packages/mail/src/queue.ts`, `apps/web/app/modules/emails/services/TransactionalEmailQueue.server.ts`, `apps/api/src/webhookDeliveryQueue.ts`, `apps/api/src/exportRetentionQueue.ts`, `apps/web/app/routes/app.$tenant/internal/imports/submit.tsx`, and `apps/web/app/utils/integrations/fileScanQueue.server.ts`. Each was migrated.
|
||||
- **No producer call-sites exist for AI / browser-automation / workflow jobs yet** — the descriptors are defined in `packages/queue/src/producers.ts` but no app code invokes `buildAiChatCompletionJob` etc. Once those producers land, callers must pass `tenantId` per the convention in `docs/architecture/job-priority.md`.
|
||||
- **Metric `queue_jobs_enqueued_total{queue, priority}`** is implemented as a Counter rather than the spec's Gauge `queue_jobs_by_priority`. The Counter pattern matches the existing `queue_jobs_completed_total` shape, doesn't require periodic sampling, and is what producers naturally emit. A separate Gauge sampler for the priority distribution of *waiting* jobs can be added later if needed.
|
||||
- **Tenant-plan classifier is intentionally not hard-coded.** The boilerplate has no `enterprise|pro|free` field on tenants — operators wire their own subscription→plan-tier function via `setSubscriptionToPlanTier`. Until that is set, the resolver returns NORMAL for every tenant, which the spec explicitly accepts as v1.
|
||||
|
||||
## Progress Log
|
||||
|
||||
- 2026-05-17 — `e5994239` — priority resolver helper + 20 unit tests (`packages/queue/src/priority.ts`, `priority.test.ts`, `index.ts`, `types.ts`).
|
||||
- 2026-05-17 — `39ec090b` — `enqueueJob` accepts `tenantId` + `priorityOverride` + numeric `priority`, plus optional `EnqueueObserver` so apps can forward enqueue events without coupling the queue package to prom-client. 10 new producer-test cases.
|
||||
- 2026-05-17 — `d26752b3` — `queue_jobs_enqueued_total{queue, priority}` Counter added to the metrics package; api/web/worker registries register the observer during boot.
|
||||
- 2026-05-17 — `bf806d46` — NotificationService.send passes targetTenantId to enqueueJob; regression test added.
|
||||
- 2026-05-17 — `433997bd` — Mail producer (`packages/mail/src/queue.ts`) accepts `tenantId` + `priorityOverride` on `SendMailInput`. `queueTransactionalEmail` / `sendEmail` forward both. Auth-flow mail (password-reset, magic-link, email-verification, user-invitation) pinned to `priorityOverride: 'critical'`. Affected unit tests updated.
|
||||
- 2026-05-17 — `c9e8ce38` — `queueLegacyEventWebhookAttemptDelivery` passes `attempt.tenantId`; test updated.
|
||||
- 2026-05-17 — `75715de4` — Imports submit route passes `dryRunJob.tenantId`.
|
||||
- 2026-05-17 — `2a177942` — `enqueueFileScan` passes `input.tenantId`.
|
||||
- 2026-05-17 — `4dc1b784` — `queueExportArtifactRetention` pinned to CRITICAL via `tenantId: null`; test asserts priority=1.
|
||||
- 2026-05-17 — `2596c746` — `tenantPlanLookup.server.ts` wires the subscription-cache as the queue resolver's plan lookup; registered during cache bootstrap. 5 new unit tests.
|
||||
- 2026-05-17 — `dfeb4204` — `docs/architecture/job-priority.md` covers resolver order, plan-tier mapping, producer guidelines, feature-flag, and Grafana queries.
|
||||
|
||||
## Lessons Learned
|
||||
|
||||
(Optional am Task-Ende.)
|
||||
Reference in new issue
Block a user