122 lines
4.5 KiB
Markdown
122 lines
4.5 KiB
Markdown
# Job Priority
|
|
|
|
Plan-based BullMQ job priority. Lets enterprise tenants jump the queue
|
|
over free-tier batches and pins internal/system jobs to the top.
|
|
|
|
## Why
|
|
|
|
Before this feature every BullMQ job ran at the default priority. A
|
|
free-tier tenant scheduling 10 000 mail jobs would block an enterprise
|
|
tenant's password-reset behind them. Cron and retention jobs competed
|
|
with user workload. There was no way to express "this auth mail must
|
|
run now" semantically.
|
|
|
|
## Priority levels
|
|
|
|
| Constant | Value | Use case |
|
|
| ------------------------ | ----- | --------------------------------------------------------- |
|
|
| `JOB_PRIORITY_CRITICAL` | `1` | Internal jobs, auth flows (password-reset, magic-link) |
|
|
| `JOB_PRIORITY_HIGH` | `3` | System-triggered workflows that aren't auth |
|
|
| `JOB_PRIORITY_NORMAL` | `5` | Default for tenant-scoped jobs without a plan lookup |
|
|
| `JOB_PRIORITY_LOW` | `10` | Marketing / bulk batches; free-tier when a lookup is set |
|
|
|
|
BullMQ treats `priority: 1` as highest. We intentionally use only 4
|
|
distinct levels — Grafana dashboards stay legible and the priority
|
|
sorted-set in Redis stays small.
|
|
|
|
## Resolution order (in `enqueueJob`)
|
|
|
|
```
|
|
1. options.priority (explicit numeric) → wins everything
|
|
2. options.priorityOverride ('critical'…) → mapped via priorityFromOverride
|
|
3. options.tenantId === null → CRITICAL (internal job)
|
|
4. options.tenantId set, lookup registered → resolveJobPriorityForTenant
|
|
5. options.tenantId set, no lookup → NORMAL
|
|
6. options absent → no priority field set
|
|
```
|
|
|
|
## Plan → priority mapping
|
|
|
|
| Plan slug | Priority |
|
|
| ------------ | ------------------ |
|
|
| `enterprise` | CRITICAL (1) |
|
|
| `pro` | NORMAL (5) |
|
|
| `free` | LOW (10) |
|
|
| unknown/null | NORMAL (5) |
|
|
|
|
The mapping lives in `packages/queue/src/priority.ts` in
|
|
`priorityFromPlan(plan)`.
|
|
|
|
## Producer guidelines
|
|
|
|
1. **Always pass `tenantId`** for tenant-scoped jobs. The descriptor
|
|
already carries the tenantId in most cases; pass the same value to
|
|
the enqueue options:
|
|
```ts
|
|
await enqueueJob(queue, descriptor, { tenantId });
|
|
```
|
|
2. **Pass `tenantId: null` for internal jobs** — retention, cleanup,
|
|
audit-compaction, backup triggers. They get CRITICAL automatically.
|
|
3. **Use `priorityOverride: 'critical'` for auth flows** regardless of
|
|
plan: password-reset, email-verification, magic-link, user
|
|
invitation. A free-tier user must still receive their reset mail
|
|
instantly.
|
|
4. **Use `priorityOverride: 'low'` for marketing batches.** Don't let
|
|
newsletter batches starve transactional mail.
|
|
5. **Never set `priorityOverride: 'critical'` from untrusted input.**
|
|
The override hook is server-only.
|
|
|
|
## Registering a tenant-plan lookup
|
|
|
|
The queue package keeps its plan-lookup pluggable so workers don't
|
|
have to import apps/web. Web boots the lookup during cache bootstrap:
|
|
|
|
```ts
|
|
// apps/web/app/utils/services/.server/tenantPlanLookup.server.ts
|
|
setSubscriptionToPlanTier((subscription) => {
|
|
// your classifier here — e.g. map stripe product ids to plan slugs
|
|
return subscription?.products[0]?.subscriptionProductId === "prod_enterprise"
|
|
? "enterprise"
|
|
: subscription?.products.length
|
|
? "pro"
|
|
: "free";
|
|
});
|
|
wireTenantPlanLookup();
|
|
```
|
|
|
|
Until a classifier is wired, the resolver returns NORMAL for every
|
|
tenant (CRITICAL still applies to internal jobs).
|
|
|
|
## Feature-flag
|
|
|
|
`JOB_PRIORITY_ENABLED=false` neutralises plan-based priority — every
|
|
tenant-scoped enqueue receives NORMAL (5). Internal jobs still get
|
|
CRITICAL because `tenantId: null` short-circuits before the flag is
|
|
checked. Use this for emergency rollback without a code change.
|
|
|
|
## Monitoring
|
|
|
|
The Counter `queue_jobs_enqueued_total{queue, priority}` increments
|
|
on every enqueue. The `priority` label is one of
|
|
`critical|high|normal|low|other` (low-cardinality).
|
|
|
|
Useful Grafana queries:
|
|
|
|
```promql
|
|
# Enqueue rate by priority for a queue
|
|
sum by (priority) (rate(queue_jobs_enqueued_total{queue="toolctrl.email"}[5m]))
|
|
|
|
# Share of critical vs total
|
|
sum(rate(queue_jobs_enqueued_total{priority="critical"}[5m]))
|
|
/
|
|
sum(rate(queue_jobs_enqueued_total[5m]))
|
|
```
|
|
|
|
## Rollback
|
|
|
|
1. Set `JOB_PRIORITY_ENABLED=false` in env → next enqueue runs with
|
|
NORMAL.
|
|
2. To remove the feature entirely: revert the producer-call-site
|
|
commits; the resolver helper stays but goes unused.
|
|
3. In-flight jobs keep the priority assigned at enqueue time.
|