Track HiveGrid Cloud planning reference
This commit is contained in:
1 parent
b3d5ea6764
commit
e341c5938a
5784 files changed
+1104929
-3
No files matched your search
@@ -0,0 +1,121 @@
|
||||
# Job Priority
|
||||
|
||||
Plan-based BullMQ job priority. Lets enterprise tenants jump the queue
|
||||
over free-tier batches and pins internal/system jobs to the top.
|
||||
|
||||
## Why
|
||||
|
||||
Before this feature every BullMQ job ran at the default priority. A
|
||||
free-tier tenant scheduling 10 000 mail jobs would block an enterprise
|
||||
tenant's password-reset behind them. Cron and retention jobs competed
|
||||
with user workload. There was no way to express "this auth mail must
|
||||
run now" semantically.
|
||||
|
||||
## Priority levels
|
||||
|
||||
| Constant | Value | Use case |
|
||||
| ------------------------ | ----- | --------------------------------------------------------- |
|
||||
| `JOB_PRIORITY_CRITICAL` | `1` | Internal jobs, auth flows (password-reset, magic-link) |
|
||||
| `JOB_PRIORITY_HIGH` | `3` | System-triggered workflows that aren't auth |
|
||||
| `JOB_PRIORITY_NORMAL` | `5` | Default for tenant-scoped jobs without a plan lookup |
|
||||
| `JOB_PRIORITY_LOW` | `10` | Marketing / bulk batches; free-tier when a lookup is set |
|
||||
|
||||
BullMQ treats `priority: 1` as highest. We intentionally use only 4
|
||||
distinct levels — Grafana dashboards stay legible and the priority
|
||||
sorted-set in Redis stays small.
|
||||
|
||||
## Resolution order (in `enqueueJob`)
|
||||
|
||||
```
|
||||
1. options.priority (explicit numeric) → wins everything
|
||||
2. options.priorityOverride ('critical'…) → mapped via priorityFromOverride
|
||||
3. options.tenantId === null → CRITICAL (internal job)
|
||||
4. options.tenantId set, lookup registered → resolveJobPriorityForTenant
|
||||
5. options.tenantId set, no lookup → NORMAL
|
||||
6. options absent → no priority field set
|
||||
```
|
||||
|
||||
## Plan → priority mapping
|
||||
|
||||
| Plan slug | Priority |
|
||||
| ------------ | ------------------ |
|
||||
| `enterprise` | CRITICAL (1) |
|
||||
| `pro` | NORMAL (5) |
|
||||
| `free` | LOW (10) |
|
||||
| unknown/null | NORMAL (5) |
|
||||
|
||||
The mapping lives in `packages/queue/src/priority.ts` in
|
||||
`priorityFromPlan(plan)`.
|
||||
|
||||
## Producer guidelines
|
||||
|
||||
1. **Always pass `tenantId`** for tenant-scoped jobs. The descriptor
|
||||
already carries the tenantId in most cases; pass the same value to
|
||||
the enqueue options:
|
||||
```ts
|
||||
await enqueueJob(queue, descriptor, { tenantId });
|
||||
```
|
||||
2. **Pass `tenantId: null` for internal jobs** — retention, cleanup,
|
||||
audit-compaction, backup triggers. They get CRITICAL automatically.
|
||||
3. **Use `priorityOverride: 'critical'` for auth flows** regardless of
|
||||
plan: password-reset, email-verification, magic-link, user
|
||||
invitation. A free-tier user must still receive their reset mail
|
||||
instantly.
|
||||
4. **Use `priorityOverride: 'low'` for marketing batches.** Don't let
|
||||
newsletter batches starve transactional mail.
|
||||
5. **Never set `priorityOverride: 'critical'` from untrusted input.**
|
||||
The override hook is server-only.
|
||||
|
||||
## Registering a tenant-plan lookup
|
||||
|
||||
The queue package keeps its plan-lookup pluggable so workers don't
|
||||
have to import apps/web. Web boots the lookup during cache bootstrap:
|
||||
|
||||
```ts
|
||||
// apps/web/app/utils/services/.server/tenantPlanLookup.server.ts
|
||||
setSubscriptionToPlanTier((subscription) => {
|
||||
// your classifier here — e.g. map stripe product ids to plan slugs
|
||||
return subscription?.products[0]?.subscriptionProductId === "prod_enterprise"
|
||||
? "enterprise"
|
||||
: subscription?.products.length
|
||||
? "pro"
|
||||
: "free";
|
||||
});
|
||||
wireTenantPlanLookup();
|
||||
```
|
||||
|
||||
Until a classifier is wired, the resolver returns NORMAL for every
|
||||
tenant (CRITICAL still applies to internal jobs).
|
||||
|
||||
## Feature-flag
|
||||
|
||||
`JOB_PRIORITY_ENABLED=false` neutralises plan-based priority — every
|
||||
tenant-scoped enqueue receives NORMAL (5). Internal jobs still get
|
||||
CRITICAL because `tenantId: null` short-circuits before the flag is
|
||||
checked. Use this for emergency rollback without a code change.
|
||||
|
||||
## Monitoring
|
||||
|
||||
The Counter `queue_jobs_enqueued_total{queue, priority}` increments
|
||||
on every enqueue. The `priority` label is one of
|
||||
`critical|high|normal|low|other` (low-cardinality).
|
||||
|
||||
Useful Grafana queries:
|
||||
|
||||
```promql
|
||||
# Enqueue rate by priority for a queue
|
||||
sum by (priority) (rate(queue_jobs_enqueued_total{queue="toolctrl.email"}[5m]))
|
||||
|
||||
# Share of critical vs total
|
||||
sum(rate(queue_jobs_enqueued_total{priority="critical"}[5m]))
|
||||
/
|
||||
sum(rate(queue_jobs_enqueued_total[5m]))
|
||||
```
|
||||
|
||||
## Rollback
|
||||
|
||||
1. Set `JOB_PRIORITY_ENABLED=false` in env → next enqueue runs with
|
||||
NORMAL.
|
||||
2. To remove the feature entirely: revert the producer-call-site
|
||||
commits; the resolver helper stays but goes unused.
|
||||
3. In-flight jobs keep the priority assigned at enqueue time.
|
||||
Reference in new issue
Block a user