HiveGrid-Wireframes/hivegrid-cloud/docs/architecture/job-priority.md

Job Priority

Plan-based BullMQ job priority. Lets enterprise tenants jump the queue over free-tier batches and pins internal/system jobs to the top.

Why

Before this feature every BullMQ job ran at the default priority. A free-tier tenant scheduling 10 000 mail jobs would block an enterprise tenant's password-reset behind them. Cron and retention jobs competed with user workload. There was no way to express "this auth mail must run now" semantically.

Priority levels

Constant	Value	Use case
JOB_PRIORITY_CRITICAL	1	Internal jobs, auth flows (password-reset, magic-link)
JOB_PRIORITY_HIGH	3	System-triggered workflows that aren't auth
JOB_PRIORITY_NORMAL	5	Default for tenant-scoped jobs without a plan lookup
JOB_PRIORITY_LOW	10	Marketing / bulk batches; free-tier when a lookup is set

BullMQ treats priority: 1 as highest. We intentionally use only 4 distinct levels — Grafana dashboards stay legible and the priority sorted-set in Redis stays small.

Resolution order (in enqueueJob)

1. options.priority (explicit numeric)      → wins everything
2. options.priorityOverride ('critical'…)   → mapped via priorityFromOverride
3. options.tenantId === null                → CRITICAL (internal job)
4. options.tenantId set, lookup registered  → resolveJobPriorityForTenant
5. options.tenantId set, no lookup          → NORMAL
6. options absent                           → no priority field set

Plan → priority mapping

Plan slug	Priority
enterprise	CRITICAL (1)
pro	NORMAL (5)
free	LOW (10)
unknown/null	NORMAL (5)

The mapping lives in packages/queue/src/priority.ts in priorityFromPlan(plan).

Producer guidelines

1. Always pass tenantId for tenant-scoped jobs. The descriptor already carries the tenantId in most cases; pass the same value to the enqueue options:

   await enqueueJob(queue, descriptor, { tenantId });
   

2. Pass tenantId: null for internal jobs — retention, cleanup, audit-compaction, backup triggers. They get CRITICAL automatically.
3. Use priorityOverride: 'critical' for auth flows regardless of plan: password-reset, email-verification, magic-link, user invitation. A free-tier user must still receive their reset mail instantly.
4. Use priorityOverride: 'low' for marketing batches. Don't let newsletter batches starve transactional mail.
5. Never set priorityOverride: 'critical' from untrusted input. The override hook is server-only.

Registering a tenant-plan lookup

The queue package keeps its plan-lookup pluggable so workers don't have to import apps/web. Web boots the lookup during cache bootstrap:

// apps/web/app/utils/services/.server/tenantPlanLookup.server.ts
setSubscriptionToPlanTier((subscription) => {
  // your classifier here — e.g. map stripe product ids to plan slugs
  return subscription?.products[0]?.subscriptionProductId === "prod_enterprise"
    ? "enterprise"
    : subscription?.products.length
      ? "pro"
      : "free";
});
wireTenantPlanLookup();

Until a classifier is wired, the resolver returns NORMAL for every tenant (CRITICAL still applies to internal jobs).

Feature-flag

JOB_PRIORITY_ENABLED=false neutralises plan-based priority — every tenant-scoped enqueue receives NORMAL (5). Internal jobs still get CRITICAL because tenantId: null short-circuits before the flag is checked. Use this for emergency rollback without a code change.

Monitoring

The Counter queue_jobs_enqueued_total{queue, priority} increments on every enqueue. The priority label is one of critical|high|normal|low|other (low-cardinality).

Useful Grafana queries:

# Enqueue rate by priority for a queue
sum by (priority) (rate(queue_jobs_enqueued_total{queue="toolctrl.email"}[5m]))

# Share of critical vs total
sum(rate(queue_jobs_enqueued_total{priority="critical"}[5m]))
/
sum(rate(queue_jobs_enqueued_total[5m]))

Rollback

1. Set JOB_PRIORITY_ENABLED=false in env → next enqueue runs with NORMAL.
2. To remove the feature entirely: revert the producer-call-site commits; the resolver helper stays but goes unused.
3. In-flight jobs keep the priority assigned at enqueue time.