HiveGrid-Wireframes/hivegrid-cloud/docs/features/ai-cost-quotas.md

AI Cost Quotas (Phase 2)

Production-grade quota enforcement for LLM spend. Builds on docs/features/ai-cost-tracking.md (Phase 1) which provides the AICostEvent pipeline and TenantAICostSummary aggregate.

What this layer adds

1. Plan-default monthly spend caps with optional per-tenant overrides.
2. A pre-call assertion that runs inside executeLLMCall and either allows the call, soft-degrades it to a cheaper model, or throws AIQuotaExceededError (mapped to HTTP 402 by the route layer).
3. Concurrency limiting per tenant (defaults: 5 for Pro, 20 for Business) using a shared-store counter that auto-recovers from crashes via a 10-minute TTL.
4. Idempotent 80% / 100% alerts driven both by the dispatcher (on-demand after every persisted event) and by a 30-minute cron.

Plan defaults

Plan	Total cents / month	Concurrency	Per-capability caps
Starter	0 (AI disabled)	1	n/a — every call denied with ai-disabled
Pro	2000 ($20)	5	chat 1000, embedding 400, completion 600, tool-call 600, rerank 200
Business	20000 ($200)	20	chat 10000, embedding 4000, completion 6000, tool-call 6000, rerank 2000

Defaults live in packages/entitlements/src/ai-quotas.ts. They were chosen so the cheapest plan blocks AI entirely (no free- tier abuse) and the higher plans leave enough headroom for normal SaaS usage while preventing any single capability from draining the total budget.

Per-tenant override

The Tenant.aiQuotaOverrideJson column holds a JSON blob shaped as TenantAIQuotaOverride. Override fields take precedence over the plan defaults; missing keys fall through. Useful for sales deals ("give this Pro customer a 5000-cent quota for the launch").

{
  "aiQuotaCentsPerMonth": 5000,
  "aiMaxConcurrentCalls": 10,
  "aiPerCapabilityCaps": { "chat": 3000 },
  "aiDegradeTo": { "provider": "openai", "model": "gpt-4o-mini" },
  "note": "SALES-DEAL-#4321"
}

The parsing helper parseTenantAIQuotaOverride() strips unknown fields defensively so a malformed blob never breaks enforcement.

Operator CLI

toolctrl-cli ai-cost exposes three quota subcommands. Each mutation writes an audit event (admin.ai.quota.override.{set,reset}, actorType: "cli") so override changes always leave a trail.

# Inspect the resolved quota + current-month spend
toolctrl-cli ai-cost show-quota --tenant <id> [--json]

# Set / update an override — fields merge over any existing override,
# so a caps-only call keeps a previously-set total. Per-capability
# caps merge key-by-key.
toolctrl-cli ai-cost set-quota --tenant <id> \
  --cents 5000 \
  --max-concurrent 10 \
  --caps chat=3000,embedding=400 \
  --degrade openai/gpt-4o-mini \
  --note "SALES-DEAL-#4321"

# Remove the override entirely (revert to the plan default)
toolctrl-cli ai-cost reset-quota --tenant <id> [--json]

set-quota round-trips the merged blob through parseTenantAIQuotaOverride() before writing, so a malformed merge can never be persisted. The plan id is inferred from the tenant's most-recent subscription product; an unresolved id falls back to the Starter blueprint.

The admin/tenant web UIs for quota visualisation remain a follow-up (see the Phase-05 #55 detail-md Deviations) — the CLI is the supported operator path until they land.

Pre-call assertion lifecycle

caller --> executeLLMCall(args)
              |
              v
        assertTenantAICreditAccess
          1. Load plan default + tenant override
          2. Load current-month spend from cache (60s TTL)
          3. Reserve concurrency slot
          4. Check capability sub-cap, then total quota
              |
              +-- allow ---------------+ continue
              |                         |
              +-- soft-degrade (flag) --+ swap (provider, model)
              |
              +-- throw AIQuotaExceededError
                  (route layer -> HTTP 402)
              |
              v
            run(resolved)
              |
              v
          releaseConcurrencySlot (finally)

Key files:

• packages/ai/src/cost/assertCredit.ts — the assertion engine.
• packages/ai/src/cost/estimate.ts — pessimistic upper-bound used as input. Never undercounts: missing maxOutputTokens falls back to the per-kind default (chat: 4096, embedding: 0).
• packages/ai/src/cost/spend-cache.ts — shared-store cache with per-tenant invalidation hooks.
• packages/ai/src/cost/executeLLMCall.ts — orchestrator. Pass args.quota = { estimatedCostCents, deps } to opt the call into enforcement. System / admin calls (tenantId === null) bypass.

Soft-degradation

Only triggers when ALL of the following hold:

1. The tenant has the ai-quota-soft-degradation feature flag enabled (see packages/feature-flags). Inject the evaluator via AssertTenantQuotaDeps.isSoftDegradationEnabled.
2. Current spend is already at or above 90% of the limit (the soft threshold — see SOFT_DEGRADATION_THRESHOLD).
3. The resolved quota carries an aiDegradeTo target model.

When all three hold the call is allowed with degradeTo returned to executeLLMCall, which passes the swap to the run callback. The post-call event records the resolved provider/model so cost attribution stays accurate.

If any of the three is false the call is hard-denied with reason: "quota-exceeded".

Concurrency limit

Implemented with a shared-store counter at key ai-cost:concurrent:<tenantId>. Reservation happens inside the assertion; release runs inside the wrapper's finally block so a provider failure does not leak the slot. A 10-minute TTL on the counter key safeguards against worker crashes.

Concurrency is checked AFTER the per-capability and BEFORE the total-spend check. This ordering means a tenant at 5 concurrent calls but only 50% of quota will see concurrency-limit-exceeded (not quota-exceeded), which is more actionable.

Alerts

• 80% threshold: notify tenant admins, no action.
• 100% threshold: notify tenant admins; optional auto-disable hook is reserved for future work.

Idempotency uses Tenant.aiCostLastAlertedThreshold paired with aiCostLastAlertedPeriod (e.g. "2026-05"). The alert engine:

• Skips emit when the same threshold has already fired this month.
• Re-arms automatically when the period changes — new month, new alerts.

Two entry points share a single decision function maybeAlertTenant():

1. runAICostAlertJob({ tenantId }) — from the dispatcher after every successful AICostEvent persist.
2. runAICostAlertCron() — every 30 minutes; iterates every tenant with a configured quota.

Source: apps/worker/src/jobs/ai-cost-alert.ts.

HTTP 402 client handling

Routes wrap LLM calls and re-throw AIQuotaExceededError as Response.json({ error: "ai-quota-exceeded", reason, … }, { status: 402 }). For chatbot streams, emit an SSE event: error with the same payload and close the stream. The client renders an "Upgrade plan" CTA pointing at the tenant subscription route /:tenant/settings/subscription.

isAIQuotaExceededError(error) from @toolctrl/ai/cost is the recommended type-guard in route action handlers.

Killswitches

Env var	Effect
AI_QUOTA_ENFORCEMENT_ENABLED=false	Skip every assertion; allow all calls. Production hotfix knob.
AI_QUOTA_ALERTS_ENABLED=false	Skip the alert engine entirely (cron + on-demand).
AI_COST_TRACKING_ENABLED=false	Phase 1 — disables the outbox publisher (no events written).

The soft-degradation feature flag can be globally toggled off in the admin feature-flag UI to force every breach into a hard-deny while leaving enforcement on.

Audit

The CLI quota override commands write audit entries for set/reset operations. Future admin UI changes for the same override surface must keep that same audit contract. The TenantAIQuotaOverride.note field is the operator's place to reference the originating ticket / sales agreement.

Troubleshooting

• Calls denied despite the dashboard showing room: the spend cache may have a 60-second lag. The quota check uses the conservative estimate currentSpend + estimatedCost > limit; check that estimateLLMCost is not picking a huge maxOutputTokens.
• Concurrency counter stuck: TTL is 10 minutes. Force-clear via Redis CLI DEL shared-store:<namespace>:ai-cost:concurrent:<tenant> if a long-running operator process leaked a slot.
• No alert fired at 80%: check Tenant.aiCostLastAlertedThreshold. If it equals or exceeds 80 for the current period, the engine considers it already alerted. Reset to NULL to re-arm.
• Soft-degrade not firing: confirm the ai-quota-soft-degradation flag is enabled for the tenant AND that aiDegradeTo is set on the plan default or override. Without a degrade target the engine hard-denies.

Related files

• prisma/migrations/20260517150000_05_ai_cost_phase_2_tenant_quota/migration.sql
• docs/improvement-plan/05-business-features/ai-cost-phase-2.md
• docs/features/ai-cost-tracking.md (Phase 1)