# AI Cost Quotas (Phase 2) Production-grade quota enforcement for LLM spend. Builds on `docs/features/ai-cost-tracking.md` (Phase 1) which provides the `AICostEvent` pipeline and `TenantAICostSummary` aggregate. ## What this layer adds 1. Plan-default monthly spend caps with optional per-tenant overrides. 2. A pre-call assertion that runs inside `executeLLMCall` and either allows the call, soft-degrades it to a cheaper model, or throws `AIQuotaExceededError` (mapped to HTTP 402 by the route layer). 3. Concurrency limiting per tenant (defaults: 5 for Pro, 20 for Business) using a shared-store counter that auto-recovers from crashes via a 10-minute TTL. 4. Idempotent 80% / 100% alerts driven both by the dispatcher (on-demand after every persisted event) and by a 30-minute cron. ## Plan defaults | Plan | Total cents / month | Concurrency | Per-capability caps | | -------- | ------------------- | ----------- | ---------------------------------------------------------------------------------- | | Starter | `0` (AI disabled) | 1 | n/a — every call denied with `ai-disabled` | | Pro | `2000` ($20) | 5 | chat `1000`, embedding `400`, completion `600`, tool-call `600`, rerank `200` | | Business | `20000` ($200) | 20 | chat `10000`, embedding `4000`, completion `6000`, tool-call `6000`, rerank `2000` | Defaults live in [`packages/entitlements/src/ai-quotas.ts`](../../packages/entitlements/src/ai-quotas.ts). They were chosen so the cheapest plan blocks AI entirely (no free- tier abuse) and the higher plans leave enough headroom for normal SaaS usage while preventing any single capability from draining the total budget. ## Per-tenant override The `Tenant.aiQuotaOverrideJson` column holds a JSON blob shaped as `TenantAIQuotaOverride`. Override fields take precedence over the plan defaults; missing keys fall through. Useful for sales deals ("give this Pro customer a 5000-cent quota for the launch"). ```json { "aiQuotaCentsPerMonth": 5000, "aiMaxConcurrentCalls": 10, "aiPerCapabilityCaps": { "chat": 3000 }, "aiDegradeTo": { "provider": "openai", "model": "gpt-4o-mini" }, "note": "SALES-DEAL-#4321" } ``` The parsing helper `parseTenantAIQuotaOverride()` strips unknown fields defensively so a malformed blob never breaks enforcement. ### Operator CLI `toolctrl-cli ai-cost` exposes three quota subcommands. Each mutation writes an audit event (`admin.ai.quota.override.{set,reset}`, `actorType: "cli"`) so override changes always leave a trail. ```bash # Inspect the resolved quota + current-month spend toolctrl-cli ai-cost show-quota --tenant [--json] # Set / update an override — fields merge over any existing override, # so a caps-only call keeps a previously-set total. Per-capability # caps merge key-by-key. toolctrl-cli ai-cost set-quota --tenant \ --cents 5000 \ --max-concurrent 10 \ --caps chat=3000,embedding=400 \ --degrade openai/gpt-4o-mini \ --note "SALES-DEAL-#4321" # Remove the override entirely (revert to the plan default) toolctrl-cli ai-cost reset-quota --tenant [--json] ``` `set-quota` round-trips the merged blob through `parseTenantAIQuotaOverride()` before writing, so a malformed merge can never be persisted. The plan id is inferred from the tenant's most-recent subscription product; an unresolved id falls back to the Starter blueprint. The admin/tenant **web UIs** for quota visualisation remain a follow-up (see the Phase-05 #55 detail-md Deviations) — the CLI is the supported operator path until they land. ## Pre-call assertion lifecycle ``` caller --> executeLLMCall(args) | v assertTenantAICreditAccess 1. Load plan default + tenant override 2. Load current-month spend from cache (60s TTL) 3. Reserve concurrency slot 4. Check capability sub-cap, then total quota | +-- allow ---------------+ continue | | +-- soft-degrade (flag) --+ swap (provider, model) | +-- throw AIQuotaExceededError (route layer -> HTTP 402) | v run(resolved) | v releaseConcurrencySlot (finally) ``` Key files: - `packages/ai/src/cost/assertCredit.ts` — the assertion engine. - `packages/ai/src/cost/estimate.ts` — pessimistic upper-bound used as input. Never undercounts: missing `maxOutputTokens` falls back to the per-kind default (chat: 4096, embedding: 0). - `packages/ai/src/cost/spend-cache.ts` — shared-store cache with per-tenant invalidation hooks. - `packages/ai/src/cost/executeLLMCall.ts` — orchestrator. Pass `args.quota = { estimatedCostCents, deps }` to opt the call into enforcement. System / admin calls (`tenantId === null`) bypass. ## Soft-degradation Only triggers when ALL of the following hold: 1. The tenant has the `ai-quota-soft-degradation` feature flag enabled (see `packages/feature-flags`). Inject the evaluator via `AssertTenantQuotaDeps.isSoftDegradationEnabled`. 2. Current spend is already at or above 90% of the limit (the soft threshold — see `SOFT_DEGRADATION_THRESHOLD`). 3. The resolved quota carries an `aiDegradeTo` target model. When all three hold the call is allowed with `degradeTo` returned to `executeLLMCall`, which passes the swap to the `run` callback. The post-call event records the resolved provider/model so cost attribution stays accurate. If any of the three is false the call is hard-denied with `reason: "quota-exceeded"`. ## Concurrency limit Implemented with a shared-store counter at key `ai-cost:concurrent:`. Reservation happens inside the assertion; release runs inside the wrapper's `finally` block so a provider failure does not leak the slot. A 10-minute TTL on the counter key safeguards against worker crashes. Concurrency is checked AFTER the per-capability and BEFORE the total-spend check. This ordering means a tenant at 5 concurrent calls but only 50% of quota will see `concurrency-limit-exceeded` (not `quota-exceeded`), which is more actionable. ## Alerts - 80% threshold: notify tenant admins, no action. - 100% threshold: notify tenant admins; optional auto-disable hook is reserved for future work. Idempotency uses `Tenant.aiCostLastAlertedThreshold` paired with `aiCostLastAlertedPeriod` (e.g. `"2026-05"`). The alert engine: - Skips emit when the same threshold has already fired this month. - Re-arms automatically when the period changes — new month, new alerts. Two entry points share a single decision function `maybeAlertTenant()`: 1. `runAICostAlertJob({ tenantId })` — from the dispatcher after every successful `AICostEvent` persist. 2. `runAICostAlertCron()` — every 30 minutes; iterates every tenant with a configured quota. Source: `apps/worker/src/jobs/ai-cost-alert.ts`. ## HTTP 402 client handling Routes wrap LLM calls and re-throw `AIQuotaExceededError` as `Response.json({ error: "ai-quota-exceeded", reason, … }, { status: 402 })`. For chatbot streams, emit an SSE `event: error` with the same payload and close the stream. The client renders an "Upgrade plan" CTA pointing at the tenant subscription route `/:tenant/settings/subscription`. `isAIQuotaExceededError(error)` from `@toolctrl/ai/cost` is the recommended type-guard in route action handlers. ## Killswitches | Env var | Effect | | ------------------------------------ | -------------------------------------------------------------- | | `AI_QUOTA_ENFORCEMENT_ENABLED=false` | Skip every assertion; allow all calls. Production hotfix knob. | | `AI_QUOTA_ALERTS_ENABLED=false` | Skip the alert engine entirely (cron + on-demand). | | `AI_COST_TRACKING_ENABLED=false` | Phase 1 — disables the outbox publisher (no events written). | The soft-degradation feature flag can be globally toggled off in the admin feature-flag UI to force every breach into a hard-deny while leaving enforcement on. ## Audit The CLI quota override commands write audit entries for set/reset operations. Future admin UI changes for the same override surface must keep that same audit contract. The `TenantAIQuotaOverride.note` field is the operator's place to reference the originating ticket / sales agreement. ## Troubleshooting - **Calls denied despite the dashboard showing room**: the spend cache may have a 60-second lag. The quota check uses the conservative estimate `currentSpend + estimatedCost > limit`; check that `estimateLLMCost` is not picking a huge `maxOutputTokens`. - **Concurrency counter stuck**: TTL is 10 minutes. Force-clear via Redis CLI `DEL shared-store::ai-cost:concurrent:` if a long-running operator process leaked a slot. - **No alert fired at 80%**: check `Tenant.aiCostLastAlertedThreshold`. If it equals or exceeds 80 for the current period, the engine considers it already alerted. Reset to `NULL` to re-arm. - **Soft-degrade not firing**: confirm the `ai-quota-soft-degradation` flag is enabled for the tenant AND that `aiDegradeTo` is set on the plan default or override. Without a degrade target the engine hard-denies. ## Related files - `prisma/migrations/20260517150000_05_ai_cost_phase_2_tenant_quota/migration.sql` - `docs/improvement-plan/05-business-features/ai-cost-phase-2.md` - `docs/features/ai-cost-tracking.md` (Phase 1)