Files
HiveGrid-Wireframes/hivegrid-cloud/docs/features/ai-cost-quotas.md
T

234 lines
9.5 KiB
Markdown

# AI Cost Quotas (Phase 2)
Production-grade quota enforcement for LLM spend. Builds on
`docs/features/ai-cost-tracking.md` (Phase 1) which provides the
`AICostEvent` pipeline and `TenantAICostSummary` aggregate.
## What this layer adds
1. Plan-default monthly spend caps with optional per-tenant overrides.
2. A pre-call assertion that runs inside `executeLLMCall` and either
allows the call, soft-degrades it to a cheaper model, or throws
`AIQuotaExceededError` (mapped to HTTP 402 by the route layer).
3. Concurrency limiting per tenant (defaults: 5 for Pro, 20 for
Business) using a shared-store counter that auto-recovers from
crashes via a 10-minute TTL.
4. Idempotent 80% / 100% alerts driven both by the dispatcher
(on-demand after every persisted event) and by a 30-minute cron.
## Plan defaults
| Plan | Total cents / month | Concurrency | Per-capability caps |
| -------- | ------------------- | ----------- | ---------------------------------------------------------------------------------- |
| Starter | `0` (AI disabled) | 1 | n/a — every call denied with `ai-disabled` |
| Pro | `2000` ($20) | 5 | chat `1000`, embedding `400`, completion `600`, tool-call `600`, rerank `200` |
| Business | `20000` ($200) | 20 | chat `10000`, embedding `4000`, completion `6000`, tool-call `6000`, rerank `2000` |
Defaults live in
[`packages/entitlements/src/ai-quotas.ts`](../../packages/entitlements/src/ai-quotas.ts).
They were chosen so the cheapest plan blocks AI entirely (no free-
tier abuse) and the higher plans leave enough headroom for normal
SaaS usage while preventing any single capability from draining the
total budget.
## Per-tenant override
The `Tenant.aiQuotaOverrideJson` column holds a JSON blob shaped as
`TenantAIQuotaOverride`. Override fields take precedence over the
plan defaults; missing keys fall through. Useful for sales deals
("give this Pro customer a 5000-cent quota for the launch").
```json
{
"aiQuotaCentsPerMonth": 5000,
"aiMaxConcurrentCalls": 10,
"aiPerCapabilityCaps": { "chat": 3000 },
"aiDegradeTo": { "provider": "openai", "model": "gpt-4o-mini" },
"note": "SALES-DEAL-#4321"
}
```
The parsing helper `parseTenantAIQuotaOverride()` strips unknown
fields defensively so a malformed blob never breaks enforcement.
### Operator CLI
`toolctrl-cli ai-cost` exposes three quota subcommands. Each
mutation writes an audit event (`admin.ai.quota.override.{set,reset}`,
`actorType: "cli"`) so override changes always leave a trail.
```bash
# Inspect the resolved quota + current-month spend
toolctrl-cli ai-cost show-quota --tenant <id> [--json]
# Set / update an override — fields merge over any existing override,
# so a caps-only call keeps a previously-set total. Per-capability
# caps merge key-by-key.
toolctrl-cli ai-cost set-quota --tenant <id> \
--cents 5000 \
--max-concurrent 10 \
--caps chat=3000,embedding=400 \
--degrade openai/gpt-4o-mini \
--note "SALES-DEAL-#4321"
# Remove the override entirely (revert to the plan default)
toolctrl-cli ai-cost reset-quota --tenant <id> [--json]
```
`set-quota` round-trips the merged blob through
`parseTenantAIQuotaOverride()` before writing, so a malformed merge
can never be persisted. The plan id is inferred from the tenant's
most-recent subscription product; an unresolved id falls back to the
Starter blueprint.
The admin/tenant **web UIs** for quota visualisation remain a
follow-up (see the Phase-05 #55 detail-md Deviations) — the CLI is the
supported operator path until they land.
## Pre-call assertion lifecycle
```
caller --> executeLLMCall(args)
|
v
assertTenantAICreditAccess
1. Load plan default + tenant override
2. Load current-month spend from cache (60s TTL)
3. Reserve concurrency slot
4. Check capability sub-cap, then total quota
|
+-- allow ---------------+ continue
| |
+-- soft-degrade (flag) --+ swap (provider, model)
|
+-- throw AIQuotaExceededError
(route layer -> HTTP 402)
|
v
run(resolved)
|
v
releaseConcurrencySlot (finally)
```
Key files:
- `packages/ai/src/cost/assertCredit.ts` — the assertion engine.
- `packages/ai/src/cost/estimate.ts` — pessimistic upper-bound used
as input. Never undercounts: missing `maxOutputTokens` falls back
to the per-kind default (chat: 4096, embedding: 0).
- `packages/ai/src/cost/spend-cache.ts` — shared-store cache with
per-tenant invalidation hooks.
- `packages/ai/src/cost/executeLLMCall.ts` — orchestrator. Pass
`args.quota = { estimatedCostCents, deps }` to opt the call into
enforcement. System / admin calls (`tenantId === null`) bypass.
## Soft-degradation
Only triggers when ALL of the following hold:
1. The tenant has the `ai-quota-soft-degradation` feature flag
enabled (see `packages/feature-flags`). Inject the evaluator via
`AssertTenantQuotaDeps.isSoftDegradationEnabled`.
2. Current spend is already at or above 90% of the limit (the soft
threshold — see `SOFT_DEGRADATION_THRESHOLD`).
3. The resolved quota carries an `aiDegradeTo` target model.
When all three hold the call is allowed with `degradeTo` returned to
`executeLLMCall`, which passes the swap to the `run` callback. The
post-call event records the resolved provider/model so cost
attribution stays accurate.
If any of the three is false the call is hard-denied with
`reason: "quota-exceeded"`.
## Concurrency limit
Implemented with a shared-store counter at key
`ai-cost:concurrent:<tenantId>`. Reservation happens inside the
assertion; release runs inside the wrapper's `finally` block so a
provider failure does not leak the slot. A 10-minute TTL on the
counter key safeguards against worker crashes.
Concurrency is checked AFTER the per-capability and BEFORE the
total-spend check. This ordering means a tenant at 5 concurrent
calls but only 50% of quota will see `concurrency-limit-exceeded`
(not `quota-exceeded`), which is more actionable.
## Alerts
- 80% threshold: notify tenant admins, no action.
- 100% threshold: notify tenant admins; optional auto-disable hook
is reserved for future work.
Idempotency uses `Tenant.aiCostLastAlertedThreshold` paired with
`aiCostLastAlertedPeriod` (e.g. `"2026-05"`). The alert engine:
- Skips emit when the same threshold has already fired this month.
- Re-arms automatically when the period changes — new month, new
alerts.
Two entry points share a single decision function
`maybeAlertTenant()`:
1. `runAICostAlertJob({ tenantId })` — from the dispatcher after
every successful `AICostEvent` persist.
2. `runAICostAlertCron()` — every 30 minutes; iterates every tenant
with a configured quota.
Source: `apps/worker/src/jobs/ai-cost-alert.ts`.
## HTTP 402 client handling
Routes wrap LLM calls and re-throw `AIQuotaExceededError` as
`Response.json({ error: "ai-quota-exceeded", reason, … }, { status: 402 })`.
For chatbot streams, emit an SSE `event: error` with the same
payload and close the stream. The client renders an "Upgrade plan"
CTA pointing at the tenant subscription route
`/:tenant/settings/subscription`.
`isAIQuotaExceededError(error)` from `@toolctrl/ai/cost` is the
recommended type-guard in route action handlers.
## Killswitches
| Env var | Effect |
| ------------------------------------ | -------------------------------------------------------------- |
| `AI_QUOTA_ENFORCEMENT_ENABLED=false` | Skip every assertion; allow all calls. Production hotfix knob. |
| `AI_QUOTA_ALERTS_ENABLED=false` | Skip the alert engine entirely (cron + on-demand). |
| `AI_COST_TRACKING_ENABLED=false` | Phase 1 — disables the outbox publisher (no events written). |
The soft-degradation feature flag can be globally toggled off in the
admin feature-flag UI to force every breach into a hard-deny while
leaving enforcement on.
## Audit
The CLI quota override commands write audit entries for set/reset operations.
Future admin UI changes for the same override surface must keep that same
audit contract. The `TenantAIQuotaOverride.note` field is the operator's place
to reference the originating ticket / sales agreement.
## Troubleshooting
- **Calls denied despite the dashboard showing room**: the spend
cache may have a 60-second lag. The quota check uses the conservative
estimate `currentSpend + estimatedCost > limit`; check that
`estimateLLMCost` is not picking a huge `maxOutputTokens`.
- **Concurrency counter stuck**: TTL is 10 minutes. Force-clear via
Redis CLI `DEL shared-store:<namespace>:ai-cost:concurrent:<tenant>`
if a long-running operator process leaked a slot.
- **No alert fired at 80%**: check `Tenant.aiCostLastAlertedThreshold`.
If it equals or exceeds 80 for the current period, the engine
considers it already alerted. Reset to `NULL` to re-arm.
- **Soft-degrade not firing**: confirm the `ai-quota-soft-degradation`
flag is enabled for the tenant AND that `aiDegradeTo` is set on
the plan default or override. Without a degrade target the engine
hard-denies.
## Related files
- `prisma/migrations/20260517150000_05_ai_cost_phase_2_tenant_quota/migration.sql`
- `docs/improvement-plan/05-business-features/ai-cost-phase-2.md`
- `docs/features/ai-cost-tracking.md` (Phase 1)