234 lines
9.5 KiB
Markdown
234 lines
9.5 KiB
Markdown
# AI Cost Quotas (Phase 2)
|
|
|
|
Production-grade quota enforcement for LLM spend. Builds on
|
|
`docs/features/ai-cost-tracking.md` (Phase 1) which provides the
|
|
`AICostEvent` pipeline and `TenantAICostSummary` aggregate.
|
|
|
|
## What this layer adds
|
|
|
|
1. Plan-default monthly spend caps with optional per-tenant overrides.
|
|
2. A pre-call assertion that runs inside `executeLLMCall` and either
|
|
allows the call, soft-degrades it to a cheaper model, or throws
|
|
`AIQuotaExceededError` (mapped to HTTP 402 by the route layer).
|
|
3. Concurrency limiting per tenant (defaults: 5 for Pro, 20 for
|
|
Business) using a shared-store counter that auto-recovers from
|
|
crashes via a 10-minute TTL.
|
|
4. Idempotent 80% / 100% alerts driven both by the dispatcher
|
|
(on-demand after every persisted event) and by a 30-minute cron.
|
|
|
|
## Plan defaults
|
|
|
|
| Plan | Total cents / month | Concurrency | Per-capability caps |
|
|
| -------- | ------------------- | ----------- | ---------------------------------------------------------------------------------- |
|
|
| Starter | `0` (AI disabled) | 1 | n/a — every call denied with `ai-disabled` |
|
|
| Pro | `2000` ($20) | 5 | chat `1000`, embedding `400`, completion `600`, tool-call `600`, rerank `200` |
|
|
| Business | `20000` ($200) | 20 | chat `10000`, embedding `4000`, completion `6000`, tool-call `6000`, rerank `2000` |
|
|
|
|
Defaults live in
|
|
[`packages/entitlements/src/ai-quotas.ts`](../../packages/entitlements/src/ai-quotas.ts).
|
|
They were chosen so the cheapest plan blocks AI entirely (no free-
|
|
tier abuse) and the higher plans leave enough headroom for normal
|
|
SaaS usage while preventing any single capability from draining the
|
|
total budget.
|
|
|
|
## Per-tenant override
|
|
|
|
The `Tenant.aiQuotaOverrideJson` column holds a JSON blob shaped as
|
|
`TenantAIQuotaOverride`. Override fields take precedence over the
|
|
plan defaults; missing keys fall through. Useful for sales deals
|
|
("give this Pro customer a 5000-cent quota for the launch").
|
|
|
|
```json
|
|
{
|
|
"aiQuotaCentsPerMonth": 5000,
|
|
"aiMaxConcurrentCalls": 10,
|
|
"aiPerCapabilityCaps": { "chat": 3000 },
|
|
"aiDegradeTo": { "provider": "openai", "model": "gpt-4o-mini" },
|
|
"note": "SALES-DEAL-#4321"
|
|
}
|
|
```
|
|
|
|
The parsing helper `parseTenantAIQuotaOverride()` strips unknown
|
|
fields defensively so a malformed blob never breaks enforcement.
|
|
|
|
### Operator CLI
|
|
|
|
`toolctrl-cli ai-cost` exposes three quota subcommands. Each
|
|
mutation writes an audit event (`admin.ai.quota.override.{set,reset}`,
|
|
`actorType: "cli"`) so override changes always leave a trail.
|
|
|
|
```bash
|
|
# Inspect the resolved quota + current-month spend
|
|
toolctrl-cli ai-cost show-quota --tenant <id> [--json]
|
|
|
|
# Set / update an override — fields merge over any existing override,
|
|
# so a caps-only call keeps a previously-set total. Per-capability
|
|
# caps merge key-by-key.
|
|
toolctrl-cli ai-cost set-quota --tenant <id> \
|
|
--cents 5000 \
|
|
--max-concurrent 10 \
|
|
--caps chat=3000,embedding=400 \
|
|
--degrade openai/gpt-4o-mini \
|
|
--note "SALES-DEAL-#4321"
|
|
|
|
# Remove the override entirely (revert to the plan default)
|
|
toolctrl-cli ai-cost reset-quota --tenant <id> [--json]
|
|
```
|
|
|
|
`set-quota` round-trips the merged blob through
|
|
`parseTenantAIQuotaOverride()` before writing, so a malformed merge
|
|
can never be persisted. The plan id is inferred from the tenant's
|
|
most-recent subscription product; an unresolved id falls back to the
|
|
Starter blueprint.
|
|
|
|
The admin/tenant **web UIs** for quota visualisation remain a
|
|
follow-up (see the Phase-05 #55 detail-md Deviations) — the CLI is the
|
|
supported operator path until they land.
|
|
|
|
## Pre-call assertion lifecycle
|
|
|
|
```
|
|
caller --> executeLLMCall(args)
|
|
|
|
|
v
|
|
assertTenantAICreditAccess
|
|
1. Load plan default + tenant override
|
|
2. Load current-month spend from cache (60s TTL)
|
|
3. Reserve concurrency slot
|
|
4. Check capability sub-cap, then total quota
|
|
|
|
|
+-- allow ---------------+ continue
|
|
| |
|
|
+-- soft-degrade (flag) --+ swap (provider, model)
|
|
|
|
|
+-- throw AIQuotaExceededError
|
|
(route layer -> HTTP 402)
|
|
|
|
|
v
|
|
run(resolved)
|
|
|
|
|
v
|
|
releaseConcurrencySlot (finally)
|
|
```
|
|
|
|
Key files:
|
|
|
|
- `packages/ai/src/cost/assertCredit.ts` — the assertion engine.
|
|
- `packages/ai/src/cost/estimate.ts` — pessimistic upper-bound used
|
|
as input. Never undercounts: missing `maxOutputTokens` falls back
|
|
to the per-kind default (chat: 4096, embedding: 0).
|
|
- `packages/ai/src/cost/spend-cache.ts` — shared-store cache with
|
|
per-tenant invalidation hooks.
|
|
- `packages/ai/src/cost/executeLLMCall.ts` — orchestrator. Pass
|
|
`args.quota = { estimatedCostCents, deps }` to opt the call into
|
|
enforcement. System / admin calls (`tenantId === null`) bypass.
|
|
|
|
## Soft-degradation
|
|
|
|
Only triggers when ALL of the following hold:
|
|
|
|
1. The tenant has the `ai-quota-soft-degradation` feature flag
|
|
enabled (see `packages/feature-flags`). Inject the evaluator via
|
|
`AssertTenantQuotaDeps.isSoftDegradationEnabled`.
|
|
2. Current spend is already at or above 90% of the limit (the soft
|
|
threshold — see `SOFT_DEGRADATION_THRESHOLD`).
|
|
3. The resolved quota carries an `aiDegradeTo` target model.
|
|
|
|
When all three hold the call is allowed with `degradeTo` returned to
|
|
`executeLLMCall`, which passes the swap to the `run` callback. The
|
|
post-call event records the resolved provider/model so cost
|
|
attribution stays accurate.
|
|
|
|
If any of the three is false the call is hard-denied with
|
|
`reason: "quota-exceeded"`.
|
|
|
|
## Concurrency limit
|
|
|
|
Implemented with a shared-store counter at key
|
|
`ai-cost:concurrent:<tenantId>`. Reservation happens inside the
|
|
assertion; release runs inside the wrapper's `finally` block so a
|
|
provider failure does not leak the slot. A 10-minute TTL on the
|
|
counter key safeguards against worker crashes.
|
|
|
|
Concurrency is checked AFTER the per-capability and BEFORE the
|
|
total-spend check. This ordering means a tenant at 5 concurrent
|
|
calls but only 50% of quota will see `concurrency-limit-exceeded`
|
|
(not `quota-exceeded`), which is more actionable.
|
|
|
|
## Alerts
|
|
|
|
- 80% threshold: notify tenant admins, no action.
|
|
- 100% threshold: notify tenant admins; optional auto-disable hook
|
|
is reserved for future work.
|
|
|
|
Idempotency uses `Tenant.aiCostLastAlertedThreshold` paired with
|
|
`aiCostLastAlertedPeriod` (e.g. `"2026-05"`). The alert engine:
|
|
|
|
- Skips emit when the same threshold has already fired this month.
|
|
- Re-arms automatically when the period changes — new month, new
|
|
alerts.
|
|
|
|
Two entry points share a single decision function
|
|
`maybeAlertTenant()`:
|
|
|
|
1. `runAICostAlertJob({ tenantId })` — from the dispatcher after
|
|
every successful `AICostEvent` persist.
|
|
2. `runAICostAlertCron()` — every 30 minutes; iterates every tenant
|
|
with a configured quota.
|
|
|
|
Source: `apps/worker/src/jobs/ai-cost-alert.ts`.
|
|
|
|
## HTTP 402 client handling
|
|
|
|
Routes wrap LLM calls and re-throw `AIQuotaExceededError` as
|
|
`Response.json({ error: "ai-quota-exceeded", reason, … }, { status: 402 })`.
|
|
For chatbot streams, emit an SSE `event: error` with the same
|
|
payload and close the stream. The client renders an "Upgrade plan"
|
|
CTA pointing at the tenant subscription route
|
|
`/:tenant/settings/subscription`.
|
|
|
|
`isAIQuotaExceededError(error)` from `@toolctrl/ai/cost` is the
|
|
recommended type-guard in route action handlers.
|
|
|
|
## Killswitches
|
|
|
|
| Env var | Effect |
|
|
| ------------------------------------ | -------------------------------------------------------------- |
|
|
| `AI_QUOTA_ENFORCEMENT_ENABLED=false` | Skip every assertion; allow all calls. Production hotfix knob. |
|
|
| `AI_QUOTA_ALERTS_ENABLED=false` | Skip the alert engine entirely (cron + on-demand). |
|
|
| `AI_COST_TRACKING_ENABLED=false` | Phase 1 — disables the outbox publisher (no events written). |
|
|
|
|
The soft-degradation feature flag can be globally toggled off in the
|
|
admin feature-flag UI to force every breach into a hard-deny while
|
|
leaving enforcement on.
|
|
|
|
## Audit
|
|
|
|
The CLI quota override commands write audit entries for set/reset operations.
|
|
Future admin UI changes for the same override surface must keep that same
|
|
audit contract. The `TenantAIQuotaOverride.note` field is the operator's place
|
|
to reference the originating ticket / sales agreement.
|
|
|
|
## Troubleshooting
|
|
|
|
- **Calls denied despite the dashboard showing room**: the spend
|
|
cache may have a 60-second lag. The quota check uses the conservative
|
|
estimate `currentSpend + estimatedCost > limit`; check that
|
|
`estimateLLMCost` is not picking a huge `maxOutputTokens`.
|
|
- **Concurrency counter stuck**: TTL is 10 minutes. Force-clear via
|
|
Redis CLI `DEL shared-store:<namespace>:ai-cost:concurrent:<tenant>`
|
|
if a long-running operator process leaked a slot.
|
|
- **No alert fired at 80%**: check `Tenant.aiCostLastAlertedThreshold`.
|
|
If it equals or exceeds 80 for the current period, the engine
|
|
considers it already alerted. Reset to `NULL` to re-arm.
|
|
- **Soft-degrade not firing**: confirm the `ai-quota-soft-degradation`
|
|
flag is enabled for the tenant AND that `aiDegradeTo` is set on
|
|
the plan default or override. Without a degrade target the engine
|
|
hard-denies.
|
|
|
|
## Related files
|
|
|
|
- `prisma/migrations/20260517150000_05_ai_cost_phase_2_tenant_quota/migration.sql`
|
|
- `docs/improvement-plan/05-business-features/ai-cost-phase-2.md`
|
|
- `docs/features/ai-cost-tracking.md` (Phase 1)
|