Track HiveGrid Cloud planning reference
This commit is contained in:
1 parent
b3d5ea6764
commit
e341c5938a
5784 files changed
+1104929
-3
No files matched your search
@@ -0,0 +1,233 @@
|
||||
# AI Cost Quotas (Phase 2)
|
||||
|
||||
Production-grade quota enforcement for LLM spend. Builds on
|
||||
`docs/features/ai-cost-tracking.md` (Phase 1) which provides the
|
||||
`AICostEvent` pipeline and `TenantAICostSummary` aggregate.
|
||||
|
||||
## What this layer adds
|
||||
|
||||
1. Plan-default monthly spend caps with optional per-tenant overrides.
|
||||
2. A pre-call assertion that runs inside `executeLLMCall` and either
|
||||
allows the call, soft-degrades it to a cheaper model, or throws
|
||||
`AIQuotaExceededError` (mapped to HTTP 402 by the route layer).
|
||||
3. Concurrency limiting per tenant (defaults: 5 for Pro, 20 for
|
||||
Business) using a shared-store counter that auto-recovers from
|
||||
crashes via a 10-minute TTL.
|
||||
4. Idempotent 80% / 100% alerts driven both by the dispatcher
|
||||
(on-demand after every persisted event) and by a 30-minute cron.
|
||||
|
||||
## Plan defaults
|
||||
|
||||
| Plan | Total cents / month | Concurrency | Per-capability caps |
|
||||
| -------- | ------------------- | ----------- | ---------------------------------------------------------------------------------- |
|
||||
| Starter | `0` (AI disabled) | 1 | n/a — every call denied with `ai-disabled` |
|
||||
| Pro | `2000` ($20) | 5 | chat `1000`, embedding `400`, completion `600`, tool-call `600`, rerank `200` |
|
||||
| Business | `20000` ($200) | 20 | chat `10000`, embedding `4000`, completion `6000`, tool-call `6000`, rerank `2000` |
|
||||
|
||||
Defaults live in
|
||||
[`packages/entitlements/src/ai-quotas.ts`](../../packages/entitlements/src/ai-quotas.ts).
|
||||
They were chosen so the cheapest plan blocks AI entirely (no free-
|
||||
tier abuse) and the higher plans leave enough headroom for normal
|
||||
SaaS usage while preventing any single capability from draining the
|
||||
total budget.
|
||||
|
||||
## Per-tenant override
|
||||
|
||||
The `Tenant.aiQuotaOverrideJson` column holds a JSON blob shaped as
|
||||
`TenantAIQuotaOverride`. Override fields take precedence over the
|
||||
plan defaults; missing keys fall through. Useful for sales deals
|
||||
("give this Pro customer a 5000-cent quota for the launch").
|
||||
|
||||
```json
|
||||
{
|
||||
"aiQuotaCentsPerMonth": 5000,
|
||||
"aiMaxConcurrentCalls": 10,
|
||||
"aiPerCapabilityCaps": { "chat": 3000 },
|
||||
"aiDegradeTo": { "provider": "openai", "model": "gpt-4o-mini" },
|
||||
"note": "SALES-DEAL-#4321"
|
||||
}
|
||||
```
|
||||
|
||||
The parsing helper `parseTenantAIQuotaOverride()` strips unknown
|
||||
fields defensively so a malformed blob never breaks enforcement.
|
||||
|
||||
### Operator CLI
|
||||
|
||||
`toolctrl-cli ai-cost` exposes three quota subcommands. Each
|
||||
mutation writes an audit event (`admin.ai.quota.override.{set,reset}`,
|
||||
`actorType: "cli"`) so override changes always leave a trail.
|
||||
|
||||
```bash
|
||||
# Inspect the resolved quota + current-month spend
|
||||
toolctrl-cli ai-cost show-quota --tenant <id> [--json]
|
||||
|
||||
# Set / update an override — fields merge over any existing override,
|
||||
# so a caps-only call keeps a previously-set total. Per-capability
|
||||
# caps merge key-by-key.
|
||||
toolctrl-cli ai-cost set-quota --tenant <id> \
|
||||
--cents 5000 \
|
||||
--max-concurrent 10 \
|
||||
--caps chat=3000,embedding=400 \
|
||||
--degrade openai/gpt-4o-mini \
|
||||
--note "SALES-DEAL-#4321"
|
||||
|
||||
# Remove the override entirely (revert to the plan default)
|
||||
toolctrl-cli ai-cost reset-quota --tenant <id> [--json]
|
||||
```
|
||||
|
||||
`set-quota` round-trips the merged blob through
|
||||
`parseTenantAIQuotaOverride()` before writing, so a malformed merge
|
||||
can never be persisted. The plan id is inferred from the tenant's
|
||||
most-recent subscription product; an unresolved id falls back to the
|
||||
Starter blueprint.
|
||||
|
||||
The admin/tenant **web UIs** for quota visualisation remain a
|
||||
follow-up (see the Phase-05 #55 detail-md Deviations) — the CLI is the
|
||||
supported operator path until they land.
|
||||
|
||||
## Pre-call assertion lifecycle
|
||||
|
||||
```
|
||||
caller --> executeLLMCall(args)
|
||||
|
|
||||
v
|
||||
assertTenantAICreditAccess
|
||||
1. Load plan default + tenant override
|
||||
2. Load current-month spend from cache (60s TTL)
|
||||
3. Reserve concurrency slot
|
||||
4. Check capability sub-cap, then total quota
|
||||
|
|
||||
+-- allow ---------------+ continue
|
||||
| |
|
||||
+-- soft-degrade (flag) --+ swap (provider, model)
|
||||
|
|
||||
+-- throw AIQuotaExceededError
|
||||
(route layer -> HTTP 402)
|
||||
|
|
||||
v
|
||||
run(resolved)
|
||||
|
|
||||
v
|
||||
releaseConcurrencySlot (finally)
|
||||
```
|
||||
|
||||
Key files:
|
||||
|
||||
- `packages/ai/src/cost/assertCredit.ts` — the assertion engine.
|
||||
- `packages/ai/src/cost/estimate.ts` — pessimistic upper-bound used
|
||||
as input. Never undercounts: missing `maxOutputTokens` falls back
|
||||
to the per-kind default (chat: 4096, embedding: 0).
|
||||
- `packages/ai/src/cost/spend-cache.ts` — shared-store cache with
|
||||
per-tenant invalidation hooks.
|
||||
- `packages/ai/src/cost/executeLLMCall.ts` — orchestrator. Pass
|
||||
`args.quota = { estimatedCostCents, deps }` to opt the call into
|
||||
enforcement. System / admin calls (`tenantId === null`) bypass.
|
||||
|
||||
## Soft-degradation
|
||||
|
||||
Only triggers when ALL of the following hold:
|
||||
|
||||
1. The tenant has the `ai-quota-soft-degradation` feature flag
|
||||
enabled (see `packages/feature-flags`). Inject the evaluator via
|
||||
`AssertTenantQuotaDeps.isSoftDegradationEnabled`.
|
||||
2. Current spend is already at or above 90% of the limit (the soft
|
||||
threshold — see `SOFT_DEGRADATION_THRESHOLD`).
|
||||
3. The resolved quota carries an `aiDegradeTo` target model.
|
||||
|
||||
When all three hold the call is allowed with `degradeTo` returned to
|
||||
`executeLLMCall`, which passes the swap to the `run` callback. The
|
||||
post-call event records the resolved provider/model so cost
|
||||
attribution stays accurate.
|
||||
|
||||
If any of the three is false the call is hard-denied with
|
||||
`reason: "quota-exceeded"`.
|
||||
|
||||
## Concurrency limit
|
||||
|
||||
Implemented with a shared-store counter at key
|
||||
`ai-cost:concurrent:<tenantId>`. Reservation happens inside the
|
||||
assertion; release runs inside the wrapper's `finally` block so a
|
||||
provider failure does not leak the slot. A 10-minute TTL on the
|
||||
counter key safeguards against worker crashes.
|
||||
|
||||
Concurrency is checked AFTER the per-capability and BEFORE the
|
||||
total-spend check. This ordering means a tenant at 5 concurrent
|
||||
calls but only 50% of quota will see `concurrency-limit-exceeded`
|
||||
(not `quota-exceeded`), which is more actionable.
|
||||
|
||||
## Alerts
|
||||
|
||||
- 80% threshold: notify tenant admins, no action.
|
||||
- 100% threshold: notify tenant admins; optional auto-disable hook
|
||||
is reserved for future work.
|
||||
|
||||
Idempotency uses `Tenant.aiCostLastAlertedThreshold` paired with
|
||||
`aiCostLastAlertedPeriod` (e.g. `"2026-05"`). The alert engine:
|
||||
|
||||
- Skips emit when the same threshold has already fired this month.
|
||||
- Re-arms automatically when the period changes — new month, new
|
||||
alerts.
|
||||
|
||||
Two entry points share a single decision function
|
||||
`maybeAlertTenant()`:
|
||||
|
||||
1. `runAICostAlertJob({ tenantId })` — from the dispatcher after
|
||||
every successful `AICostEvent` persist.
|
||||
2. `runAICostAlertCron()` — every 30 minutes; iterates every tenant
|
||||
with a configured quota.
|
||||
|
||||
Source: `apps/worker/src/jobs/ai-cost-alert.ts`.
|
||||
|
||||
## HTTP 402 client handling
|
||||
|
||||
Routes wrap LLM calls and re-throw `AIQuotaExceededError` as
|
||||
`Response.json({ error: "ai-quota-exceeded", reason, … }, { status: 402 })`.
|
||||
For chatbot streams, emit an SSE `event: error` with the same
|
||||
payload and close the stream. The client renders an "Upgrade plan"
|
||||
CTA pointing at the tenant subscription route
|
||||
`/:tenant/settings/subscription`.
|
||||
|
||||
`isAIQuotaExceededError(error)` from `@toolctrl/ai/cost` is the
|
||||
recommended type-guard in route action handlers.
|
||||
|
||||
## Killswitches
|
||||
|
||||
| Env var | Effect |
|
||||
| ------------------------------------ | -------------------------------------------------------------- |
|
||||
| `AI_QUOTA_ENFORCEMENT_ENABLED=false` | Skip every assertion; allow all calls. Production hotfix knob. |
|
||||
| `AI_QUOTA_ALERTS_ENABLED=false` | Skip the alert engine entirely (cron + on-demand). |
|
||||
| `AI_COST_TRACKING_ENABLED=false` | Phase 1 — disables the outbox publisher (no events written). |
|
||||
|
||||
The soft-degradation feature flag can be globally toggled off in the
|
||||
admin feature-flag UI to force every breach into a hard-deny while
|
||||
leaving enforcement on.
|
||||
|
||||
## Audit
|
||||
|
||||
The CLI quota override commands write audit entries for set/reset operations.
|
||||
Future admin UI changes for the same override surface must keep that same
|
||||
audit contract. The `TenantAIQuotaOverride.note` field is the operator's place
|
||||
to reference the originating ticket / sales agreement.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
- **Calls denied despite the dashboard showing room**: the spend
|
||||
cache may have a 60-second lag. The quota check uses the conservative
|
||||
estimate `currentSpend + estimatedCost > limit`; check that
|
||||
`estimateLLMCost` is not picking a huge `maxOutputTokens`.
|
||||
- **Concurrency counter stuck**: TTL is 10 minutes. Force-clear via
|
||||
Redis CLI `DEL shared-store:<namespace>:ai-cost:concurrent:<tenant>`
|
||||
if a long-running operator process leaked a slot.
|
||||
- **No alert fired at 80%**: check `Tenant.aiCostLastAlertedThreshold`.
|
||||
If it equals or exceeds 80 for the current period, the engine
|
||||
considers it already alerted. Reset to `NULL` to re-arm.
|
||||
- **Soft-degrade not firing**: confirm the `ai-quota-soft-degradation`
|
||||
flag is enabled for the tenant AND that `aiDegradeTo` is set on
|
||||
the plan default or override. Without a degrade target the engine
|
||||
hard-denies.
|
||||
|
||||
## Related files
|
||||
|
||||
- `prisma/migrations/20260517150000_05_ai_cost_phase_2_tenant_quota/migration.sql`
|
||||
- `docs/improvement-plan/05-business-features/ai-cost-phase-2.md`
|
||||
- `docs/features/ai-cost-tracking.md` (Phase 1)
|
||||
Reference in new issue
Block a user