138 lines
4.9 KiB
Markdown
138 lines
4.9 KiB
Markdown
# AI Cost Tracking — Phase 1
|
|
|
|
**Status:** Phase-1 (foundation) shipped. Phases 2 (quota enforcement)
|
|
and 3 (Stripe overage billing) build on the pipeline below.
|
|
|
|
## Architecture
|
|
|
|
Three-layer pipeline that attributes every LLM call to a tenant +
|
|
user and persists the cost asynchronously via the transactional
|
|
outbox:
|
|
|
|
```
|
|
LLM call-site
|
|
│
|
|
▼
|
|
executeLLMCall ← packages/ai/src/cost/executeLLMCall.ts
|
|
├─ countTokens() ← packages/ai/src/cost/tokens.ts
|
|
├─ countCallCost() ← packages/ai/src/cost/cost.ts
|
|
└─ publishOutboxEvent ← @toolctrl/db/outbox
|
|
│
|
|
▼
|
|
OutboxEvent (PENDING)
|
|
│ dispatcher polls (1s)
|
|
▼
|
|
BullMQ outbox.ai-cost queue
|
|
│
|
|
▼
|
|
ai-cost.persist handler ← apps/worker/src/jobs/ai-cost-persist.ts
|
|
bootstrapped by worker-general
|
|
│
|
|
▼
|
|
AICostEvent (append-only row)
|
|
```
|
|
|
|
`apps/worker/src/index.ts` registers the `outbox.ai-cost` subscriber on
|
|
`worker-general` when Redis is configured. Set
|
|
`AI_COST_PERSIST_ENABLED=false` to pause persistence during incident response;
|
|
leave it enabled in staging/production so the outbox does not report cost
|
|
events as dispatched while no consumer writes `AICostEvent` rows.
|
|
|
|
The `apps/worker/src/jobs/ai-cost-aggregate.ts` rollup helper upserts
|
|
`TenantAICostSummary` rows so dashboard top-line lookups stay O(1).
|
|
`apps/worker/src/index.ts` registers the dedicated
|
|
`toolctrl.ai-cost-aggregate` scheduler/worker on the `general` worker
|
|
group when Redis is configured. Set `AI_COST_AGGREGATE_ENABLED=false`
|
|
or `0` to disable it during incident response.
|
|
|
|
## Why a wrapper instead of provider-native usage?
|
|
|
|
- **Multi-provider consistency.** OpenAI, Anthropic, Mistral and
|
|
Scaleway expose usage telemetry in slightly different shapes. The
|
|
wrapper normalises to one schema.
|
|
- **Hot-path latency.** Synchronous cost-write would block the LLM
|
|
stream. The outbox lets us commit the cost-event in the surrounding
|
|
transaction and persist async.
|
|
- **PII hygiene.** The wrapper closes over the prompt/response text;
|
|
only counts + metadata leak into the outbox payload (see "Security"
|
|
below).
|
|
|
|
## API
|
|
|
|
```ts
|
|
import { executeLLMCall } from "@toolctrl/ai/cost";
|
|
|
|
const result = await executeLLMCall({
|
|
provider: "openai",
|
|
model: "gpt-4o-mini",
|
|
capability: "chat",
|
|
ctx: { tenantId, userId, requestId: crypto.randomUUID() },
|
|
run: async () => {
|
|
const response = await openai.chat.completions.create({...});
|
|
return {
|
|
output: response.choices[0].message.content ?? "",
|
|
usage: {
|
|
inputTokens: response.usage?.prompt_tokens,
|
|
outputTokens: response.usage?.completion_tokens,
|
|
},
|
|
};
|
|
},
|
|
});
|
|
```
|
|
|
|
When the provider does not return usage numbers, populate
|
|
`inputForCount` / `outputForCount` so the wrapper falls back to
|
|
`countTokens`.
|
|
|
|
## Pricing tables
|
|
|
|
Pricing is checked-in (`packages/ai/src/cost/pricing/<version>.ts`)
|
|
and **immutable once shipped**. Adding a price change:
|
|
|
|
1. Copy the latest `v<YYYY-MM-DD>.ts` to a new file with today's date.
|
|
2. Edit the rows that changed.
|
|
3. Update `LATEST_PRICING_VERSION` in `pricing/index.ts`.
|
|
4. Append the new entry to `PRICING_VERSIONS`.
|
|
|
|
Historical events keep referring to their original pricing version
|
|
via `AICostEvent.pricingVersion` — so a 6-month-old event still
|
|
renders with the cents it was originally billed at.
|
|
|
|
Phase-1 ships covers OpenAI (10 models), Anthropic (6), Mistral (7),
|
|
Scaleway (5) and Aleph-Alpha (5).
|
|
|
|
## Operational toggles
|
|
|
|
| Env | Default | Effect |
|
|
|---|---|---|
|
|
| `AI_COST_TRACKING_ENABLED=false` | enabled | Skip outbox publish entirely; LLM hot-path keeps working |
|
|
|
|
## Idempotency
|
|
|
|
Every event carries a `requestId` which the publisher writes as the
|
|
outbox `dispatchKey` AND which is uniquely indexed on `AICostEvent`.
|
|
Re-deliveries collapse to a no-op via the existing outbox
|
|
dedup-on-conflict path.
|
|
|
|
## Tenant isolation
|
|
|
|
- `AICostEvent.tenantId` is nullable to support admin-only chats; rows
|
|
cascade-delete with the Tenant.
|
|
- Dashboard loaders strictly filter by `tenantId`.
|
|
- Admin analytics endpoints are gated behind the `admin` role.
|
|
|
|
## Security
|
|
|
|
- The outbox payload contains **counts + metadata only**. Prompt and
|
|
response text never leave the wrapper closure.
|
|
- CSV-exports cap at 100k rows; larger exports go through the
|
|
background-job queue (planned).
|
|
- Pricing tables are git-versioned — no DB override path.
|
|
- Cost wrapper adds <5ms in the LLM hot-path (tokenizer is sync;
|
|
outbox insert is one round-trip co-located with existing tx).
|
|
|
|
## Spec
|
|
|
|
Full spec, acceptance criteria and migration plan:
|
|
[`docs/improvement-plan/05-business-features/ai-cost-phase-1.md`](../improvement-plan/05-business-features/ai-cost-phase-1.md).
|