AI Cost Tracking — Phase 1

Status: Phase-1 (foundation) shipped. Phases 2 (quota enforcement) and 3 (Stripe overage billing) build on the pipeline below.

Architecture

Three-layer pipeline that attributes every LLM call to a tenant + user and persists the cost asynchronously via the transactional outbox:

LLM call-site
    │
    ▼
executeLLMCall          ← packages/ai/src/cost/executeLLMCall.ts
  ├─ countTokens()      ← packages/ai/src/cost/tokens.ts
  ├─ countCallCost()    ← packages/ai/src/cost/cost.ts
  └─ publishOutboxEvent ← @toolctrl/db/outbox
                        │
                        ▼
                 OutboxEvent (PENDING)
                        │  dispatcher polls (1s)
                        ▼
                 BullMQ outbox.ai-cost queue
                        │
                        ▼
                 ai-cost.persist handler  ← apps/worker/src/jobs/ai-cost-persist.ts
                                                bootstrapped by worker-general
                        │
                        ▼
                 AICostEvent (append-only row)

apps/worker/src/index.ts registers the outbox.ai-cost subscriber on worker-general when Redis is configured. Set AI_COST_PERSIST_ENABLED=false to pause persistence during incident response; leave it enabled in staging/production so the outbox does not report cost events as dispatched while no consumer writes AICostEvent rows.

The apps/worker/src/jobs/ai-cost-aggregate.ts rollup helper upserts TenantAICostSummary rows so dashboard top-line lookups stay O(1). apps/worker/src/index.ts registers the dedicated toolctrl.ai-cost-aggregate scheduler/worker on the general worker group when Redis is configured. Set AI_COST_AGGREGATE_ENABLED=false or 0 to disable it during incident response.

Why a wrapper instead of provider-native usage?

Multi-provider consistency. OpenAI, Anthropic, Mistral and Scaleway expose usage telemetry in slightly different shapes. The wrapper normalises to one schema.
Hot-path latency. Synchronous cost-write would block the LLM stream. The outbox lets us commit the cost-event in the surrounding transaction and persist async.
PII hygiene. The wrapper closes over the prompt/response text; only counts + metadata leak into the outbox payload (see "Security" below).

API

import { executeLLMCall } from "@toolctrl/ai/cost";

const result = await executeLLMCall({
  provider: "openai",
  model: "gpt-4o-mini",
  capability: "chat",
  ctx: { tenantId, userId, requestId: crypto.randomUUID() },
  run: async () => {
    const response = await openai.chat.completions.create({...});
    return {
      output: response.choices[0].message.content ?? "",
      usage: {
        inputTokens: response.usage?.prompt_tokens,
        outputTokens: response.usage?.completion_tokens,
      },
    };
  },
});

When the provider does not return usage numbers, populate inputForCount / outputForCount so the wrapper falls back to countTokens.

Pricing tables

Pricing is checked-in (packages/ai/src/cost/pricing/<version>.ts) and immutable once shipped. Adding a price change:

Copy the latest v<YYYY-MM-DD>.ts to a new file with today's date.
Edit the rows that changed.
Update LATEST_PRICING_VERSION in pricing/index.ts.
Append the new entry to PRICING_VERSIONS.

Historical events keep referring to their original pricing version via AICostEvent.pricingVersion — so a 6-month-old event still renders with the cents it was originally billed at.

Phase-1 ships covers OpenAI (10 models), Anthropic (6), Mistral (7), Scaleway (5) and Aleph-Alpha (5).

Operational toggles

Env	Default	Effect
`AI_COST_TRACKING_ENABLED=false`	enabled	Skip outbox publish entirely; LLM hot-path keeps working

Idempotency

Every event carries a requestId which the publisher writes as the outbox dispatchKey AND which is uniquely indexed on AICostEvent. Re-deliveries collapse to a no-op via the existing outbox dedup-on-conflict path.

Tenant isolation

AICostEvent.tenantId is nullable to support admin-only chats; rows cascade-delete with the Tenant.
Dashboard loaders strictly filter by tenantId.
Admin analytics endpoints are gated behind the admin role.

Security

The outbox payload contains counts + metadata only. Prompt and response text never leave the wrapper closure.
CSV-exports cap at 100k rows; larger exports go through the background-job queue (planned).
Pricing tables are git-versioned — no DB override path.
Cost wrapper adds <5ms in the LLM hot-path (tokenizer is sync; outbox insert is one round-trip co-located with existing tx).

Spec

Full spec, acceptance criteria and migration plan: docs/improvement-plan/05-business-features/ai-cost-phase-1.md.

4.9 KiB Raw Blame History