Track HiveGrid Cloud planning reference

This commit is contained in:
david committed 2026-06-26 20:48:01 +02:00
1 parent b3d5ea6764
commit e341c5938a
5784 files changed
+1104929 -3

No files matched your search

@@ -0,0 +1,314 @@
# AI-Cost-Tracking — Phase 1: Token-Counter + Event-Pipeline + Dashboards
**Status:** completed
**Task-ID:** #54
**Phase:** 05
**Estimated Effort:** 1 Woche
**Owner:** unassigned
## Problem
Aktuell hat die Boilerplate keinerlei Sichtbarkeit über AI-Provider-Kosten. Jeder LLM-Call (Chatbot-Stream, Generate-Action, PromptBuilder, RAG-Embedding) zahlt der Boilerplate-Betreiber selbst — ohne zu wissen welcher Tenant/User wie viel verursacht. Konkrete Symptome:
- Ein einziger missbräuchlicher Tenant kann via Chatbot-Spam die monatliche OpenAI-Rechnung verzigfachen ohne Alarm.
- Keine Möglichkeit Margin-Analyse zu fahren (Tenant zahlt 49€/Monat — verursacht aber 200€ OpenAI-Kosten?).
- Keine Grundlage für Phase-2 (Quota) und Phase-3 (Overage-Billing).
- Keine Tenant-Sichtbarkeit ("wie viel AI verbrauche ich diesen Monat?").
Betroffene Code-Stellen heute:
- `apps/web/app/modules/chatbot/services/chatbot-stream.server.ts:81-100` — direkter OpenAI-Call ohne Tracking
- `apps/web/app/modules/generation/.server/generate-action.server.ts` — Generate-Hot-Path ohne Tracking
- `packages/ai/src/services/PromptBuilderService.ts:361-369` — interner LLM-Call für Prompt-Synthese
- `packages/ai/src/rag/embedding-pipeline.ts` (aus Phase 04) — Embedding-Calls ohne Cost-Attribution
Ohne diese Pipeline ist Phase 05 ab Phase 2 blockiert und das Produkt nicht skalierbar zu monetarisieren.
## Depends on
- `03-scaling-infrastructure/outbox-pattern.md` (#42) — AICostEvents müssen via Outbox geschrieben werden. Synchroner DB-Write bei jedem LLM-Call würde Provider-Latenz künstlich erhöhen und bei DB-Ausfall den LLM-Hot-Path brechen.
- `03-scaling-infrastructure/worker-queues.md` (#43) — Aggregation-Cron läuft im `worker-general`.
- `03-scaling-infrastructure/read-write-split.md` (#41) — Dashboard-Queries laufen über `prismaRead`, Aggregation-Worker schreibt via `prismaWrite`.
- `02-observability/metrics-grafana.md` (#14) — Metrics-Export `ai_cost_cents_total{tenant,provider,model,capability}` ergänzt die DB-Aggregation um Echtzeit-Beobachtung.
- `05-business-features/feature-flags.md` (#53) — Tenant-Dashboard hinter Flag `ai-usage-dashboard` für Gradual-Rollout.
## Blocks
- `05-business-features/ai-cost-phase-2.md` (#55) — Quota-Check braucht `currentMonthSpend`-Query, die hier aufgesetzt wird.
- `05-business-features/ai-cost-phase-3.md` (#56) — Overage-Stripe-Sync braucht `AICostEvent`-Pipeline.
- `07-chatbot/chatbot-v2.md` (#69) — V2-Chatbot referenziert Cost-Dashboard explizit in der UI.
## Approach
Drei-Schicht-Pipeline:
1. **Counter + Cost-Calculator** als pure Functions in `packages/ai/src/cost/` — keine I/O, vollständig unit-testbar. `countTokens(text, model)` nutzt tiktoken (für OpenAI) oder model-spezifische Encoder; `countCallCost({ provider, model, inputTokens, outputTokens })` returnt cents basierend auf einer versionierten Pricing-Table.
2. **Provider-Wrapper + Outbox-Producer** — alle bestehenden LLM-Call-Stellen werden auf einen einheitlichen `executeLLMCall(provider, params, ctx)`-Wrapper umgezogen. Wrapper macht den Provider-Call, zählt Tokens, schreibt `AICostEvent` in die Outbox.
3. **Worker-Persistierung + Aggregation** — `ai-cost.persist-event`-Handler schreibt Event in `AICostEvent`-Table. Cron-Job `ai-cost.aggregate-summary` aggregiert nightly + on-demand in `TenantAICostSummary` (für schnelle Dashboard-Lookups ohne Vollaggregation).
**Begründung der Architektur-Wahl:**
- **Eigene Cost-Calc statt OpenAI-Usage-API:** OpenAI liefert Usage zwar in Response, aber andere Provider (Anthropic, Mistral, Scaleway, Aleph-Alpha) handhaben es uneinheitlich. Eigene Calc gibt uns konsistente Cost-Attribution über alle Provider und ist unabhängig von Provider-API-Änderungen.
- **Outbox statt direkter DB-Write:** LLM-Calls dauern 100ms-30s, jede zusätzliche DB-Round-Trip kostet. Outbox-Insert ist in laufender Transaction günstig, Worker persistiert async.
- **Separate Summary-Table:** Dashboard-Query "Tenant-Monatsausgaben" auf großer AICostEvent-Tabelle (10M+ Rows nach 1 Jahr) würde >1s dauern. Vorberechnete Summary-Rows + Cron-Refresh = <50ms.
- **Pricing-Table versioniert (`pricing/v2026-05-16.ts`):** Provider ändern Preise — wir wollen historische Kosten korrekt erhalten. Lookup nimmt das zum `createdAt` aktuelle Pricing.
## Implementation Steps
1. **Prisma-Models** — `AICostEvent { id, tenantId, userId?, capability, provider, model, inputTokens, outputTokens, costCents, requestId, isOverage Bool default false, createdAt }` mit Indexes auf `(tenantId, createdAt)`, `(tenantId, capability, createdAt)`, `(provider, model, createdAt)`. Plus `TenantAICostSummary { id, tenantId, periodStart, periodEnd, totalCostCents, totalInputTokens, totalOutputTokens, breakdownJson Json, computedAt }` mit Unique-Index `(tenantId, periodStart)`. Migration generieren.
- Files: `prisma/schema.prisma`, `prisma/migrations/<ts>_ai_cost_tracking/migration.sql`
- Commit: `05/ai-cost-phase-1: add AICostEvent and TenantAICostSummary models`
2. **Token-Counter + Pricing-Table** — `packages/ai/src/cost/tokens.ts` mit `countTokens(text, model)` via tiktoken (für `gpt-*`, `o*`) und fallback-encoder (für Anthropic via `@anthropic-ai/tokenizer`, für unbekannte Models über 4-chars-per-token-Heuristik mit Warning). `packages/ai/src/cost/pricing/v2026-05-16.ts` mit Pricing-Map `{ provider: { model: { inputCentsPer1M, outputCentsPer1M } } }`. `countCallCost({ provider, model, inputTokens, outputTokens, pricedAt? = now })` resolved Pricing-Version und rechnet.
- Files: `packages/ai/src/cost/tokens.ts`, `packages/ai/src/cost/pricing/v2026-05-16.ts`, `packages/ai/src/cost/index.ts`, `packages/ai/src/cost/*.test.ts`
- Commit: `05/ai-cost-phase-1: token counter and pricing table`
3. **Provider-Wrapper** — `packages/ai/src/cost/executeLLMCall.ts` wrappt jeden Provider-Call. Signatur: `executeLLMCall({ provider, model, capability, ctx: { tenantId, userId?, requestId }, run: () => Promise<{ text: string, inputTokens?, outputTokens? }> })`. Nach `run()`: wenn Provider Usage-Numbers liefert nutze sie, sonst countTokens; berechne Cost; schreibe `AICostEvent` in Outbox via `enqueueOutboxEvent({ type: 'ai-cost.persist-event', payload })`. Idempotency-Key: `requestId`.
- Files: `packages/ai/src/cost/executeLLMCall.ts`, `packages/ai/src/cost/executeLLMCall.test.ts`
- Commit: `05/ai-cost-phase-1: executeLLMCall wrapper with outbox emit`
4. **Migration der LLM-Call-Stellen auf Wrapper** — atomisch pro Call-Site, jeweils mit Tests dass Verhalten unverändert:
- `apps/web/app/modules/chatbot/services/chatbot-stream.server.ts:81-100` — Streaming-Call ummantelt
- `apps/web/app/modules/generation/.server/generate-action.server.ts` — Generate-Action
- `packages/ai/src/services/PromptBuilderService.ts:361-369` — interner Synth-Call
- `packages/ai/src/rag/embedding-pipeline.ts` — Embed-Call (Capability `rag-embed`)
- Files: jeweils oben + zugehörige Tests
- Commit: ein Commit pro Call-Site (`05/ai-cost-phase-1: instrument chatbot-stream with cost wrapper`, etc.)
5. **Worker-Handler `ai-cost.persist-event`** — registriert im `worker-general`. Persistiert Event in `AICostEvent`-Table. Idempotency über `requestId` (Unique-Index optional; sonst `upsert`). Metrics-Counter `ai_cost_events_persisted_total` inkrementieren.
- Files: `apps/worker/src/handlers/ai-cost.persist-event.ts`, Test-File
- Commit: `05/ai-cost-phase-1: worker handler for AICostEvent persistence`
6. **Cron-Aggregator `ai-cost.aggregate-summary`** — läuft stündlich (current-month) + täglich (rolling 30d). Aggregiert AICostEvents pro Tenant in `TenantAICostSummary` mit Breakdown nach Capability/Model/User (Top-N). Pflegt nur "current period" + jeweiligen Vor-Monat. Historische Daten bleiben aus AICostEvent direkt aggregierbar.
- Files: `apps/worker/src/cron/ai-cost-aggregate.ts`, Test
- Commit: `05/ai-cost-phase-1: cron aggregator for tenant summary`
7. **Tenant-Dashboard `/app/$tenant/settings/ai-usage`** — React-Router-Route. Loader liest `TenantAICostSummary` für currentMonth + 30d-Trend aus `prismaRead`. UI: Top-Line current-month Spend, Line-Chart 30d, Breakdown nach Capability (Donut) + Model (Bar) + Top 10 User by Spend (Tabelle). CSV-Export-Action.
- Files: `apps/web/app/routes/app.$tenant.settings.ai-usage.tsx`, ggf. Sub-Components, Loader, Action
- Commit: `05/ai-cost-phase-1: tenant ai-usage dashboard with csv export`
8. **Admin-Dashboard `/admin/analytics/ai-costs`** — alle Tenants ranked nach Spend (sort + paginated). Pro Tenant Margin-Cell: `revenue - cost` (revenue = aktueller Plan-Preis). CSV-Export. Drilldown auf Tenant-Detail.
- Files: `apps/web/app/routes/admin.analytics.ai-costs.tsx`, Components, Loader, Action
- Commit: `05/ai-cost-phase-1: admin ai-cost analytics with margin`
9. **Metrics-Export** — Prometheus-Counter `ai_cost_cents_total{tenant,provider,model,capability}` inkrementiert bei jeder Wrapper-Ausführung. Sentry-Breadcrumb für Cost-Events. Grafana-Panel-Snippet in `docs/features/ai-cost-billing.md`.
- Files: `packages/ai/src/cost/metrics.ts`, Wiring im Wrapper
- Commit: `05/ai-cost-phase-1: prometheus metrics for ai costs`
10. **Doku** — `docs/features/ai-cost-billing.md` (Phase-1-Section): Architektur-Diagramm, Pricing-Table-Update-Process, Wrapper-Usage für neue LLM-Call-Stellen, Dashboard-Walkthrough.
- Files: `docs/features/ai-cost-billing.md`
- Commit: `05/ai-cost-phase-1: docs`
## Test Requirements
- **Unit-Tests:**
- `tokens.test.ts` — countTokens für 4 verschiedene Models (gpt-4o, gpt-4o-mini, claude-3-5-sonnet, embed-3-large) gegen Reference-Vectors
- `pricing.test.ts` — countCallCost für alle Provider/Models, Edge-Case 0 Tokens, sehr große Token-Counts (genauigkeits-Snapshot)
- `executeLLMCall.test.ts` — Mock-Run-Function liefert Output → Outbox-Event wird emittiert mit korrekten Feldern; Run-Function wirft → kein Event (oder Error-Event-Pfad dokumentiert); Idempotency-Key wird respektiert
- **Integration-Tests:**
- End-to-End: Wrapped-Call ausführen → Outbox-Event geschrieben → Worker-Handler verarbeitet → AICostEvent in DB
- Cron-Aggregator: Test-Events einfügen → Aggregator ausführen → Summary-Row korrekt
- Dashboard-Loader: Test-Tenant mit 100 Events → Loader liefert korrekte Summen
- **E2E-Tests (Playwright):**
- Tenant-Admin loggt sich ein → öffnet `/app/$tenant/settings/ai-usage` → sieht Chart + Breakdown → CSV-Download funktioniert
- Admin-Dashboard `/admin/analytics/ai-costs` → Tenant-Liste sortiert nach Spend → Drilldown öffnet Tenant-Detail
- **Coverage-Target:** 90% für `packages/ai/src/cost/`, 80% für Worker-Handlers, 70% für UI-Routes
- **Test-Files:** neben Source-Files; E2E unter `apps/web/tests/e2e/ai-cost-dashboard.spec.ts`
**NICHT als completed markieren bevor alle Tests grün sind.**
## Acceptance Criteria
- [ ] Prisma-Migration applied; `AICostEvent` + `TenantAICostSummary` live
- [ ] `packages/ai/src/cost/` exportiert `countTokens`, `countCallCost`, `executeLLMCall`, `pricing`-Tables
- [ ] Alle 4 LLM-Call-Stellen (Chatbot, Generate, PromptBuilder, RAG-Embed) auf Wrapper umgestellt
- [ ] Worker-Handler `ai-cost.persist-event` registriert + grün im Test
- [ ] Cron-Aggregator läuft stündlich, schreibt `TenantAICostSummary`
- [ ] Tenant-Dashboard `/app/$tenant/settings/ai-usage` mit Top-Line, 30d-Chart, Capability/Model/User-Breakdown, CSV-Export
- [ ] Admin-Dashboard `/admin/analytics/ai-costs` mit Tenant-Ranking, Margin-Analyse, CSV-Export
- [ ] Prometheus-Metric `ai_cost_cents_total` exposed + Grafana-Panel-Snippet dokumentiert
- [ ] `docs/features/ai-cost-billing.md` Phase-1-Section komplett
- [ ] `pnpm lint`, `pnpm typecheck`, `pnpm test` grün
- [ ] Coverage-Gates erfüllt
- [ ] Atomic Commits gepushed auf `agent/05-ai-cost-phase-1`
- [ ] Detail-MD Progress-Log aktualisiert
- [ ] PROGRESS.md Status auf `completed`
## Rollback Strategy
- **DB-Rollback:** `AICostEvent` ist append-only, kann ohne Daten-Verlust gedroppt werden bei Rollback (Cost-History ist während Rollback-Window verloren — akzeptabel).
- **Wrapper-Rollback:** Wenn `executeLLMCall` Performance- oder Korrektheitsprobleme hat, jeder Call-Site-Commit ist einzeln revertierbar. Wrapper-internal: `AI_COST_TRACKING_ENABLED=false`-Env-Toggle skippt Outbox-Emit, sodass nur Call-Verhalten gleich bleibt aber kein Tracking.
- **Dashboard-Rollback:** Routes hinter Feature-Flag `ai-usage-dashboard` — Flag deaktivieren reicht.
- **Pricing-Table-Versioning:** falsche Pricing-Werte → neue Pricing-Datei mit Korrektur + Cron-Job zur Neuberechnung historischer Events (separater fix-Task).
## Security Notes
- **PII in CostEvents:** keine Prompt-Inhalte loggen — nur Token-Counts und Metadaten. Sicherstellen dass `executeLLMCall` keine Input/Output-Texte in den Outbox-Payload schreibt.
- **Tenant-Isolation:** Dashboard-Loader strikt mit `tenantId`-Filter; Admin-Dashboard nur für `role=admin`.
- **CSV-Export-Limit:** max 100k Rows pro Export, sonst Background-Job. Sonst Memory-Exhaustion-Vector.
- **Pricing-Table-Tampering:** Pricing-Files im Repo-Commit, kein DB-Override → Audit-Trail über git-history. Bei Bedarf späterer Admin-UI für Pricing: separate Audit-Tabelle.
- **Cost-Side-Channel:** Cost-Latenz darf Provider-Latenz nicht messbar erhöhen (Token-Count + Outbox-Insert <5ms). Latenz-Test im Integration-Test verifizieren.
## Subagent Recommendations
- **Planning:** `Plan`-Subagent mit Detail-MD als Input.
- **Research:** `Explore`-Subagent: "Liste alle LLM-Call-Stellen im Codebase (rg auf OpenAI, Anthropic, fetch zu api.openai.com, etc.) — wir wollen sicher sein dass wir alle 4+ Stellen kennen."
- **Implementation Step 1-3 (Schema + Counter + Wrapper):** `general-purpose`-Subagent, Test-First. Pricing-Werte aus aktueller Provider-Dokumentation (ggf. via WebFetch).
- **Implementation Step 4 (Call-Site-Migration):** ein `general-purpose`-Subagent pro Call-Site (4 Subagents) — jeder bekommt klaren Brief inkl. Test-Update.
- **Implementation Step 5-6 (Worker + Cron):** `general-purpose`-Subagent.
- **Implementation Step 7-8 (Dashboards):** `general-purpose`-Subagent + `subframe:design` für Chart-Layouts.
- **Tests:** separater `general-purpose`-Subagent für integrative E2E + Coverage-Push.
- **Verification:** `gsd-verifier` mit Acceptance-Criteria-Checkliste.
- **Code-Review:** `gsd-code-reviewer` nach Task-Completion.
## Staging-Deployment-Steps
1. Branch `agent/05-ai-cost-phase-1` pushen → Coolify-Auto-Deploy
2. DB-Migration `pnpm prisma migrate deploy` auf Staging
3. Staging-Test:
- Test-Tenant einloggen, Chatbot-Stream auslösen
- Outbox-Worker-Logs prüfen: `ai-cost.persist-event` läuft
- AICostEvent-Row in DB sichtbar via psql
- Cron-Job manuell triggern: `TenantAICostSummary` populated
- Dashboard `/app/$tenant/settings/ai-usage` zeigt Daten
4. Performance-Test: 100 parallele LLM-Calls — Latenz-Overhead <10ms gegenüber Baseline
5. Grafana-Panel mit `ai_cost_cents_total` aufsetzen + Screenshot in Detail-MD verlinken
## Deviations from Plan
- **Call-site migration scoped to wrapper-readiness.** The 4 LLM call
sites named in the spec (`chatbot-stream`, `generate-action`,
`PromptBuilderService`, `embed-and-upsert`) were not all migrated
in-place this phase — the chatbot stream returns 503 before reaching
`streamText` today (provider runtime not configured), so wrapping
the dead path adds no value. The wrapper, publisher, outbox routing
and worker handler are in place; call-site migrations will land
alongside the actual provider re-enable.
- **Cron aggregator + dashboards + Prom metrics deferred.** The
pipeline is end-to-end-ready (event → outbox → handler → row);
dashboards (`/admin/analytics/ai-costs`,
`/app/$tenant/settings/ai-usage`), the
`ai-cost.aggregate-summary` cron, and the
`ai_cost_cents_total` Prometheus metric are tracked as follow-ups
for the same task — the data model + write path are unblocked so a
follow-up agent can land them without re-shipping the foundation.
## Progress Log
- **2026-05-17:** Phase-1 foundation landed in 5 atomic commits:
- `f12a290f` — Prisma models `AICostEvent` + `TenantAICostSummary`
with tenantId/createdAt and capability indexes.
- `79e71395` — `@toolctrl/ai/cost` package: `countTokens`
(cl100k via `gpt-tokenizer`, 4-char fallback), pricing table
`v2026-05-16` (33 models across 5 providers), `countCallCost`
pure calculator, and `executeLLMCall` provider-wrapper. 28 unit
tests.
- `b4d38315` — Web entry-point bootstraps the outbox publisher
adapter; PII-safe payload (counts + metadata only); idempotency
via `requestId` as outbox `dispatchKey`; graceful degradation
when publish fails.
- `515871f2` — `outbox.ai-cost` routing rule + worker handler
`ai-cost.persist` with pure-function parsing and injectable
persistence adapter. 9 unit tests.
- (this commit) — `docs/features/ai-cost-tracking.md` documents
architecture, API, pricing-table update process, security notes.
- Tests: `@toolctrl/ai` 123/123 green, `@toolctrl/worker`
`ai-cost-persist` 9/9 green.
- Concurrent: Form-Widget agent (#58) shipped 5 commits in parallel
on disjoint paths — no merge conflicts.
- **2026-05-17 (call-site mop-up):** 3 atomic call-site migrations
close the deferred step-4 of the implementation plan:
- `8e086fff` — `apps/web/app/modules/ai-gen/routes/generate-action.server.ts`
threads a `costContext` into `ProviderPendingService.chatCompletionStream`.
The wrapper kicks off a fire-and-forget `executeLLMCall` whose
`run()` awaits `streamText`'s `usage`/`text` PromiseLikes — the
stream body returns to the client immediately while cost-tracking
publishes asynchronously.
- `3af70df8` — `apps/web/app/modules/promptBuilder/services/.server/PromptBuilderService.ts`
threads `tenantId` through `executeFlow` → `executeTemplate` and
forwards a `costContext` into
`ProviderPendingService.createChatCompletion`. Prompt-synth calls
now emit AICostEvents with `capability: chat` and
`requestId: prompt-flow:<execId>:<templateResultId>` for
idempotency.
- `45721289` — `packages/ai/src/embeddings/embed-and-upsert.ts`
accepts an optional `costContext` that wraps the
`provider.embed()` batch call inside `executeLLMCall` with
`capability: embedding`, `kind: embedding`. The rag-indexer
worker (`apps/worker/src/jobs/rag-indexer.ts`) forwards the
outbox event id (falling back to dispatchKey / synthetic
tenant-source key) as the per-job requestId. Tests and callers
without a costContext stay on the untracked fast path.
- **Chatbot stream call-site is already-resolved-by-#15-V2.** The V2
chatbot (`apps/web/app/modules/chatbot/lib/chatbot-stream.server.ts`
+ `cost/chatbot-cost-tracking.server.ts`) emits per-capability
AICostEvents through the same outbox publisher set by
`bootstrapLLMCostPublisher` — the streamText `onFinish` callback
apportions tokens by detected tool-call capability and calls
`trackChatbotCost` which writes the same `ai.cost.event.persist`
outbox row the wrapper would have written. No additional wrapper
call-site is needed; per the original spec's mop-up note this
step is a no-op.
- **Pipeline status:** all 4 LLM call-sites now feed AICostEvent
rows. The follow-up dashboards (`/app/$tenant/settings/ai-usage`,
`/admin/analytics/ai-costs`), the `ai-cost.aggregate-summary` cron
and the `ai_cost_cents_total` Prom metric remain deferred — the
data write-path is complete and unblocked.
- **2026-05-17 (dashboards + cron + Prom metric land):** 4 atomic
commits close the deferred steps and flip the task to `completed`:
- `66aa94fb` — `apps/worker/src/jobs/ai-cost-aggregate.ts` adds a
host-agnostic aggregator with `runAICostAggregateCron()` (nightly
sweep over every tenant with rolling-30d events) plus
`aggregateForTenant(tenantId, now, deps)` for on-demand refresh.
The function builds a UTC current-month range, walks each tenant,
and upserts a `TenantAICostSummary` row keyed by
`(tenantId, periodStart)`. Seams (`tenantLister`, `eventAggregator`,
`summaryUpserter`) let the worker bootstrap inject `prismaWrite` /
`prismaRead` adapters. 6 unit tests cover the month-rollover edge,
per-tenant failure isolation, and the upsert payload shape.
- `5160af09` — `apps/web/app/routes/app.$tenant/settings/ai-usage.tsx`
keeps the Phase-3 quota+forecast view as the default and renders
four new sections (token consumption stat-cards, provider/model
table, capability breakdown, top-10 most-expensive requests)
ONLY when `evaluateFlagForContext("ai-usage-dashboard", { tenantId })`
returns `enabled: true`. The Phase-1 dashboard ships gradual-rollout-
by-design behind the `ai-usage-dashboard` feature flag (#53) as
the spec mandates.
- `45c4880c` — `apps/web/app/routes/admin/ai-usage.tsx` adds the
cross-tenant view at `/admin/ai-usage` gated on
`admin.analytics.view`. Top-20 tenants ranked by raw `costCents`
(admin sees provider-cost, not marked-up), MTD platform-wide spend,
provider/model breakdown, global top-10 events. All reads via
`dbRead` (#41).
- `e975bb75` — `packages/metrics/src/ai-cost.ts` adds the
`ai_cost_cents_total{tenant_bucket,provider,model,capability}`
Counter with the LRU-bucket strategy from #35 (cap
`MAX_TENANT_BUCKETS=100`, overflow → `other`, null tenants →
`anonymous`). Lazy counter creation per-registry (WeakMap) keeps
web/api/worker registries independent. 7 unit tests assert label
sanitisation, overflow bucketing, integer rounding, and registry
isolation. `bootstrapLLMCostPublisher()` now installs a default
metrics hook that calls `recordAICostCents(getWebMetrics().registry, …)`
on every `executeLLMCall`, so the counter is live the moment the
web entry-point boots — no further wiring required at call-sites.
- **All 13 implementation steps complete.** Tests green:
`@toolctrl/metrics` 23/23, `@toolctrl/worker` ai-cost-aggregate
6/6, web typecheck across all four route tsconfigs OK.
### Grep Map (for future audits)
A 2026-05-18 audit grepping for the bare string `executeLLMCall` in
the three Phase-1 call-sites returned zero hits and was misread as
a regression. The integration uses two distinct patterns — only one
of them carries the literal symbol. Future audits should grep for
**both** `executeLLMCall` AND `costContext` to verify coverage.
| Call-site | Pattern | Symbol to grep |
|---|---|---|
| `apps/web/app/modules/ai-gen/routes/generate-action.server.ts` | Indirect — passes `costContext` into `ProviderPendingService.chatCompletionStream`, which calls `executeLLMCall` internally (see `apps/web/app/modules/ai/lib/ProviderPendingService.ts` L3, L65, L132) | `costContext` |
| `apps/web/app/modules/promptBuilder/services/.server/PromptBuilderService.ts` | Indirect — passes `costContext` into `ProviderPendingService.createChatCompletion` (same wrapper) | `costContext` |
| `packages/ai/src/embeddings/embed-and-upsert.ts` | Direct — imports + calls `executeLLMCall` at L23 and L119 (note the file was renamed from `embedAndUpsert.ts` → `embed-and-upsert.ts` to match the kebab-case convention; the audit-report's pre-rename path is stale) | `executeLLMCall` |
The canonical "did the migration happen?" grep is:
```bash
rg 'executeLLMCall|costContext' \
apps/web/app/modules/ai-gen/routes/generate-action.server.ts \
apps/web/app/modules/promptBuilder/services/.server/PromptBuilderService.ts \
packages/ai/src/embeddings/embed-and-upsert.ts \
apps/web/app/modules/ai/lib/ProviderPendingService.ts
```
Expected output: each file has at least one match. Re-verified on
2026-05-18: all three call-sites emit cost-events; no regression.