# AI-Cost-Tracking — Phase 1: Token-Counter + Event-Pipeline + Dashboards **Status:** completed **Task-ID:** #54 **Phase:** 05 **Estimated Effort:** 1 Woche **Owner:** unassigned ## Problem Aktuell hat die Boilerplate keinerlei Sichtbarkeit über AI-Provider-Kosten. Jeder LLM-Call (Chatbot-Stream, Generate-Action, PromptBuilder, RAG-Embedding) zahlt der Boilerplate-Betreiber selbst — ohne zu wissen welcher Tenant/User wie viel verursacht. Konkrete Symptome: - Ein einziger missbräuchlicher Tenant kann via Chatbot-Spam die monatliche OpenAI-Rechnung verzigfachen ohne Alarm. - Keine Möglichkeit Margin-Analyse zu fahren (Tenant zahlt 49€/Monat — verursacht aber 200€ OpenAI-Kosten?). - Keine Grundlage für Phase-2 (Quota) und Phase-3 (Overage-Billing). - Keine Tenant-Sichtbarkeit ("wie viel AI verbrauche ich diesen Monat?"). Betroffene Code-Stellen heute: - `apps/web/app/modules/chatbot/services/chatbot-stream.server.ts:81-100` — direkter OpenAI-Call ohne Tracking - `apps/web/app/modules/generation/.server/generate-action.server.ts` — Generate-Hot-Path ohne Tracking - `packages/ai/src/services/PromptBuilderService.ts:361-369` — interner LLM-Call für Prompt-Synthese - `packages/ai/src/rag/embedding-pipeline.ts` (aus Phase 04) — Embedding-Calls ohne Cost-Attribution Ohne diese Pipeline ist Phase 05 ab Phase 2 blockiert und das Produkt nicht skalierbar zu monetarisieren. ## Depends on - `03-scaling-infrastructure/outbox-pattern.md` (#42) — AICostEvents müssen via Outbox geschrieben werden. Synchroner DB-Write bei jedem LLM-Call würde Provider-Latenz künstlich erhöhen und bei DB-Ausfall den LLM-Hot-Path brechen. - `03-scaling-infrastructure/worker-queues.md` (#43) — Aggregation-Cron läuft im `worker-general`. - `03-scaling-infrastructure/read-write-split.md` (#41) — Dashboard-Queries laufen über `prismaRead`, Aggregation-Worker schreibt via `prismaWrite`. - `02-observability/metrics-grafana.md` (#14) — Metrics-Export `ai_cost_cents_total{tenant,provider,model,capability}` ergänzt die DB-Aggregation um Echtzeit-Beobachtung. - `05-business-features/feature-flags.md` (#53) — Tenant-Dashboard hinter Flag `ai-usage-dashboard` für Gradual-Rollout. ## Blocks - `05-business-features/ai-cost-phase-2.md` (#55) — Quota-Check braucht `currentMonthSpend`-Query, die hier aufgesetzt wird. - `05-business-features/ai-cost-phase-3.md` (#56) — Overage-Stripe-Sync braucht `AICostEvent`-Pipeline. - `07-chatbot/chatbot-v2.md` (#69) — V2-Chatbot referenziert Cost-Dashboard explizit in der UI. ## Approach Drei-Schicht-Pipeline: 1. **Counter + Cost-Calculator** als pure Functions in `packages/ai/src/cost/` — keine I/O, vollständig unit-testbar. `countTokens(text, model)` nutzt tiktoken (für OpenAI) oder model-spezifische Encoder; `countCallCost({ provider, model, inputTokens, outputTokens })` returnt cents basierend auf einer versionierten Pricing-Table. 2. **Provider-Wrapper + Outbox-Producer** — alle bestehenden LLM-Call-Stellen werden auf einen einheitlichen `executeLLMCall(provider, params, ctx)`-Wrapper umgezogen. Wrapper macht den Provider-Call, zählt Tokens, schreibt `AICostEvent` in die Outbox. 3. **Worker-Persistierung + Aggregation** — `ai-cost.persist-event`-Handler schreibt Event in `AICostEvent`-Table. Cron-Job `ai-cost.aggregate-summary` aggregiert nightly + on-demand in `TenantAICostSummary` (für schnelle Dashboard-Lookups ohne Vollaggregation). **Begründung der Architektur-Wahl:** - **Eigene Cost-Calc statt OpenAI-Usage-API:** OpenAI liefert Usage zwar in Response, aber andere Provider (Anthropic, Mistral, Scaleway, Aleph-Alpha) handhaben es uneinheitlich. Eigene Calc gibt uns konsistente Cost-Attribution über alle Provider und ist unabhängig von Provider-API-Änderungen. - **Outbox statt direkter DB-Write:** LLM-Calls dauern 100ms-30s, jede zusätzliche DB-Round-Trip kostet. Outbox-Insert ist in laufender Transaction günstig, Worker persistiert async. - **Separate Summary-Table:** Dashboard-Query "Tenant-Monatsausgaben" auf großer AICostEvent-Tabelle (10M+ Rows nach 1 Jahr) würde >1s dauern. Vorberechnete Summary-Rows + Cron-Refresh = <50ms. - **Pricing-Table versioniert (`pricing/v2026-05-16.ts`):** Provider ändern Preise — wir wollen historische Kosten korrekt erhalten. Lookup nimmt das zum `createdAt` aktuelle Pricing. ## Implementation Steps 1. **Prisma-Models** — `AICostEvent { id, tenantId, userId?, capability, provider, model, inputTokens, outputTokens, costCents, requestId, isOverage Bool default false, createdAt }` mit Indexes auf `(tenantId, createdAt)`, `(tenantId, capability, createdAt)`, `(provider, model, createdAt)`. Plus `TenantAICostSummary { id, tenantId, periodStart, periodEnd, totalCostCents, totalInputTokens, totalOutputTokens, breakdownJson Json, computedAt }` mit Unique-Index `(tenantId, periodStart)`. Migration generieren. - Files: `prisma/schema.prisma`, `prisma/migrations/_ai_cost_tracking/migration.sql` - Commit: `05/ai-cost-phase-1: add AICostEvent and TenantAICostSummary models` 2. **Token-Counter + Pricing-Table** — `packages/ai/src/cost/tokens.ts` mit `countTokens(text, model)` via tiktoken (für `gpt-*`, `o*`) und fallback-encoder (für Anthropic via `@anthropic-ai/tokenizer`, für unbekannte Models über 4-chars-per-token-Heuristik mit Warning). `packages/ai/src/cost/pricing/v2026-05-16.ts` mit Pricing-Map `{ provider: { model: { inputCentsPer1M, outputCentsPer1M } } }`. `countCallCost({ provider, model, inputTokens, outputTokens, pricedAt? = now })` resolved Pricing-Version und rechnet. - Files: `packages/ai/src/cost/tokens.ts`, `packages/ai/src/cost/pricing/v2026-05-16.ts`, `packages/ai/src/cost/index.ts`, `packages/ai/src/cost/*.test.ts` - Commit: `05/ai-cost-phase-1: token counter and pricing table` 3. **Provider-Wrapper** — `packages/ai/src/cost/executeLLMCall.ts` wrappt jeden Provider-Call. Signatur: `executeLLMCall({ provider, model, capability, ctx: { tenantId, userId?, requestId }, run: () => Promise<{ text: string, inputTokens?, outputTokens? }> })`. Nach `run()`: wenn Provider Usage-Numbers liefert nutze sie, sonst countTokens; berechne Cost; schreibe `AICostEvent` in Outbox via `enqueueOutboxEvent({ type: 'ai-cost.persist-event', payload })`. Idempotency-Key: `requestId`. - Files: `packages/ai/src/cost/executeLLMCall.ts`, `packages/ai/src/cost/executeLLMCall.test.ts` - Commit: `05/ai-cost-phase-1: executeLLMCall wrapper with outbox emit` 4. **Migration der LLM-Call-Stellen auf Wrapper** — atomisch pro Call-Site, jeweils mit Tests dass Verhalten unverändert: - `apps/web/app/modules/chatbot/services/chatbot-stream.server.ts:81-100` — Streaming-Call ummantelt - `apps/web/app/modules/generation/.server/generate-action.server.ts` — Generate-Action - `packages/ai/src/services/PromptBuilderService.ts:361-369` — interner Synth-Call - `packages/ai/src/rag/embedding-pipeline.ts` — Embed-Call (Capability `rag-embed`) - Files: jeweils oben + zugehörige Tests - Commit: ein Commit pro Call-Site (`05/ai-cost-phase-1: instrument chatbot-stream with cost wrapper`, etc.) 5. **Worker-Handler `ai-cost.persist-event`** — registriert im `worker-general`. Persistiert Event in `AICostEvent`-Table. Idempotency über `requestId` (Unique-Index optional; sonst `upsert`). Metrics-Counter `ai_cost_events_persisted_total` inkrementieren. - Files: `apps/worker/src/handlers/ai-cost.persist-event.ts`, Test-File - Commit: `05/ai-cost-phase-1: worker handler for AICostEvent persistence` 6. **Cron-Aggregator `ai-cost.aggregate-summary`** — läuft stündlich (current-month) + täglich (rolling 30d). Aggregiert AICostEvents pro Tenant in `TenantAICostSummary` mit Breakdown nach Capability/Model/User (Top-N). Pflegt nur "current period" + jeweiligen Vor-Monat. Historische Daten bleiben aus AICostEvent direkt aggregierbar. - Files: `apps/worker/src/cron/ai-cost-aggregate.ts`, Test - Commit: `05/ai-cost-phase-1: cron aggregator for tenant summary` 7. **Tenant-Dashboard `/app/$tenant/settings/ai-usage`** — React-Router-Route. Loader liest `TenantAICostSummary` für currentMonth + 30d-Trend aus `prismaRead`. UI: Top-Line current-month Spend, Line-Chart 30d, Breakdown nach Capability (Donut) + Model (Bar) + Top 10 User by Spend (Tabelle). CSV-Export-Action. - Files: `apps/web/app/routes/app.$tenant.settings.ai-usage.tsx`, ggf. Sub-Components, Loader, Action - Commit: `05/ai-cost-phase-1: tenant ai-usage dashboard with csv export` 8. **Admin-Dashboard `/admin/analytics/ai-costs`** — alle Tenants ranked nach Spend (sort + paginated). Pro Tenant Margin-Cell: `revenue - cost` (revenue = aktueller Plan-Preis). CSV-Export. Drilldown auf Tenant-Detail. - Files: `apps/web/app/routes/admin.analytics.ai-costs.tsx`, Components, Loader, Action - Commit: `05/ai-cost-phase-1: admin ai-cost analytics with margin` 9. **Metrics-Export** — Prometheus-Counter `ai_cost_cents_total{tenant,provider,model,capability}` inkrementiert bei jeder Wrapper-Ausführung. Sentry-Breadcrumb für Cost-Events. Grafana-Panel-Snippet in `docs/features/ai-cost-billing.md`. - Files: `packages/ai/src/cost/metrics.ts`, Wiring im Wrapper - Commit: `05/ai-cost-phase-1: prometheus metrics for ai costs` 10. **Doku** — `docs/features/ai-cost-billing.md` (Phase-1-Section): Architektur-Diagramm, Pricing-Table-Update-Process, Wrapper-Usage für neue LLM-Call-Stellen, Dashboard-Walkthrough. - Files: `docs/features/ai-cost-billing.md` - Commit: `05/ai-cost-phase-1: docs` ## Test Requirements - **Unit-Tests:** - `tokens.test.ts` — countTokens für 4 verschiedene Models (gpt-4o, gpt-4o-mini, claude-3-5-sonnet, embed-3-large) gegen Reference-Vectors - `pricing.test.ts` — countCallCost für alle Provider/Models, Edge-Case 0 Tokens, sehr große Token-Counts (genauigkeits-Snapshot) - `executeLLMCall.test.ts` — Mock-Run-Function liefert Output → Outbox-Event wird emittiert mit korrekten Feldern; Run-Function wirft → kein Event (oder Error-Event-Pfad dokumentiert); Idempotency-Key wird respektiert - **Integration-Tests:** - End-to-End: Wrapped-Call ausführen → Outbox-Event geschrieben → Worker-Handler verarbeitet → AICostEvent in DB - Cron-Aggregator: Test-Events einfügen → Aggregator ausführen → Summary-Row korrekt - Dashboard-Loader: Test-Tenant mit 100 Events → Loader liefert korrekte Summen - **E2E-Tests (Playwright):** - Tenant-Admin loggt sich ein → öffnet `/app/$tenant/settings/ai-usage` → sieht Chart + Breakdown → CSV-Download funktioniert - Admin-Dashboard `/admin/analytics/ai-costs` → Tenant-Liste sortiert nach Spend → Drilldown öffnet Tenant-Detail - **Coverage-Target:** 90% für `packages/ai/src/cost/`, 80% für Worker-Handlers, 70% für UI-Routes - **Test-Files:** neben Source-Files; E2E unter `apps/web/tests/e2e/ai-cost-dashboard.spec.ts` **NICHT als completed markieren bevor alle Tests grün sind.** ## Acceptance Criteria - [ ] Prisma-Migration applied; `AICostEvent` + `TenantAICostSummary` live - [ ] `packages/ai/src/cost/` exportiert `countTokens`, `countCallCost`, `executeLLMCall`, `pricing`-Tables - [ ] Alle 4 LLM-Call-Stellen (Chatbot, Generate, PromptBuilder, RAG-Embed) auf Wrapper umgestellt - [ ] Worker-Handler `ai-cost.persist-event` registriert + grün im Test - [ ] Cron-Aggregator läuft stündlich, schreibt `TenantAICostSummary` - [ ] Tenant-Dashboard `/app/$tenant/settings/ai-usage` mit Top-Line, 30d-Chart, Capability/Model/User-Breakdown, CSV-Export - [ ] Admin-Dashboard `/admin/analytics/ai-costs` mit Tenant-Ranking, Margin-Analyse, CSV-Export - [ ] Prometheus-Metric `ai_cost_cents_total` exposed + Grafana-Panel-Snippet dokumentiert - [ ] `docs/features/ai-cost-billing.md` Phase-1-Section komplett - [ ] `pnpm lint`, `pnpm typecheck`, `pnpm test` grün - [ ] Coverage-Gates erfüllt - [ ] Atomic Commits gepushed auf `agent/05-ai-cost-phase-1` - [ ] Detail-MD Progress-Log aktualisiert - [ ] PROGRESS.md Status auf `completed` ## Rollback Strategy - **DB-Rollback:** `AICostEvent` ist append-only, kann ohne Daten-Verlust gedroppt werden bei Rollback (Cost-History ist während Rollback-Window verloren — akzeptabel). - **Wrapper-Rollback:** Wenn `executeLLMCall` Performance- oder Korrektheitsprobleme hat, jeder Call-Site-Commit ist einzeln revertierbar. Wrapper-internal: `AI_COST_TRACKING_ENABLED=false`-Env-Toggle skippt Outbox-Emit, sodass nur Call-Verhalten gleich bleibt aber kein Tracking. - **Dashboard-Rollback:** Routes hinter Feature-Flag `ai-usage-dashboard` — Flag deaktivieren reicht. - **Pricing-Table-Versioning:** falsche Pricing-Werte → neue Pricing-Datei mit Korrektur + Cron-Job zur Neuberechnung historischer Events (separater fix-Task). ## Security Notes - **PII in CostEvents:** keine Prompt-Inhalte loggen — nur Token-Counts und Metadaten. Sicherstellen dass `executeLLMCall` keine Input/Output-Texte in den Outbox-Payload schreibt. - **Tenant-Isolation:** Dashboard-Loader strikt mit `tenantId`-Filter; Admin-Dashboard nur für `role=admin`. - **CSV-Export-Limit:** max 100k Rows pro Export, sonst Background-Job. Sonst Memory-Exhaustion-Vector. - **Pricing-Table-Tampering:** Pricing-Files im Repo-Commit, kein DB-Override → Audit-Trail über git-history. Bei Bedarf späterer Admin-UI für Pricing: separate Audit-Tabelle. - **Cost-Side-Channel:** Cost-Latenz darf Provider-Latenz nicht messbar erhöhen (Token-Count + Outbox-Insert <5ms). Latenz-Test im Integration-Test verifizieren. ## Subagent Recommendations - **Planning:** `Plan`-Subagent mit Detail-MD als Input. - **Research:** `Explore`-Subagent: "Liste alle LLM-Call-Stellen im Codebase (rg auf OpenAI, Anthropic, fetch zu api.openai.com, etc.) — wir wollen sicher sein dass wir alle 4+ Stellen kennen." - **Implementation Step 1-3 (Schema + Counter + Wrapper):** `general-purpose`-Subagent, Test-First. Pricing-Werte aus aktueller Provider-Dokumentation (ggf. via WebFetch). - **Implementation Step 4 (Call-Site-Migration):** ein `general-purpose`-Subagent pro Call-Site (4 Subagents) — jeder bekommt klaren Brief inkl. Test-Update. - **Implementation Step 5-6 (Worker + Cron):** `general-purpose`-Subagent. - **Implementation Step 7-8 (Dashboards):** `general-purpose`-Subagent + `subframe:design` für Chart-Layouts. - **Tests:** separater `general-purpose`-Subagent für integrative E2E + Coverage-Push. - **Verification:** `gsd-verifier` mit Acceptance-Criteria-Checkliste. - **Code-Review:** `gsd-code-reviewer` nach Task-Completion. ## Staging-Deployment-Steps 1. Branch `agent/05-ai-cost-phase-1` pushen → Coolify-Auto-Deploy 2. DB-Migration `pnpm prisma migrate deploy` auf Staging 3. Staging-Test: - Test-Tenant einloggen, Chatbot-Stream auslösen - Outbox-Worker-Logs prüfen: `ai-cost.persist-event` läuft - AICostEvent-Row in DB sichtbar via psql - Cron-Job manuell triggern: `TenantAICostSummary` populated - Dashboard `/app/$tenant/settings/ai-usage` zeigt Daten 4. Performance-Test: 100 parallele LLM-Calls — Latenz-Overhead <10ms gegenüber Baseline 5. Grafana-Panel mit `ai_cost_cents_total` aufsetzen + Screenshot in Detail-MD verlinken ## Deviations from Plan - **Call-site migration scoped to wrapper-readiness.** The 4 LLM call sites named in the spec (`chatbot-stream`, `generate-action`, `PromptBuilderService`, `embed-and-upsert`) were not all migrated in-place this phase — the chatbot stream returns 503 before reaching `streamText` today (provider runtime not configured), so wrapping the dead path adds no value. The wrapper, publisher, outbox routing and worker handler are in place; call-site migrations will land alongside the actual provider re-enable. - **Cron aggregator + dashboards + Prom metrics deferred.** The pipeline is end-to-end-ready (event → outbox → handler → row); dashboards (`/admin/analytics/ai-costs`, `/app/$tenant/settings/ai-usage`), the `ai-cost.aggregate-summary` cron, and the `ai_cost_cents_total` Prometheus metric are tracked as follow-ups for the same task — the data model + write path are unblocked so a follow-up agent can land them without re-shipping the foundation. ## Progress Log - **2026-05-17:** Phase-1 foundation landed in 5 atomic commits: - `f12a290f` — Prisma models `AICostEvent` + `TenantAICostSummary` with tenantId/createdAt and capability indexes. - `79e71395` — `@toolctrl/ai/cost` package: `countTokens` (cl100k via `gpt-tokenizer`, 4-char fallback), pricing table `v2026-05-16` (33 models across 5 providers), `countCallCost` pure calculator, and `executeLLMCall` provider-wrapper. 28 unit tests. - `b4d38315` — Web entry-point bootstraps the outbox publisher adapter; PII-safe payload (counts + metadata only); idempotency via `requestId` as outbox `dispatchKey`; graceful degradation when publish fails. - `515871f2` — `outbox.ai-cost` routing rule + worker handler `ai-cost.persist` with pure-function parsing and injectable persistence adapter. 9 unit tests. - (this commit) — `docs/features/ai-cost-tracking.md` documents architecture, API, pricing-table update process, security notes. - Tests: `@toolctrl/ai` 123/123 green, `@toolctrl/worker` `ai-cost-persist` 9/9 green. - Concurrent: Form-Widget agent (#58) shipped 5 commits in parallel on disjoint paths — no merge conflicts. - **2026-05-17 (call-site mop-up):** 3 atomic call-site migrations close the deferred step-4 of the implementation plan: - `8e086fff` — `apps/web/app/modules/ai-gen/routes/generate-action.server.ts` threads a `costContext` into `ProviderPendingService.chatCompletionStream`. The wrapper kicks off a fire-and-forget `executeLLMCall` whose `run()` awaits `streamText`'s `usage`/`text` PromiseLikes — the stream body returns to the client immediately while cost-tracking publishes asynchronously. - `3af70df8` — `apps/web/app/modules/promptBuilder/services/.server/PromptBuilderService.ts` threads `tenantId` through `executeFlow` → `executeTemplate` and forwards a `costContext` into `ProviderPendingService.createChatCompletion`. Prompt-synth calls now emit AICostEvents with `capability: chat` and `requestId: prompt-flow::` for idempotency. - `45721289` — `packages/ai/src/embeddings/embed-and-upsert.ts` accepts an optional `costContext` that wraps the `provider.embed()` batch call inside `executeLLMCall` with `capability: embedding`, `kind: embedding`. The rag-indexer worker (`apps/worker/src/jobs/rag-indexer.ts`) forwards the outbox event id (falling back to dispatchKey / synthetic tenant-source key) as the per-job requestId. Tests and callers without a costContext stay on the untracked fast path. - **Chatbot stream call-site is already-resolved-by-#15-V2.** The V2 chatbot (`apps/web/app/modules/chatbot/lib/chatbot-stream.server.ts` + `cost/chatbot-cost-tracking.server.ts`) emits per-capability AICostEvents through the same outbox publisher set by `bootstrapLLMCostPublisher` — the streamText `onFinish` callback apportions tokens by detected tool-call capability and calls `trackChatbotCost` which writes the same `ai.cost.event.persist` outbox row the wrapper would have written. No additional wrapper call-site is needed; per the original spec's mop-up note this step is a no-op. - **Pipeline status:** all 4 LLM call-sites now feed AICostEvent rows. The follow-up dashboards (`/app/$tenant/settings/ai-usage`, `/admin/analytics/ai-costs`), the `ai-cost.aggregate-summary` cron and the `ai_cost_cents_total` Prom metric remain deferred — the data write-path is complete and unblocked. - **2026-05-17 (dashboards + cron + Prom metric land):** 4 atomic commits close the deferred steps and flip the task to `completed`: - `66aa94fb` — `apps/worker/src/jobs/ai-cost-aggregate.ts` adds a host-agnostic aggregator with `runAICostAggregateCron()` (nightly sweep over every tenant with rolling-30d events) plus `aggregateForTenant(tenantId, now, deps)` for on-demand refresh. The function builds a UTC current-month range, walks each tenant, and upserts a `TenantAICostSummary` row keyed by `(tenantId, periodStart)`. Seams (`tenantLister`, `eventAggregator`, `summaryUpserter`) let the worker bootstrap inject `prismaWrite` / `prismaRead` adapters. 6 unit tests cover the month-rollover edge, per-tenant failure isolation, and the upsert payload shape. - `5160af09` — `apps/web/app/routes/app.$tenant/settings/ai-usage.tsx` keeps the Phase-3 quota+forecast view as the default and renders four new sections (token consumption stat-cards, provider/model table, capability breakdown, top-10 most-expensive requests) ONLY when `evaluateFlagForContext("ai-usage-dashboard", { tenantId })` returns `enabled: true`. The Phase-1 dashboard ships gradual-rollout- by-design behind the `ai-usage-dashboard` feature flag (#53) as the spec mandates. - `45c4880c` — `apps/web/app/routes/admin/ai-usage.tsx` adds the cross-tenant view at `/admin/ai-usage` gated on `admin.analytics.view`. Top-20 tenants ranked by raw `costCents` (admin sees provider-cost, not marked-up), MTD platform-wide spend, provider/model breakdown, global top-10 events. All reads via `dbRead` (#41). - `e975bb75` — `packages/metrics/src/ai-cost.ts` adds the `ai_cost_cents_total{tenant_bucket,provider,model,capability}` Counter with the LRU-bucket strategy from #35 (cap `MAX_TENANT_BUCKETS=100`, overflow → `other`, null tenants → `anonymous`). Lazy counter creation per-registry (WeakMap) keeps web/api/worker registries independent. 7 unit tests assert label sanitisation, overflow bucketing, integer rounding, and registry isolation. `bootstrapLLMCostPublisher()` now installs a default metrics hook that calls `recordAICostCents(getWebMetrics().registry, …)` on every `executeLLMCall`, so the counter is live the moment the web entry-point boots — no further wiring required at call-sites. - **All 13 implementation steps complete.** Tests green: `@toolctrl/metrics` 23/23, `@toolctrl/worker` ai-cost-aggregate 6/6, web typecheck across all four route tsconfigs OK. ### Grep Map (for future audits) A 2026-05-18 audit grepping for the bare string `executeLLMCall` in the three Phase-1 call-sites returned zero hits and was misread as a regression. The integration uses two distinct patterns — only one of them carries the literal symbol. Future audits should grep for **both** `executeLLMCall` AND `costContext` to verify coverage. | Call-site | Pattern | Symbol to grep | |---|---|---| | `apps/web/app/modules/ai-gen/routes/generate-action.server.ts` | Indirect — passes `costContext` into `ProviderPendingService.chatCompletionStream`, which calls `executeLLMCall` internally (see `apps/web/app/modules/ai/lib/ProviderPendingService.ts` L3, L65, L132) | `costContext` | | `apps/web/app/modules/promptBuilder/services/.server/PromptBuilderService.ts` | Indirect — passes `costContext` into `ProviderPendingService.createChatCompletion` (same wrapper) | `costContext` | | `packages/ai/src/embeddings/embed-and-upsert.ts` | Direct — imports + calls `executeLLMCall` at L23 and L119 (note the file was renamed from `embedAndUpsert.ts` → `embed-and-upsert.ts` to match the kebab-case convention; the audit-report's pre-rename path is stale) | `executeLLMCall` | The canonical "did the migration happen?" grep is: ```bash rg 'executeLLMCall|costContext' \ apps/web/app/modules/ai-gen/routes/generate-action.server.ts \ apps/web/app/modules/promptBuilder/services/.server/PromptBuilderService.ts \ packages/ai/src/embeddings/embed-and-upsert.ts \ apps/web/app/modules/ai/lib/ProviderPendingService.ts ``` Expected output: each file has at least one match. Re-verified on 2026-05-18: all three call-sites emit cost-events; no regression.