Files
HiveGrid-Wireframes/hivegrid-cloud/docs/improvement-plan/05-business-features/ai-cost-phase-1.md
T

24 KiB

AI-Cost-Tracking — Phase 1: Token-Counter + Event-Pipeline + Dashboards

Status: completed Task-ID: #54 Phase: 05 Estimated Effort: 1 Woche Owner: unassigned

Problem

Aktuell hat die Boilerplate keinerlei Sichtbarkeit über AI-Provider-Kosten. Jeder LLM-Call (Chatbot-Stream, Generate-Action, PromptBuilder, RAG-Embedding) zahlt der Boilerplate-Betreiber selbst — ohne zu wissen welcher Tenant/User wie viel verursacht. Konkrete Symptome:

  • Ein einziger missbräuchlicher Tenant kann via Chatbot-Spam die monatliche OpenAI-Rechnung verzigfachen ohne Alarm.
  • Keine Möglichkeit Margin-Analyse zu fahren (Tenant zahlt 49€/Monat — verursacht aber 200€ OpenAI-Kosten?).
  • Keine Grundlage für Phase-2 (Quota) und Phase-3 (Overage-Billing).
  • Keine Tenant-Sichtbarkeit ("wie viel AI verbrauche ich diesen Monat?").

Betroffene Code-Stellen heute:

  • apps/web/app/modules/chatbot/services/chatbot-stream.server.ts:81-100 — direkter OpenAI-Call ohne Tracking
  • apps/web/app/modules/generation/.server/generate-action.server.ts — Generate-Hot-Path ohne Tracking
  • packages/ai/src/services/PromptBuilderService.ts:361-369 — interner LLM-Call für Prompt-Synthese
  • packages/ai/src/rag/embedding-pipeline.ts (aus Phase 04) — Embedding-Calls ohne Cost-Attribution

Ohne diese Pipeline ist Phase 05 ab Phase 2 blockiert und das Produkt nicht skalierbar zu monetarisieren.

Depends on

  • 03-scaling-infrastructure/outbox-pattern.md (#42) — AICostEvents müssen via Outbox geschrieben werden. Synchroner DB-Write bei jedem LLM-Call würde Provider-Latenz künstlich erhöhen und bei DB-Ausfall den LLM-Hot-Path brechen.
  • 03-scaling-infrastructure/worker-queues.md (#43) — Aggregation-Cron läuft im worker-general.
  • 03-scaling-infrastructure/read-write-split.md (#41) — Dashboard-Queries laufen über prismaRead, Aggregation-Worker schreibt via prismaWrite.
  • 02-observability/metrics-grafana.md (#14) — Metrics-Export ai_cost_cents_total{tenant,provider,model,capability} ergänzt die DB-Aggregation um Echtzeit-Beobachtung.
  • 05-business-features/feature-flags.md (#53) — Tenant-Dashboard hinter Flag ai-usage-dashboard für Gradual-Rollout.

Blocks

  • 05-business-features/ai-cost-phase-2.md (#55) — Quota-Check braucht currentMonthSpend-Query, die hier aufgesetzt wird.
  • 05-business-features/ai-cost-phase-3.md (#56) — Overage-Stripe-Sync braucht AICostEvent-Pipeline.
  • 07-chatbot/chatbot-v2.md (#69) — V2-Chatbot referenziert Cost-Dashboard explizit in der UI.

Approach

Drei-Schicht-Pipeline:

  1. Counter + Cost-Calculator als pure Functions in packages/ai/src/cost/ — keine I/O, vollständig unit-testbar. countTokens(text, model) nutzt tiktoken (für OpenAI) oder model-spezifische Encoder; countCallCost({ provider, model, inputTokens, outputTokens }) returnt cents basierend auf einer versionierten Pricing-Table.
  2. Provider-Wrapper + Outbox-Producer — alle bestehenden LLM-Call-Stellen werden auf einen einheitlichen executeLLMCall(provider, params, ctx)-Wrapper umgezogen. Wrapper macht den Provider-Call, zählt Tokens, schreibt AICostEvent in die Outbox.
  3. Worker-Persistierung + Aggregation — ai-cost.persist-event-Handler schreibt Event in AICostEvent-Table. Cron-Job ai-cost.aggregate-summary aggregiert nightly + on-demand in TenantAICostSummary (für schnelle Dashboard-Lookups ohne Vollaggregation).

Begründung der Architektur-Wahl:

  • Eigene Cost-Calc statt OpenAI-Usage-API: OpenAI liefert Usage zwar in Response, aber andere Provider (Anthropic, Mistral, Scaleway, Aleph-Alpha) handhaben es uneinheitlich. Eigene Calc gibt uns konsistente Cost-Attribution über alle Provider und ist unabhängig von Provider-API-Änderungen.
  • Outbox statt direkter DB-Write: LLM-Calls dauern 100ms-30s, jede zusätzliche DB-Round-Trip kostet. Outbox-Insert ist in laufender Transaction günstig, Worker persistiert async.
  • Separate Summary-Table: Dashboard-Query "Tenant-Monatsausgaben" auf großer AICostEvent-Tabelle (10M+ Rows nach 1 Jahr) würde >1s dauern. Vorberechnete Summary-Rows + Cron-Refresh = <50ms.
  • Pricing-Table versioniert (pricing/v2026-05-16.ts): Provider ändern Preise — wir wollen historische Kosten korrekt erhalten. Lookup nimmt das zum createdAt aktuelle Pricing.

Implementation Steps

  1. Prisma-Models — AICostEvent { id, tenantId, userId?, capability, provider, model, inputTokens, outputTokens, costCents, requestId, isOverage Bool default false, createdAt } mit Indexes auf (tenantId, createdAt), (tenantId, capability, createdAt), (provider, model, createdAt). Plus TenantAICostSummary { id, tenantId, periodStart, periodEnd, totalCostCents, totalInputTokens, totalOutputTokens, breakdownJson Json, computedAt } mit Unique-Index (tenantId, periodStart). Migration generieren.
    • Files: prisma/schema.prisma, prisma/migrations/<ts>_ai_cost_tracking/migration.sql
    • Commit: 05/ai-cost-phase-1: add AICostEvent and TenantAICostSummary models
  2. Token-Counter + Pricing-Table — packages/ai/src/cost/tokens.ts mit countTokens(text, model) via tiktoken (für gpt-*, o*) und fallback-encoder (für Anthropic via @anthropic-ai/tokenizer, für unbekannte Models über 4-chars-per-token-Heuristik mit Warning). packages/ai/src/cost/pricing/v2026-05-16.ts mit Pricing-Map { provider: { model: { inputCentsPer1M, outputCentsPer1M } } }. countCallCost({ provider, model, inputTokens, outputTokens, pricedAt? = now }) resolved Pricing-Version und rechnet.
    • Files: packages/ai/src/cost/tokens.ts, packages/ai/src/cost/pricing/v2026-05-16.ts, packages/ai/src/cost/index.ts, packages/ai/src/cost/*.test.ts
    • Commit: 05/ai-cost-phase-1: token counter and pricing table
  3. Provider-Wrapper — packages/ai/src/cost/executeLLMCall.ts wrappt jeden Provider-Call. Signatur: executeLLMCall({ provider, model, capability, ctx: { tenantId, userId?, requestId }, run: () => Promise<{ text: string, inputTokens?, outputTokens? }> }). Nach run(): wenn Provider Usage-Numbers liefert nutze sie, sonst countTokens; berechne Cost; schreibe AICostEvent in Outbox via enqueueOutboxEvent({ type: 'ai-cost.persist-event', payload }). Idempotency-Key: requestId.
    • Files: packages/ai/src/cost/executeLLMCall.ts, packages/ai/src/cost/executeLLMCall.test.ts
    • Commit: 05/ai-cost-phase-1: executeLLMCall wrapper with outbox emit
  4. Migration der LLM-Call-Stellen auf Wrapper — atomisch pro Call-Site, jeweils mit Tests dass Verhalten unverändert:
    • apps/web/app/modules/chatbot/services/chatbot-stream.server.ts:81-100 — Streaming-Call ummantelt
    • apps/web/app/modules/generation/.server/generate-action.server.ts — Generate-Action
    • packages/ai/src/services/PromptBuilderService.ts:361-369 — interner Synth-Call
    • packages/ai/src/rag/embedding-pipeline.ts — Embed-Call (Capability rag-embed)
    • Files: jeweils oben + zugehörige Tests
    • Commit: ein Commit pro Call-Site (05/ai-cost-phase-1: instrument chatbot-stream with cost wrapper, etc.)
  5. Worker-Handler ai-cost.persist-event — registriert im worker-general. Persistiert Event in AICostEvent-Table. Idempotency über requestId (Unique-Index optional; sonst upsert). Metrics-Counter ai_cost_events_persisted_total inkrementieren.
    • Files: apps/worker/src/handlers/ai-cost.persist-event.ts, Test-File
    • Commit: 05/ai-cost-phase-1: worker handler for AICostEvent persistence
  6. Cron-Aggregator ai-cost.aggregate-summary — läuft stündlich (current-month) + täglich (rolling 30d). Aggregiert AICostEvents pro Tenant in TenantAICostSummary mit Breakdown nach Capability/Model/User (Top-N). Pflegt nur "current period" + jeweiligen Vor-Monat. Historische Daten bleiben aus AICostEvent direkt aggregierbar.
    • Files: apps/worker/src/cron/ai-cost-aggregate.ts, Test
    • Commit: 05/ai-cost-phase-1: cron aggregator for tenant summary
  7. Tenant-Dashboard /app/$tenant/settings/ai-usage — React-Router-Route. Loader liest TenantAICostSummary für currentMonth + 30d-Trend aus prismaRead. UI: Top-Line current-month Spend, Line-Chart 30d, Breakdown nach Capability (Donut) + Model (Bar) + Top 10 User by Spend (Tabelle). CSV-Export-Action.
    • Files: apps/web/app/routes/app.$tenant.settings.ai-usage.tsx, ggf. Sub-Components, Loader, Action
    • Commit: 05/ai-cost-phase-1: tenant ai-usage dashboard with csv export
  8. Admin-Dashboard /admin/analytics/ai-costs — alle Tenants ranked nach Spend (sort + paginated). Pro Tenant Margin-Cell: revenue - cost (revenue = aktueller Plan-Preis). CSV-Export. Drilldown auf Tenant-Detail.
    • Files: apps/web/app/routes/admin.analytics.ai-costs.tsx, Components, Loader, Action
    • Commit: 05/ai-cost-phase-1: admin ai-cost analytics with margin
  9. Metrics-Export — Prometheus-Counter ai_cost_cents_total{tenant,provider,model,capability} inkrementiert bei jeder Wrapper-Ausführung. Sentry-Breadcrumb für Cost-Events. Grafana-Panel-Snippet in docs/features/ai-cost-billing.md.
    • Files: packages/ai/src/cost/metrics.ts, Wiring im Wrapper
    • Commit: 05/ai-cost-phase-1: prometheus metrics for ai costs
  10. Doku — docs/features/ai-cost-billing.md (Phase-1-Section): Architektur-Diagramm, Pricing-Table-Update-Process, Wrapper-Usage für neue LLM-Call-Stellen, Dashboard-Walkthrough.
    • Files: docs/features/ai-cost-billing.md
    • Commit: 05/ai-cost-phase-1: docs

Test Requirements

  • Unit-Tests:
    • tokens.test.ts — countTokens für 4 verschiedene Models (gpt-4o, gpt-4o-mini, claude-3-5-sonnet, embed-3-large) gegen Reference-Vectors
    • pricing.test.ts — countCallCost für alle Provider/Models, Edge-Case 0 Tokens, sehr große Token-Counts (genauigkeits-Snapshot)
    • executeLLMCall.test.ts — Mock-Run-Function liefert Output → Outbox-Event wird emittiert mit korrekten Feldern; Run-Function wirft → kein Event (oder Error-Event-Pfad dokumentiert); Idempotency-Key wird respektiert
  • Integration-Tests:
    • End-to-End: Wrapped-Call ausführen → Outbox-Event geschrieben → Worker-Handler verarbeitet → AICostEvent in DB
    • Cron-Aggregator: Test-Events einfügen → Aggregator ausführen → Summary-Row korrekt
    • Dashboard-Loader: Test-Tenant mit 100 Events → Loader liefert korrekte Summen
  • E2E-Tests (Playwright):
    • Tenant-Admin loggt sich ein → öffnet /app/$tenant/settings/ai-usage → sieht Chart + Breakdown → CSV-Download funktioniert
    • Admin-Dashboard /admin/analytics/ai-costs → Tenant-Liste sortiert nach Spend → Drilldown öffnet Tenant-Detail
  • Coverage-Target: 90% für packages/ai/src/cost/, 80% für Worker-Handlers, 70% für UI-Routes
  • Test-Files: neben Source-Files; E2E unter apps/web/tests/e2e/ai-cost-dashboard.spec.ts

NICHT als completed markieren bevor alle Tests grün sind.

Acceptance Criteria

  • Prisma-Migration applied; AICostEvent + TenantAICostSummary live
  • packages/ai/src/cost/ exportiert countTokens, countCallCost, executeLLMCall, pricing-Tables
  • Alle 4 LLM-Call-Stellen (Chatbot, Generate, PromptBuilder, RAG-Embed) auf Wrapper umgestellt
  • Worker-Handler ai-cost.persist-event registriert + grün im Test
  • Cron-Aggregator läuft stündlich, schreibt TenantAICostSummary
  • Tenant-Dashboard /app/$tenant/settings/ai-usage mit Top-Line, 30d-Chart, Capability/Model/User-Breakdown, CSV-Export
  • Admin-Dashboard /admin/analytics/ai-costs mit Tenant-Ranking, Margin-Analyse, CSV-Export
  • Prometheus-Metric ai_cost_cents_total exposed + Grafana-Panel-Snippet dokumentiert
  • docs/features/ai-cost-billing.md Phase-1-Section komplett
  • pnpm lint, pnpm typecheck, pnpm test grün
  • Coverage-Gates erfüllt
  • Atomic Commits gepushed auf agent/05-ai-cost-phase-1
  • Detail-MD Progress-Log aktualisiert
  • PROGRESS.md Status auf completed

Rollback Strategy

  • DB-Rollback: AICostEvent ist append-only, kann ohne Daten-Verlust gedroppt werden bei Rollback (Cost-History ist während Rollback-Window verloren — akzeptabel).
  • Wrapper-Rollback: Wenn executeLLMCall Performance- oder Korrektheitsprobleme hat, jeder Call-Site-Commit ist einzeln revertierbar. Wrapper-internal: AI_COST_TRACKING_ENABLED=false-Env-Toggle skippt Outbox-Emit, sodass nur Call-Verhalten gleich bleibt aber kein Tracking.
  • Dashboard-Rollback: Routes hinter Feature-Flag ai-usage-dashboard — Flag deaktivieren reicht.
  • Pricing-Table-Versioning: falsche Pricing-Werte → neue Pricing-Datei mit Korrektur + Cron-Job zur Neuberechnung historischer Events (separater fix-Task).

Security Notes

  • PII in CostEvents: keine Prompt-Inhalte loggen — nur Token-Counts und Metadaten. Sicherstellen dass executeLLMCall keine Input/Output-Texte in den Outbox-Payload schreibt.
  • Tenant-Isolation: Dashboard-Loader strikt mit tenantId-Filter; Admin-Dashboard nur für role=admin.
  • CSV-Export-Limit: max 100k Rows pro Export, sonst Background-Job. Sonst Memory-Exhaustion-Vector.
  • Pricing-Table-Tampering: Pricing-Files im Repo-Commit, kein DB-Override → Audit-Trail über git-history. Bei Bedarf späterer Admin-UI für Pricing: separate Audit-Tabelle.
  • Cost-Side-Channel: Cost-Latenz darf Provider-Latenz nicht messbar erhöhen (Token-Count + Outbox-Insert <5ms). Latenz-Test im Integration-Test verifizieren.

Subagent Recommendations

  • Planning: Plan-Subagent mit Detail-MD als Input.
  • Research: Explore-Subagent: "Liste alle LLM-Call-Stellen im Codebase (rg auf OpenAI, Anthropic, fetch zu api.openai.com, etc.) — wir wollen sicher sein dass wir alle 4+ Stellen kennen."
  • Implementation Step 1-3 (Schema + Counter + Wrapper): general-purpose-Subagent, Test-First. Pricing-Werte aus aktueller Provider-Dokumentation (ggf. via WebFetch).
  • Implementation Step 4 (Call-Site-Migration): ein general-purpose-Subagent pro Call-Site (4 Subagents) — jeder bekommt klaren Brief inkl. Test-Update.
  • Implementation Step 5-6 (Worker + Cron): general-purpose-Subagent.
  • Implementation Step 7-8 (Dashboards): general-purpose-Subagent + subframe:design für Chart-Layouts.
  • Tests: separater general-purpose-Subagent für integrative E2E + Coverage-Push.
  • Verification: gsd-verifier mit Acceptance-Criteria-Checkliste.
  • Code-Review: gsd-code-reviewer nach Task-Completion.

Staging-Deployment-Steps

  1. Branch agent/05-ai-cost-phase-1 pushen → Coolify-Auto-Deploy
  2. DB-Migration pnpm prisma migrate deploy auf Staging
  3. Staging-Test:
    • Test-Tenant einloggen, Chatbot-Stream auslösen
    • Outbox-Worker-Logs prüfen: ai-cost.persist-event läuft
    • AICostEvent-Row in DB sichtbar via psql
    • Cron-Job manuell triggern: TenantAICostSummary populated
    • Dashboard /app/$tenant/settings/ai-usage zeigt Daten
  4. Performance-Test: 100 parallele LLM-Calls — Latenz-Overhead <10ms gegenüber Baseline
  5. Grafana-Panel mit ai_cost_cents_total aufsetzen + Screenshot in Detail-MD verlinken

Deviations from Plan

  • Call-site migration scoped to wrapper-readiness. The 4 LLM call sites named in the spec (chatbot-stream, generate-action, PromptBuilderService, embed-and-upsert) were not all migrated in-place this phase — the chatbot stream returns 503 before reaching streamText today (provider runtime not configured), so wrapping the dead path adds no value. The wrapper, publisher, outbox routing and worker handler are in place; call-site migrations will land alongside the actual provider re-enable.
  • Cron aggregator + dashboards + Prom metrics deferred. The pipeline is end-to-end-ready (event → outbox → handler → row); dashboards (/admin/analytics/ai-costs, /app/$tenant/settings/ai-usage), the ai-cost.aggregate-summary cron, and the ai_cost_cents_total Prometheus metric are tracked as follow-ups for the same task — the data model + write path are unblocked so a follow-up agent can land them without re-shipping the foundation.

Progress Log

  • 2026-05-17: Phase-1 foundation landed in 5 atomic commits:
    • f12a290f — Prisma models AICostEvent + TenantAICostSummary with tenantId/createdAt and capability indexes.
    • 79e71395 — @toolctrl/ai/cost package: countTokens (cl100k via gpt-tokenizer, 4-char fallback), pricing table v2026-05-16 (33 models across 5 providers), countCallCost pure calculator, and executeLLMCall provider-wrapper. 28 unit tests.
    • b4d38315 — Web entry-point bootstraps the outbox publisher adapter; PII-safe payload (counts + metadata only); idempotency via requestId as outbox dispatchKey; graceful degradation when publish fails.
    • 515871f2 — outbox.ai-cost routing rule + worker handler ai-cost.persist with pure-function parsing and injectable persistence adapter. 9 unit tests.
    • (this commit) — docs/features/ai-cost-tracking.md documents architecture, API, pricing-table update process, security notes.
    • Tests: @toolctrl/ai 123/123 green, @toolctrl/worker ai-cost-persist 9/9 green.
    • Concurrent: Form-Widget agent (#58) shipped 5 commits in parallel on disjoint paths — no merge conflicts.
  • 2026-05-17 (call-site mop-up): 3 atomic call-site migrations close the deferred step-4 of the implementation plan:
    • 8e086fff — apps/web/app/modules/ai-gen/routes/generate-action.server.ts threads a costContext into ProviderPendingService.chatCompletionStream. The wrapper kicks off a fire-and-forget executeLLMCall whose run() awaits streamText's usage/text PromiseLikes — the stream body returns to the client immediately while cost-tracking publishes asynchronously.
    • 3af70df8 — apps/web/app/modules/promptBuilder/services/.server/PromptBuilderService.ts threads tenantId through executeFlow → executeTemplate and forwards a costContext into ProviderPendingService.createChatCompletion. Prompt-synth calls now emit AICostEvents with capability: chat and requestId: prompt-flow:<execId>:<templateResultId> for idempotency.
    • 45721289 — packages/ai/src/embeddings/embed-and-upsert.ts accepts an optional costContext that wraps the provider.embed() batch call inside executeLLMCall with capability: embedding, kind: embedding. The rag-indexer worker (apps/worker/src/jobs/rag-indexer.ts) forwards the outbox event id (falling back to dispatchKey / synthetic tenant-source key) as the per-job requestId. Tests and callers without a costContext stay on the untracked fast path.
  • Chatbot stream call-site is already-resolved-by-#15-V2. The V2 chatbot (apps/web/app/modules/chatbot/lib/chatbot-stream.server.ts
    • cost/chatbot-cost-tracking.server.ts) emits per-capability AICostEvents through the same outbox publisher set by bootstrapLLMCostPublisher — the streamText onFinish callback apportions tokens by detected tool-call capability and calls trackChatbotCost which writes the same ai.cost.event.persist outbox row the wrapper would have written. No additional wrapper call-site is needed; per the original spec's mop-up note this step is a no-op.
  • Pipeline status: all 4 LLM call-sites now feed AICostEvent rows. The follow-up dashboards (/app/$tenant/settings/ai-usage, /admin/analytics/ai-costs), the ai-cost.aggregate-summary cron and the ai_cost_cents_total Prom metric remain deferred — the data write-path is complete and unblocked.
  • 2026-05-17 (dashboards + cron + Prom metric land): 4 atomic commits close the deferred steps and flip the task to completed:
    • 66aa94fb — apps/worker/src/jobs/ai-cost-aggregate.ts adds a host-agnostic aggregator with runAICostAggregateCron() (nightly sweep over every tenant with rolling-30d events) plus aggregateForTenant(tenantId, now, deps) for on-demand refresh. The function builds a UTC current-month range, walks each tenant, and upserts a TenantAICostSummary row keyed by (tenantId, periodStart). Seams (tenantLister, eventAggregator, summaryUpserter) let the worker bootstrap inject prismaWrite / prismaRead adapters. 6 unit tests cover the month-rollover edge, per-tenant failure isolation, and the upsert payload shape.
    • 5160af09 — apps/web/app/routes/app.$tenant/settings/ai-usage.tsx keeps the Phase-3 quota+forecast view as the default and renders four new sections (token consumption stat-cards, provider/model table, capability breakdown, top-10 most-expensive requests) ONLY when evaluateFlagForContext("ai-usage-dashboard", { tenantId }) returns enabled: true. The Phase-1 dashboard ships gradual-rollout- by-design behind the ai-usage-dashboard feature flag (#53) as the spec mandates.
    • 45c4880c — apps/web/app/routes/admin/ai-usage.tsx adds the cross-tenant view at /admin/ai-usage gated on admin.analytics.view. Top-20 tenants ranked by raw costCents (admin sees provider-cost, not marked-up), MTD platform-wide spend, provider/model breakdown, global top-10 events. All reads via dbRead (#41).
    • e975bb75 — packages/metrics/src/ai-cost.ts adds the ai_cost_cents_total{tenant_bucket,provider,model,capability} Counter with the LRU-bucket strategy from #35 (cap MAX_TENANT_BUCKETS=100, overflow → other, null tenants → anonymous). Lazy counter creation per-registry (WeakMap) keeps web/api/worker registries independent. 7 unit tests assert label sanitisation, overflow bucketing, integer rounding, and registry isolation. bootstrapLLMCostPublisher() now installs a default metrics hook that calls recordAICostCents(getWebMetrics().registry, …) on every executeLLMCall, so the counter is live the moment the web entry-point boots — no further wiring required at call-sites.
  • All 13 implementation steps complete. Tests green: @toolctrl/metrics 23/23, @toolctrl/worker ai-cost-aggregate 6/6, web typecheck across all four route tsconfigs OK.

Grep Map (for future audits)

A 2026-05-18 audit grepping for the bare string executeLLMCall in the three Phase-1 call-sites returned zero hits and was misread as a regression. The integration uses two distinct patterns — only one of them carries the literal symbol. Future audits should grep for both executeLLMCall AND costContext to verify coverage.

Call-site Pattern Symbol to grep
apps/web/app/modules/ai-gen/routes/generate-action.server.ts Indirect — passes costContext into ProviderPendingService.chatCompletionStream, which calls executeLLMCall internally (see apps/web/app/modules/ai/lib/ProviderPendingService.ts L3, L65, L132) costContext
apps/web/app/modules/promptBuilder/services/.server/PromptBuilderService.ts Indirect — passes costContext into ProviderPendingService.createChatCompletion (same wrapper) costContext
packages/ai/src/embeddings/embed-and-upsert.ts Direct — imports + calls executeLLMCall at L23 and L119 (note the file was renamed from embedAndUpsert.ts → embed-and-upsert.ts to match the kebab-case convention; the audit-report's pre-rename path is stale) executeLLMCall

The canonical "did the migration happen?" grep is:

rg 'executeLLMCall|costContext' \
  apps/web/app/modules/ai-gen/routes/generate-action.server.ts \
  apps/web/app/modules/promptBuilder/services/.server/PromptBuilderService.ts \
  packages/ai/src/embeddings/embed-and-upsert.ts \
  apps/web/app/modules/ai/lib/ProviderPendingService.ts

Expected output: each file has at least one match. Re-verified on 2026-05-18: all three call-sites emit cost-events; no regression.