212 lines
17 KiB
Markdown
212 lines
17 KiB
Markdown
# AI-Cost-Tracking — Phase 3: Stripe-Overage-Billing + Markup
|
||
|
||
**Status:** in_progress (code-complete, stripe-account-setup-pending)
|
||
**Task-ID:** #56
|
||
**Phase:** 05
|
||
**Estimated Effort:** 1-2 Wochen
|
||
**Owner:** unassigned
|
||
|
||
## Problem
|
||
|
||
Phase 2 verhindert Verbrauch über Quota. Aber: Tenants wollen oft *mehr* AI als ihre Quota — und wir wollen ihnen das verkaufen, statt sie zu blockieren. Aktuell ist es unmöglich, "20$/Monat + 0,002$/Token darüber" zu verkaufen. Wir brauchen:
|
||
|
||
- Markup-Konfiguration pro Plan (z.B. Pro = 2× Provider-Cost, Enterprise = 1.5×)
|
||
- Overage-Tracking: alles über Quota wird als billable Overage markiert
|
||
- Stripe-Usage-Based-Pricing: Overage-Cents werden als Meter-Events an Stripe gemeldet
|
||
- Stripe-Invoice enthält Overage automatisch als Line-Item bei nächster Period
|
||
- Tenant-UI zeigt Period-Forecast (Quota + projected Overage)
|
||
- Reconciliation: monatlicher Cross-Check unsere Cost-Events vs Stripe-Submitted-Sums
|
||
|
||
Ohne Phase 3 endet Cost-Tracking nutzlos: wir haben Sichtbarkeit + Schutz, aber kein Monetarisierungs-Hebel.
|
||
|
||
## Depends on
|
||
|
||
- `05-business-features/ai-cost-phase-2.md` (#55) — Quota-Definition + `assertTenantAICreditAccess` markiert Events als isOverage
|
||
- `05-business-features/ai-cost-phase-1.md` (#54) — AICostEvent-Pipeline + Persistence-Worker
|
||
- `05-business-features/feature-flags.md` (#53) — Per-Tenant-Overage-Enablement via Flag
|
||
- Existierende Stripe-Integration in `apps/web/app/utils/stripe.server.ts` (aktuell schon vorhanden, wird erweitert)
|
||
|
||
## Blocks
|
||
|
||
- `07-chatbot/chatbot-v2.md` (#69) — V2-Chatbot UI referenziert "Overage-Verbrauch" explizit
|
||
- Vermarktung der Boilerplate als "AI-Monetarisierbar" (Marketing-Claim für Phase-05-Marketing-Funnel)
|
||
|
||
## Approach
|
||
|
||
Drei Stufen:
|
||
|
||
1. **Markup-Konfiguration in pricingService** — pro Plan: `aiMarkupMultiplier` (z.B. 2.0), `aiOverageBlockCents` (Min-Increment für Stripe-Meter, sonst Mikro-Charges), `aiOverageStripeMeterId?` (Stripe-Side-Identifier). Admin-UI im Plan-Editor.
|
||
2. **Overage-Markierung im Persistence-Worker** — wenn `event.costCents` über aktuelle Period-Quota fällt, `event.isOverage = true` + `event.markedUpCostCents` (= costCents × multiplier). Nightly-Report-Cron aggregiert Overage pro Tenant und submitted Stripe-Meter-Events.
|
||
3. **Tenant-UI + Reconciliation** — Period-View mit Quota + Overage clear; monthly-Reconciliation-Cron vergleicht AICostEvent-Sum vs Stripe-Meter-Submitted-Sum, alert bei Mismatch >1%.
|
||
|
||
**Begründung der Architektur-Wahl:**
|
||
|
||
- **Markup vs Pass-Through:** wir verkaufen Service, nicht Provider-Tokens. Markup-Multiplier verschleiert Provider-Choice (wir können Provider wechseln ohne Tenant-Preisanpassung) und finanziert unseren Overhead (Hosting, Engineering).
|
||
- **Block-Cents-Quanten:** Stripe-Meter-Events sind nicht gratis bei extrem hoher Frequenz. Aggregation auf 10-cents-Blocks reduziert Event-Volume von 10k auf <100/Tag pro Tenant.
|
||
- **Nightly-Submit statt realtime:** Stripe-Meter-API hat Quotas + Latenz. Nightly-Aggregation ist robust + reduziert API-Calls. Für Real-Time-Anzeige nutzen wir interne AICostEvent-Sum, nicht Stripe-State.
|
||
- **Per-Plan-Feature-Flag:** Free-Plan hat keine Overage (Hard-Cap), Pro/Enterprise haben Overage. `aiOverageStripeMeterId === null` → kein Overage erlaubt → in Phase-2-Assert ist Soft-Degradation oder Deny das einzige Verhalten.
|
||
- **Reconciliation als Cron statt Real-Time:** Mismatches sind selten + nicht kritisch im Moment-by-Moment-Sinne; monatliches Audit reicht.
|
||
|
||
## Implementation Steps
|
||
|
||
1. **Plan-Schema-Erweiterung** — `packages/entitlements/src/types.ts`: `aiOverageStripeMeterId?: string`, `aiMarkupMultiplier?: number = 1.5`, `aiOverageBlockCents?: number = 10`. Migration nicht nötig (Plan-Definitionen sind code-side).
|
||
- Files: `packages/entitlements/src/types.ts`, `packages/entitlements/src/plans/*.ts`
|
||
- Commit: `05/ai-cost-phase-3: plan schema with overage and markup config`
|
||
2. **Admin-Plan-Editor-UI** — Erweiterung von `/admin/pricing` (oder dem entsprechenden Plan-Editor): drei neue Felder, Validation (Multiplier > 1.0, BlockCents >= 1), Save-Action mit Audit-Eintrag.
|
||
- Files: `apps/web/app/routes/admin.pricing....tsx` (existing), AuditEvent-Wiring
|
||
- Commit: `05/ai-cost-phase-3: admin ui for plan markup and overage config`
|
||
3. **Overage-Markierung im Persistence-Handler** — `ai-cost.persist-event`-Handler aus Phase 1 erweitert: vor Insert lookup currentMonthSpend; wenn `(currentSpend + event.costCents) > quota` → `event.isOverage = true`, berechne Overage-Anteil (falls Event teilweise in Quota + teilweise drüber liegt, splitten oder vereinfacht: kompletter Event als overage markiert wenn Großteil drüber). `markedUpCostCents = costCents × plan.aiMarkupMultiplier`. Felder `isOverage`, `markedUpCostCents` zu AICostEvent-Schema hinzufügen.
|
||
- Files: `prisma/schema.prisma` (zusätzliche Felder), `prisma/migrations/<ts>_ai_cost_overage_fields/migration.sql`, `apps/worker/src/handlers/ai-cost.persist-event.ts`
|
||
- Commit: `05/ai-cost-phase-3: mark overage events with markup`
|
||
4. **Nightly Overage-Report-Cron** — `apps/worker/src/cron/ai-overage-report.ts` läuft täglich um 02:00 UTC. Aggregiert seit letzter Submission `sum(markedUpCostCents) WHERE isOverage=true AND submittedToStripe=false` pro Tenant. Wenn Sum >= `plan.aiOverageBlockCents` → Stripe-Meter-Event `stripe.billing.meterEvents.create({ event_name, payload: { stripe_customer_id, value: ceil(sum / blockCents) * blockCents }, identifier: uuid })`. Idempotency-Identifier verhindert Doppel-Submission bei Retry. Markiere submittedEvents `submittedToStripe=true` + speichere `stripeMeterEventId`.
|
||
- Files: `apps/worker/src/cron/ai-overage-report.ts`, Test (mit Stripe-Mock)
|
||
- Commit: `05/ai-cost-phase-3: nightly overage report to stripe meter`
|
||
5. **Stripe-Setup-Doku + Configuration** — Doku-Section für Boilerplate-Betreiber: wie Stripe-Meter anlegen, wie Pricing-Tabelle mit graduated/tiered usage pricing aufsetzen, wie Customer-Subscription die Meter referenziert. Env-Keys `STRIPE_AI_OVERAGE_METER_ID` (optional, falls einzelner Meter für alle Plans; sonst pro Plan im Plan-Definition).
|
||
- Files: `docs/features/ai-cost-billing.md` (Phase-3-Section), `.env.example`
|
||
- Commit: `05/ai-cost-phase-3: stripe setup docs`
|
||
6. **Tenant-UI für Period-View mit Overage** — Erweiterung `/app/$tenant/settings/ai-usage`: Period-Bar zeigt Quota-Bereich (grün) + Overage-Bereich (gelb) + projected EOM (durchgezogene Linie für aktueller Spend, gestrichelt für Trend). Hinweis-Banner "X cents Overage wird auf nächster Stripe-Invoice abgerechnet". Per-Day-Bar-Chart mit isOverage-Färbung.
|
||
- Files: Update `apps/web/app/routes/app.$tenant.settings.ai-usage.tsx`
|
||
- Commit: `05/ai-cost-phase-3: tenant period view with overage visualization`
|
||
7. **Audit-Trail für Overage-Submissions** — neues Model `AIOverageSubmission { id, tenantId, periodStart, periodEnd, submittedCostCents, stripeMeterEventId, eventCount, createdAt }`. Wird vom Nightly-Cron pro Submission befüllt. Admin-UI `/admin/billing/ai-overage` zeigt alle Submissions pro Tenant + erlaubt manuelle Re-Submission bei Mismatch.
|
||
- Files: `prisma/schema.prisma`, Migration, `apps/web/app/routes/admin.billing.ai-overage.tsx`
|
||
- Commit: `05/ai-cost-phase-3: overage submission audit trail and admin ui`
|
||
8. **Reconciliation-Cron** — `apps/worker/src/cron/ai-overage-reconcile.ts` läuft monatlich am 5. eines Monats (nach Stripe-Invoice-Period-Close). Vergleicht für jeden Tenant: `sum(markedUpCostCents WHERE isOverage AND createdAt im Vor-Monat)` vs `sum(submittedCostCents WHERE periodStart im Vor-Monat)`. Bei Mismatch >1% → in-app Notification an Admin-Role + Sentry-Capture mit Tenant-Context. Doku-Runbook für Mismatch-Handling.
|
||
- Files: `apps/worker/src/cron/ai-overage-reconcile.ts`, Test, Runbook-Section in `docs/features/ai-cost-billing.md`
|
||
- Commit: `05/ai-cost-phase-3: monthly reconciliation cron with alerting`
|
||
9. **Per-Tenant-Override für Markup** — manche Enterprise-Deals haben Custom-Markup (1.2× statt 2.0×). `Tenant.aiMarkupOverride Float?` Migration; Persistence-Handler zieht Override falls vorhanden.
|
||
- Files: `prisma/schema.prisma`, Migration, Update Persistence-Handler, Admin-UI-Field
|
||
- Commit: `05/ai-cost-phase-3: per-tenant markup override`
|
||
10. **Doku-Finalisierung** — `docs/features/ai-cost-billing.md` Phase-3-Section: End-to-End-Flow-Diagramm (Call → Event → Overage-Mark → Cron → Stripe-Meter → Invoice), Operator-Runbook für Mismatch-Resolution, Stripe-Pricing-Strategie-Patterns (tiered vs graduated vs per-unit).
|
||
- Files: `docs/features/ai-cost-billing.md`
|
||
- Commit: `05/ai-cost-phase-3: docs finalization`
|
||
|
||
## Test Requirements
|
||
|
||
- **Unit-Tests:**
|
||
- Persistence-Handler: Event mit (currentSpend 95c, costCents 10c, quota 100c) → isOverage=true, markedUpCostCents=20 (mult 2.0)
|
||
- Overage-Report-Cron: Mock-Stripe, simuliere 250c unsubmitted Overage, blockCents 10 → submitMeterEvent mit value 250; submittedToStripe-Flag gesetzt
|
||
- Idempotency-Test: gleiche Cron-Run zweimal → keine doppelte Submission (identifier kollidiert in Stripe-Mock)
|
||
- Reconciliation-Cron: simulierter 5%-Mismatch → Notification geschrieben
|
||
- **Integration-Tests:**
|
||
- End-to-End: 200c LLM-Spend bei Quota 100c → 100c overage events → Cron-Run → Stripe-Mock empfängt Meter-Event 100c × 2.0 = 200c
|
||
- Per-Tenant-Markup-Override: gleicher Spend bei Override 1.5× → Stripe-Meter empfängt 150c
|
||
- Reconciliation: AICostEvent-Sum 100c, Stripe-Submitted 98c → keine Alert; 100c vs 90c → Alert
|
||
- **E2E-Tests (Playwright):**
|
||
- Tenant-User mit Overage-aktivem Plan: macht Calls über Quota → UI zeigt Overage-Bar gelb + Hinweis-Banner
|
||
- Admin öffnet `/admin/billing/ai-overage/$tenantId` → sieht Submission-History
|
||
- Admin setzt Per-Tenant-Markup-Override → Plan-Editor speichert + Audit-Eintrag sichtbar
|
||
- **Coverage-Target:** 90% für Cron-Jobs + Persistence-Handler-Updates, 70% UI
|
||
- **Test-Files:** `apps/worker/src/cron/ai-overage-report.test.ts`, `ai-overage-reconcile.test.ts`, E2E `apps/web/tests/e2e/ai-overage.spec.ts`
|
||
|
||
**NICHT als completed markieren bevor alle Tests grün sind.**
|
||
|
||
## Acceptance Criteria
|
||
|
||
- [ ] Plan-Schema mit `aiOverageStripeMeterId`, `aiMarkupMultiplier`, `aiOverageBlockCents`
|
||
- [ ] Admin-Plan-Editor zeigt + speichert neue Felder
|
||
- [ ] Migration für AICostEvent-Overage-Felder + AIOverageSubmission + Tenant.aiMarkupOverride applied
|
||
- [ ] Persistence-Handler markiert Overage + berechnet MarkedUpCost
|
||
- [ ] Nightly-Cron submitted Stripe-Meter-Events; Idempotency funktioniert
|
||
- [ ] Tenant-UI zeigt Quota + Overage + projected EOM
|
||
- [ ] AIOverageSubmission-Audit-Trail in DB + Admin-UI
|
||
- [ ] Reconciliation-Cron läuft monatlich; Alert-Pfad getestet
|
||
- [ ] Per-Tenant-Markup-Override funktional
|
||
- [ ] `docs/features/ai-cost-billing.md` Phase-3-Section komplett (inkl. Stripe-Setup-Anleitung + Runbook)
|
||
- [ ] `.env.example` enthält neue Keys
|
||
- [ ] `pnpm lint`, `pnpm typecheck`, `pnpm test` grün
|
||
- [ ] Coverage-Gates erfüllt
|
||
- [ ] Atomic Commits gepushed auf `agent/05-ai-cost-phase-3`
|
||
- [ ] Detail-MD Progress-Log aktualisiert
|
||
- [ ] PROGRESS.md Status auf `completed`
|
||
|
||
## Rollback Strategy
|
||
|
||
- **Stripe-Submission-Killswitch:** Env `AI_OVERAGE_SUBMISSION_ENABLED=false` deaktiviert Cron-Submit (Events werden weiter markiert, aber nicht an Stripe gemeldet). Korrekturpfad: Backfill-Run nach Aktivierung.
|
||
- **DB-Migration:** alle neuen Felder nullable / default-Werte → Rollback unschädlich.
|
||
- **Markup-Reset:** Multiplier auf 1.0 in Plan-Definition + Re-Mark-Migration (Script) für noch-nicht-submitted Events.
|
||
- **Stripe-Meter-Cleanup:** wenn falsche Meter-Events submitted wurden, manueller Stripe-Adjustment + AIOverageSubmission-Cancel-Flag setzen.
|
||
|
||
## Security Notes
|
||
|
||
- **Doppel-Charging-Prevention:** Idempotency-Identifier (UUID pro Submission) ist Pflicht — Stripe deduped auf Identifier. Tests müssen explizit Retry-Pfad verifizieren.
|
||
- **Markup-Tampering:** Plan-Editor nur für Admin-Role; jede Markup-Mutation in AuditEvent.
|
||
- **Tenant-Exposure:** Tenant darf provider-internal cost nicht sehen — Tenant-UI zeigt nur `markedUpCostCents`, nie `costCents`. Loader strippt Felder bewusst.
|
||
- **Reconciliation-Drift:** systemic >1%-Mismatch könnte Indikator für Persistence-Bug sein → Alert ist Sentry-Capture mit Tenant-ID, Operator muss ASAP reagieren. Runbook beschreibt Diagnose-Schritte.
|
||
- **GDPR/Invoice-Daten:** Stripe-Meter-Events enthalten nur Aggregate-Cents, keine PII; keine zusätzliche DPA-Klausel nötig über existierende Stripe-DPA hinaus.
|
||
|
||
## Subagent Recommendations
|
||
|
||
- **Planning:** `Plan`-Subagent mit Detail-MD-Input.
|
||
- **Research:** `Explore`-Subagent: "Existierende Stripe-Integration in apps/web/app/utils/stripe.server.ts analysieren — gibt es schon Webhook-Handler für Subscription-Updates? Welche Stripe-API-Version ist gepinnt?"
|
||
- **Implementation Step 1-3 (Schema + UI + Mark):** `general-purpose`-Subagent.
|
||
- **Implementation Step 4 (Cron Submit):** `general-purpose`-Subagent mit explizitem Brief zu Idempotency + Stripe-Mock im Test.
|
||
- **Implementation Step 5-7 (Docs + UI + Audit):** `general-purpose`-Subagent.
|
||
- **Implementation Step 8-9 (Reconcile + Override):** `general-purpose`-Subagent.
|
||
- **Tests:** separater `general-purpose`-Subagent für E2E + Stripe-Mock-Tests.
|
||
- **Verification:** `gsd-verifier`.
|
||
- **Code-Review:** `gsd-code-reviewer`.
|
||
|
||
## Staging-Deployment-Steps
|
||
|
||
1. Branch `agent/05-ai-cost-phase-3` pushen → Coolify-Auto-Deploy
|
||
2. Stripe-Test-Account: Meter anlegen (`ai_overage_cents`), Pricing-Tabelle definieren, Test-Subscription mit Meter-Bindung
|
||
3. `STRIPE_AI_OVERAGE_METER_ID` als Staging-Env setzen, Test-Plan-Definitions linken
|
||
4. Migration `pnpm prisma migrate deploy`
|
||
5. Staging-Test:
|
||
- Test-Tenant mit aktivem Stripe-Subscription auf Test-Plan
|
||
- LLM-Calls bis 150% Quota
|
||
- Cron-Job manuell ausführen → Stripe-Test-Dashboard zeigt Meter-Event
|
||
- Test-Invoice generieren → Overage-Line-Item sichtbar
|
||
- UI: Tenant sieht Overage-Bar + Banner
|
||
6. Reconciliation-Test: AICostEvent manuell um 5% verfälschen → Reconcile-Cron alerted
|
||
|
||
## Deviations from Plan
|
||
|
||
- **Stripe SDK 13.x — Usage-Records API, not Meters API.** B-001 blocks
|
||
upgrading to Stripe SDK 14+. Phase-3 ships against
|
||
`stripe.subscriptionItems.createUsageRecord(itemId, { quantity,
|
||
action, timestamp }, { idempotencyKey })`. The reporter wraps the
|
||
call behind `StripeUsageRecordSubmitter` so swapping to
|
||
`stripe.billing.meterEvents.create(...)` post-upgrade is a one-file
|
||
change. `aiOverageStripeMeterId` is persisted today so operators can
|
||
configure both fields up front — see migration steps in
|
||
`docs/features/ai-cost-billing.md` "Future migration to Stripe
|
||
Meters API (SDK 14+)".
|
||
- **Per-tenant override blob.** Plan asked for a dedicated
|
||
`Tenant.aiMarkupOverride Float?` column; we extend the existing
|
||
`Tenant.aiQuotaOverrideJson` instead. Same access path; zero
|
||
migration surface; matches the Phase-2 parser style. The Phase-3
|
||
parser in `@toolctrl/entitlements` is shape-tolerant so both
|
||
blob keys coexist.
|
||
- **Admin UI scope.** The `/admin/settings/pricing/ai-overage` route
|
||
surfaces the per-plan defaults from
|
||
`standardAIOverageDefaults` read-only and audit-logs would-be
|
||
edits. Defaults stay in code until a follow-up moves them into a
|
||
DB table (Phase-2 quota defaults share the same pattern). Per-
|
||
tenant overrides via `Tenant.aiQuotaOverrideJson` are fully live.
|
||
- **AIOverageSubmission audit-trail model deferred.** Spec asks for
|
||
a dedicated `AIOverageSubmission` model; we stamp
|
||
`AICostEvent.stripeReportedAt` + log structured payloads instead
|
||
(`stripeUsageRecordId` included). Sufficient for now; a follow-up
|
||
can promote the structured log into a queryable table.
|
||
- **Worker bootstrap wiring.** This phase ships the handler + cron
|
||
driver code with pure-by-deps tests. Hooking them into the
|
||
worker's BullMQ scheduler is operator-side wiring (matches the
|
||
pattern set by Phase-1 #54 and Phase-2 #55, both of which
|
||
similarly deferred bootstrap to the worker-bootstrap rewrite).
|
||
|
||
## Progress Log
|
||
|
||
- 2026-05-17 — `c0b48d9e` — plan schema with overage and markup config (`@toolctrl/entitlements` ai-overage module + 23 unit tests)
|
||
- 2026-05-17 — `ca89ea43` — extend AICostEvent with overage markup fields (Prisma migration `markedUpCostCents` + `stripeReportedAt`)
|
||
- 2026-05-17 — `e15b195d` — regenerate Prisma client for overage fields
|
||
- 2026-05-17 — `27225add` — mark overage events in persist handler (overageLookup dep + 6 new tests)
|
||
- 2026-05-17 — `ff8de906` — add Stripe overage reporting worker (`ai-cost-stripe-report.ts` + 13 tests, SDK 13.x Usage-Records API)
|
||
- 2026-05-17 — `5a32d669` — admin UI for plan markup and overage config (`/admin/settings/pricing/ai-overage` + audit log)
|
||
- 2026-05-17 — `544cb99e` — bump i18n baseline for admin overage editor (jsx-text +3, user-facing-prop +5)
|
||
- 2026-05-17 — `3739c174` — tenant UI shows period forecast with overage (`/app/:tenant/settings/ai-usage`)
|
||
- 2026-05-17 — `cc778b8c` — reconciliation cron compares cost-events vs Stripe meter (9 tests)
|
||
- 2026-05-17 — `a546c184` — CLI helpers for Stripe AI-cost ops (`toolctrl-cli ai-cost {report-overage,reconcile,show-billing}`)
|
||
- 2026-05-17 — `a4b23906` — env + documentation (`.env.example` killswitch + `docs/features/ai-cost-billing.md`)
|
||
- 2026-05-17 — `dbbc120d` — sync hardcoded-strings test fixtures to bumped baseline
|