Track HiveGrid Cloud planning reference

This commit is contained in:
david committed 2026-06-26 20:48:01 +02:00
1 parent b3d5ea6764
commit e341c5938a
5784 files changed
+1104929 -3

No files matched your search

@@ -0,0 +1,202 @@
# AI-Cost Billing — Stripe Overage (Phase 3)
> Status: daily Stripe overage reporting is worker-wired on
> `worker-general` (Stripe SDK 19.x + Billing Meter Events). Stripe meter
> setup and tenant/plan configuration are operator-owned. Monthly
> reconciliation is worker-wired on `worker-general` when Redis and
> `STRIPE_SK` are configured; the CLI remains an offline preview path.
> See "Operator runbook" below.
## Overview
The Phase 3 engine converts per-tenant AI overage (every cent of provider
spend beyond the plan's `aiQuotaCentsPerMonth`) into Stripe invoice line
items, with a markup multiplier so the boilerplate finances its own
hosting + engineering overhead.
End-to-end flow:
```
executeLLMCall(...) ─► outbox ─► ai-cost.persist (worker)
├─ resolveTenantAIQuota
├─ resolveTenantAIOverage
├─ if projectedSpend > quota:
│ isOverage = true
│ markedUpCostCents = costCents × mult
└─ write AICostEvent
│
▼
ai-cost-stripe-report (nightly cron)
├─ aggregate WHERE isOverage = true
│ AND stripeReportedAt IS NULL
├─ round UP to plan.aiOverageBlockCents
├─ stripe.billing.meterEvents
│ .create({
│ event_name,
│ payload: { stripe_customer_id, value }
│ }, { idempotencyKey })
└─ stamp stripeReportedAt = now
│
▼
Stripe invoice (next period close)
│
▼
ai-cost-reconciliation (monthly worker cron)
compares internal vs
stripe.billing.meters
.listEventSummaries
alert when diff > 1%
```
## Plan markup defaults
| Plan | Multiplier | Block (c) | Default state |
| -------- | ---------- | --------- | ----------------------------------------------- |
| Starter | 1.0× | 10 | No overage (quota = 0 forces per-tenant unlock) |
| Pro | 2.0× | 10 | Overage on once Stripe meter is configured |
| Business | 1.5× | 10 | Overage on once Stripe meter is configured |
Defaults live in
[`packages/entitlements/src/ai-overage.ts`](../../packages/entitlements/src/ai-overage.ts).
Per-tenant overrides live on `Tenant.aiQuotaOverrideJson` (the same Json
column used by Phase 2 quotas; the override parser is shape-tolerant so
both blobs coexist).
## Operator runbook
### 1. Create the Stripe meter / price
This repo runs Stripe SDK 19 with Billing Meter Events. Setup:
1. Stripe dashboard → Products → New product.
2. Add a **metered** recurring price backed by a Billing Meter. Choose either:
- **Per unit** (e.g. `0.001 EUR / cent`) — simplest, what Phase 3
unit-tests assume.
- **Tiered (graduated)** — preferred when bigger tenants should get
volume discounts. The reporter submits `quantity = cents` either
way.
3. Copy the Billing Meter event name (default `ai-overage`) and attach the
price to the subscriptions of the relevant plan (Subscription → Add
product → choose the metered price you created).
4. Admin → AI Overage Pricing (`/admin/settings/pricing/ai-overage`) —
paste the meter event name into the Stripe Meter field for that plan.
The action handler audit-logs the change.
5. Stripe → API keys → set `STRIPE_SK` (test mode for staging,
`sk_live_...` for production).
### 2. Configure killswitch + cron
- `.env`: ensure `AI_OVERAGE_SUBMISSION_ENABLED` is unset or `true`.
- Worker bootstrap: `apps/worker/src/index.ts` registers
`registerAICostStripeReportCron(...)` on `worker-general` when Redis is
configured. The maintenance handler reads the killswitch and bails early
when set.
- Monthly reconciliation: `apps/worker/src/index.ts` registers
`registerAICostReconciliationCron(...)` on `worker-general` when Redis,
`STRIPE_SK`, and `AI_COST_RECONCILIATION_ENABLED!=false` are present. It
runs on the dedicated `toolctrl.ai-cost-reconciliation` queue on the 5th
of each month at 03:00 UTC.
### 3. Verify on staging
```
toolctrl-cli ai-cost show-billing --tenant <tenantId>
toolctrl-cli ai-cost report-overage --tenant <tenantId> # dry-run
toolctrl-cli ai-cost reconcile --tenant <tenantId> --month 2026-05
```
End-to-end smoke:
1. Set `Tenant.aiQuotaOverrideJson = { aiQuotaCentsPerMonth: 100 }` on
the test tenant.
2. Drive `executeLLMCall` until total cost exceeds 100c. Confirm
`AICostEvent.isOverage = true` and `markedUpCostCents = 2x cost` for
Pro.
3. Trigger the nightly cron. Confirm a meter event appears on the
Stripe dashboard for the configured meter/customer.
4. Wait until the next Stripe invoice period — verify the overage
appears as an invoice line item.
## Idempotency
The reporter constructs the Stripe idempotency key deterministically:
```
ai-overage:<tenantId>:<periodKey>:<roundedCents>:<eventCount>
```
Retries with identical inputs collapse to a single Stripe-side charge.
The cron's per-tenant failure isolation logs the failure, leaves
`stripeReportedAt` NULL, and a later run retries.
## Reconciliation procedure
The worker reconciliation path sums internal overage
`markedUpCostCents WHERE createdAt IN <prior-month>` and compares it to
Stripe Billing Meter event summaries for the same customer + meter window.
Diffs above 1% are logged through the worker notifier. The CLI preview still
sums internal totals offline and is useful before enabling `STRIPE_SK` in a
worker environment.
Remaining precision gap: the reporter rounds each submission up to the
tenant's `aiOverageBlockCents`, but the database does not yet persist a
submission ledger with the rounded quantity per Stripe meter event. The
worker therefore reconciles raw expected overage cents against Stripe's
meter summaries; exact penny-level reconciliation needs the planned
`AIOverageSubmission` audit table.
Manual investigation steps on mismatch:
1. `toolctrl-cli ai-cost reconcile --tenant <tenantId> --month <m>` —
inspect internal totals offline.
2. Stripe dashboard → Billing Meter → event summaries → diff the reported
quantities.
3. If a row in `AICostEvent` is missing `stripeReportedAt` but Stripe
shows it, the most common cause is a previous `markEventsReported`
crashed between the Stripe success and DB stamp; rerun the cron and
stamp manually if necessary.
## Security
Per the Phase-3 spec security notes:
- Tenant UI shows `markedUpCostCents` only — provider `costCents` is
stripped from the loader payload.
- Every markup mutation flows through the audit log
(`createAdminLog` → `AuditEvent`).
- Submission carries no PII — only aggregate cents.
## Deviations from plan
- **Stripe SDK migration complete.** Older iterations used legacy
subscription-item usage records. The current implementation uses
`stripe.billing.meterEvents.create(...)` behind the
`StripeMeterEventSubmitter` interface and persists only
`aiOverageStripeMeterId` for plan configuration.
- **Plan defaults edit-via-code.** The admin UI surfaces the
per-plan defaults read-only and audit-logs would-be changes; the
canonical numbers stay in `standardAIOverageDefaults` until a
follow-up moves them into a DB table. Per-tenant overrides on
`Tenant.aiQuotaOverrideJson` are fully live today and recommended
for enterprise deals.
- **Per-tenant override field name.** Phase-3 spec called for a
dedicated `Tenant.aiMarkupOverride Float?` column; we instead extend
the existing `aiQuotaOverrideJson` blob with new optional keys. This
keeps the migration surface zero and matches the Phase-2 override
parser style.
## Stripe Meters API (SDK 19, current)
The repo runs Stripe SDK 19 (`apiVersion: "2025-10-29.clover"`).
Overage submission goes through `stripe.billing.meterEvents.create`
with payload `{ stripe_customer_id, value: rounded }` and the
operator-provisioned Meter event name (default `ai-overage`). The
production wireup lives in
`apps/worker/src/jobs/ai-cost-stripe-submitter.ts`; the cron is
registered by `apps/worker/src/jobs/ai-cost-stripe-report-cron.ts`.
Per-tenant configuration uses `aiOverageStripeMeterId` only; the
legacy `aiOverageStripeSubscriptionItemId` field was retired.
For reconciliation use `stripe.billing.meters.listEventSummaries`
with `value_grouping_window=hour` and an explicit `customer` +
`start_time` / `end_time` window.