Files
HiveGrid-Wireframes/hivegrid-cloud/docs/features/marketing-funnel.md
T

405 lines
18 KiB
Markdown

# Marketing-Funnel Overview
This page is the canonical overview for the shipped lead-capture and
self-service conversion foundation. Email sequences are covered separately in
`docs/features/email-sequences.md`.
This page also keeps the marketing-outbound boundary explicit: the
`packages/marketing-outbound` adapters and
`apps/worker/src/jobs/marketing-outbound.ts` handler are a shipped general
handler foundation, but the general subscriber is not bootstrapped from
`apps/worker/src/index.ts`. That foundation is separate from the
`email-marketing` CRM-contact campaign plugin candidate in
`docs/feature-decisions/email-marketing-decision-report.md`.
The marketing-funnel ships a four-layer pipeline that lets external landing
pages, ads, and blog CTAs feed structured leads into the operator-side admin
inbox without an external CRM.
| Layer | Component |
|------------------|------------------------------------------------------------|
| 1. Data model | `Lead` + `LeadEvent` Prisma tables (`prisma/schema/marketingLeads.prisma`) |
| 2. Ingestion API | `POST /api/leads/capture` (`apps/web/app/routes/api.leads.capture.tsx`) |
| 3. SDK | `apps/web/public/embed/marketing.js` (origin-served mini-snippet; CDN-ready while A7/B2 remains backlog) |
| 4. Admin UI | `/admin/marketing-leads` (list) + `/admin/marketing-leads/:id` (detail) |
## Status machine
Leads move through seven states:
```
new → contacted → qualified → trial → customer
↘ churned
↘ disqualified
```
Transitions are recorded as `LeadEvent` rows of type `status-changed` for the
activity-timeline in the detail-view. Operators move leads forward (or to a
terminal state) from the detail page. The `score`, `notes`, and `convertedAt`
columns are operator-managed.
## Attribution
Every Lead carries first-touch attribution columns populated by the SDK on
the initial visit and persisted in localStorage so subsequent page visits
attribute to the same source:
| Column | Source |
|---------------|-----------------------------------------------------|
| `utmSource` | `?utm_source=…` |
| `utmMedium` | `?utm_medium=…` |
| `utmCampaign` | `?utm_campaign=…` |
| `utmTerm` | `?utm_term=…` |
| `utmContent` | `?utm_content=…` |
| `gclid` | `?gclid=…` (Google Ads click identifier) |
| `fbclid` | `?fbclid=…` (Meta Ads click identifier) |
| `referrer` | `document.referrer` |
| `landingPage` | `location.href` at the first visit |
| `firstSeenAt` | `new Date()` on initial insert (never overwritten) |
| `lastTouchAt` | `new Date()` on every re-capture |
`source` is operator-supplied (via the form's `data-source` attribute or the
SDK call's `source` field) and identifies the surface — `"landing-pro"`,
`"blog-cta"`, `"webinar-form"` — distinct from the paid-channel `utmSource`.
## Anti-spam (defence in depth)
The capture endpoint stacks four layers, each weakening different bot
classes; none of them is sufficient alone.
1. **Origin allowlist** — `MARKETING_ALLOWED_ORIGINS` (CSV with `*.host`
wildcards). 403 on miss. Defeats browser-based form-spam from
non-whitelisted pages; can be spoofed by server-to-server bots.
2. **Honeypot** — the SDK injects a hidden `website_url` field. When the
server sees it filled, it responds with a silent 204 and inserts nothing.
80%+ of crude form-fillers fall here.
3. **Sliding-window rate-limit** — 5/min per IP (configurable via
`MARKETING_RATE_LIMIT_PER_MIN`) and 1/min per email. The per-email limit
doubles as a re-capture dedupe so the LeadEvent timeline doesn't flood.
4. **Cloudflare Turnstile** (optional) — gated behind the
`marketing-funnel-turnstile` feature flag. When enabled, callers must
include a valid `turnstileToken`; verification hits the Cloudflare API.
Successful responses are intentionally identical for new and duplicate
emails (empty body, 200 OK) so bots cannot enumerate the customer list.
## API contract
### `POST /api/leads/capture`
Public, CORS-enabled. Accepts `application/json` or
`application/x-www-form-urlencoded`. Body:
```json
{
"email": "alice@example.com",
"firstName": "Alice",
"lastName": "Liddell",
"phone": "+44 …",
"company": "Acme",
"source": "landing-pro",
"attribution": {
"utmSource": "google",
"utmCampaign": "spring-launch",
"gclid": "EAIaIQ…",
"referrer": "https://google.com/",
"landingPage": "https://example.com/landing"
},
"turnstileToken": "0.OOOO…",
"website_url": ""
}
```
Responses:
| Status | Meaning |
|--------|--------------------------------------------------|
| 200 | Captured (or silently de-duplicated). Empty body.|
| 204 | Honeypot triggered or per-email rate-limit hit. |
| 400 | Invalid payload shape. |
| 403 | Origin not allowlisted or Turnstile failed. |
| 413 | Body exceeds 32 KiB. |
| 429 | Per-IP rate-limit exceeded. |
### `OPTIONS /api/leads/capture`
CORS preflight. Echoes the matched origin in `Access-Control-Allow-Origin`.
## SDK (`marketing.js`)
Drop the script tag onto every marketing page:
```html
<script src="https://cdn.example.com/embed/marketing.js"
data-api-url="https://app.example.com" defer></script>
```
The script self-bootstraps on `DOMContentLoaded`:
- Reads `?utm_source/medium/campaign/term/content`, `?gclid`, `?fbclid` from
the URL and merges them with the stored first-touch attribution.
- Persists merged attribution to `localStorage["__tcm_attr"]` so multi-page
flows hit the same lead source.
- Auto-binds any `<form data-toolctrl-lead>` on the page — submitting
POSTs to `/api/leads/capture` instead of the form's action.
Imperative API exposed as `window.toolctrlMarketing`:
```js
toolctrlMarketing.captureLead({
email: "alice@example.com",
firstName: "Alice",
source: "blog-cta",
});
toolctrlMarketing.bindFormById("lead-form");
toolctrlMarketing.bindFormsByAttr();
toolctrlMarketing.getAttribution(); // → merged first-touch object
toolctrlMarketing.resetAttribution(); // wipes localStorage cache
```
Custom events on bound forms:
- `tcm:captured` — `{ ok: boolean, status: number }`
- `tcm:error` — `{ error: string }`
The SDK is hand-minified to stay under the 3 KB gzipped budget (currently
~2 KB) and ships from `apps/web/public/embed/marketing.js`.
## Admin UI
- **List** at `/admin/marketing-leads` — filter by status (seven badge
shortcuts), source, or search by email/name/company. Paginated 25 per
page. Permission: `admin.marketing.leads.view`.
- **Detail** at `/admin/marketing-leads/:id` — profile + attribution
panels, status transition buttons (every state reachable via one click),
inline notes editor with history, convert-to-tenant action, and the
LeadEvent activity timeline. Permission: `admin.marketing.leads.manage`.
## CLI
```
toolctrl-cli leads list [--status new] [--limit 50]
toolctrl-cli leads export [--out leads.csv] [--status qualified]
toolctrl-cli leads delete --id <leadId> # DSGVO right-to-be-forgotten
```
## Configuration
| Env | Default | Purpose |
|--------------------------------|---------|-------------------------------------------------------------|
| `MARKETING_ALLOWED_ORIGINS` | (empty) | CSV of allowed origins for capture endpoint. Fail-closed. |
| `MARKETING_RATE_LIMIT_PER_MIN` | 5 | Per-IP sliding-window submission limit. |
| `TURNSTILE_SITE_KEY` | — | Cloudflare Turnstile site key (shared with form-widget). |
| `TURNSTILE_SECRET_KEY` | — | Cloudflare Turnstile secret (shared with form-widget). |
The `marketing-funnel-turnstile` feature flag must be enabled for Turnstile
to actually gate submissions; the secret alone is not enough.
## Rollback
- `MARKETING_ALLOWED_ORIGINS=` empties the allowlist → all browser captures
are rejected with 403 while leaving the admin UI usable for triage.
- Drop the `Lead` + `LeadEvent` tables to wipe captured data (no other
tables reference them today). Export with `toolctrl-cli leads export`
first.
---
# Phase 2 — Self-Service Conversion + Outbound Webhooks + Attribution
Phase 2 (task #79) extends Phase 1's inbox with three independently-shippable
layers:
1. **Self-service conversion deep-links** — a signed lead-token in the URL
pre-fills the registration form. The normal registration service creates
the User + Tenant, then records the Lead conversion against that Tenant:
`Lead.convertedToTenantId`, a `TenantAcquisition` snapshot, and the
marketing outbox event are written in one transaction.
2. **Outbound webhook foundation** — six pluggable adapters (Slack, Discord,
Mailchimp, HubSpot, Brevo, custom HTTPS) plus the
`apps/worker/src/jobs/marketing-outbound.ts` handler contract. General
marketing integration delivery is still a production follow-up because
that handler is not currently bootstrapped from `apps/worker/src/index.ts`.
3. **Attribution reports** — `/admin/marketing-funnels` aggregates leads
+ acquisitions by source, campaign, utm_source, conversionPath. Mirror
via `toolctrl-cli marketing funnel|attribution`.
## Signed lead-token
`signLeadToken({ leadId, planId, expiresInDays = 7 })` produces a token of
the shape `v1.<base64url(payload)>.<base64url(hmac-sha256)>`. The HMAC
key is `LEAD_CONVERSION_SECRET` (min 32 bytes). `verifyLeadToken(token)`
returns the payload when the signature matches AND the token has not
expired — otherwise `null`. Both functions fail-closed when the secret
env is missing.
Email pre-filled from a Lead is rendered read-only on the register form
to prevent lead-hijacking ("I use your token but with my email"). The signed
token is also submitted as a hidden field and verified again in the register
action. If email verification is enabled, the verified conversion context is
stored on the pending `Registration` row and replayed from `/verify/:token`
after the visitor sets a password. To change the email the visitor must
abandon the link and start a fresh signup.
## Atomic conversion service
`convertLeadToTenant({ leadId, tenant: { name, slug, inboundAddress }, ... })`
runs inside a single Prisma `$transaction`:
1. `tenant.create` (slug + inbound address pre-resolved by caller).
2. `lead.update` → `status` advances to `trial`/`customer`,
`convertedToTenantId` + `convertedAt` populated.
3. `leadEvent.create` of type `converted` with full conversion payload.
4. `tenantAcquisition.create` snapshot copying UTM/referrer/landingPage
from the Lead.
5. `publishOutboxEvent` with type `marketing.lead.convertedtotrial` or
`marketing.lead.convertedtocustomer` and dispatchKey
`lead-convert:<leadId>` (idempotent on retry).
Already-converted leads (`convertedToTenantId !== null`) short-circuit
and return the existing tenant.
For the self-service register flow, the already-created Tenant is linked via
`recordExistingTenantLeadConversion({ leadId, tenantId, ... })`. It reuses the
same conversion event/outbox payload and upserts the `TenantAcquisition` row
instead of creating a second Tenant.
## Admin manual override
`/admin/marketing-leads/<id>` exposes a "Create Tenant" form. Operator
can override the tenant name + supply an optional planId. The action
calls `convertLeadToTenant({ conversionPath: "admin-override", actorId })`.
A native confirm dialog guards accidental clicks.
## Outbound webhook adapters
`packages/marketing-outbound` ships six adapters under
`src/adapters/*.ts`:
| Adapter | Config keys (encrypted at rest) | Notes |
|---------|----------------------------------------------------------------|-------|
| slack | `webhookUrl`, `channel?` | Block-Kit message. |
| discord | `webhookUrl`, `username?` | Embed with color-coded event. |
| mailchimp | `apiKey`, `listId`, `status?`, `tags?` | PUT to lists/{id}/members/{md5(email)}. |
| hubspot | `accessToken` | POST /crm/v3/objects/contacts; 409 = idempotent. |
| brevo | `apiKey`, `listIds?` | POST /v3/contacts with `updateEnabled: true`. |
| webhook | `url`, `signingSecret?`, `headers?` | POSTs canonical envelope; optional HMAC-SHA256 sig. |
All adapters share a single `adapterFetch` helper that enforces:
- HTTPS only (rejects http: / localhost / RFC1918 / 169.254.* /
`metadata.google.internal` / `.internal` / `.local` / `.lan`).
- 10 s default timeout (AbortController).
- Returns structured `AdapterResult { ok, status?, errorTag? }` so the
worker can categorise failures consistently.
The `ctx.maskPii.email` toggle masks the local part of the email
(`patrick@example.com → ***@example.com`) before any payload leaves the
host. Use it for data-egress-restricted tenants.
## Apps/worker handler status
`apps/worker/src/jobs/marketing-outbound.ts` defines a pure handler contract
for marketing integration delivery, but it does not own or subscribe to
`outbox.marketing`. That queue is currently used by the email-sequence trigger
subscriber when `MARKETING_SEQUENCES_ENABLED=true`; this does not imply
Slack/Discord/Mailchimp/HubSpot/Brevo/custom webhook delivery is active.
Treat the adapters, handler, admin UI, and env flag as a shipped foundation
until a bootstrap module wires a distinct general marketing integration
subscriber into the worker runtime with explicit ownership tests.
The handler accepts only these event types today:
`marketing.lead.captured`, `marketing.lead.qualified`,
`marketing.lead.convertedtotrial`, and
`marketing.lead.convertedtocustomer`. New `marketing.lead.*` events require
an update to `MARKETING_LEAD_EVENT_TYPES` plus tests. For each accepted event:
1. Parse + validate envelope (`parseMarketingOutboundJob`).
2. Bail out when `MARKETING_WEBHOOKS_ENABLED=false`.
3. Fetch every enabled `MarketingIntegration` whose `triggerEvents`
includes the eventType.
4. For each integration: decrypt config, resolve adapter via
`resolveAdapter(type)`, call `adapter.send`. One integration's
failure does not affect siblings.
5. Persist diagnostics (`lastDeliveredAt`, `lastError`) on the
integration row so the admin UI can surface them.
## Admin integrations UI
`/admin/marketing-integrations` (Phase 05 #79). CRUD list with:
- Add form (type picker + name + trigger-events multi-select + config
JSON textarea).
- Per-integration enable/disable toggle.
- Delete with native confirm dialog.
- Last-delivered / last-error display.
- Config display masked (`***` for `webhookUrl`, `apiKey`,
`accessToken`, `signingSecret`).
Secrets are encrypted with the existing AES-256-GCM CryptoApi helper
(`apps/web/app/utils/api/.server/CryptoApi.ts`) keyed off `CRYPTO_SECRET`.
## Attribution reports
`/admin/marketing-funnels` (Phase 05 #79). Top-line cards (leads
captured, conversions, conversion rate) plus four breakdowns: campaign,
utm_source, source-label, conversionPath. Reads via `dbRead`
(replica-routed). CLI mirror:
- `toolctrl-cli marketing funnel --days 30`
- `toolctrl-cli marketing attribution --dimension campaign --days 30 --limit 25`
## Configuration (Phase 2)
| Env | Default | Purpose |
|------------------------------|---------|------------------------------------------------------------|
| `LEAD_CONVERSION_SECRET` | — | HMAC key for signed lead-tokens (min 32 bytes). |
| `LEAD_CONVERSION_ENABLED` | `true` | Set `false` to disable the conversion flow without DB rollback. |
| `MARKETING_WEBHOOKS_ENABLED` | `true` | Handler-level kill switch once the general marketing integration subscriber is bootstrapped. It does not start that subscriber by itself. |
## Webhook payload (custom adapter)
```json
{
"schema": "marketing.lead.v1",
"eventType": "marketing.lead.convertedtotrial",
"occurredAt": "2026-05-17T00:00:00.000Z",
"lead": {
"id": "lead_abc",
"email": "patrick@example.com",
"firstName": "Patrick",
"lastName": "Doe",
"company": "Acme",
"source": "landing-pro",
"utm": { "source": "google", "medium": "cpc", "campaign": "q2-launch" },
"referrer": "https://google.com",
"landingPage": "/pricing"
},
"conversion": {
"tenantId": "tenant_xyz",
"targetStatus": "trial",
"conversionPath": "self-service",
"planId": "plan_pro"
}
}
```
When `signingSecret` is set, the receiver verifies
`x-marketing-signature: sha256=<hex>` against
`hmac_sha256(body, signingSecret)`.
## Rollback (Phase 2)
- `LEAD_CONVERSION_ENABLED=false` — register loader ignores the token
query params; admin override button hidden.
- `MARKETING_WEBHOOKS_ENABLED=false` — once the general marketing
integration subscriber is bootstrapped, the worker handler short-circuits.
Today it is only a handler-level kill switch for the shipped foundation.
Outbox events accumulate; raise outbox retention if you plan a long
pause.
- Per-integration disable toggle in the admin UI — faulty integrations
can be parked without a deploy.
- `TenantAcquisition` + `MarketingIntegration` tables can be dropped
(no FK in to them from Tenant), but you'd lose attribution history.