405 lines
18 KiB
Markdown
405 lines
18 KiB
Markdown
# Marketing-Funnel Overview
|
|
|
|
This page is the canonical overview for the shipped lead-capture and
|
|
self-service conversion foundation. Email sequences are covered separately in
|
|
`docs/features/email-sequences.md`.
|
|
|
|
This page also keeps the marketing-outbound boundary explicit: the
|
|
`packages/marketing-outbound` adapters and
|
|
`apps/worker/src/jobs/marketing-outbound.ts` handler are a shipped general
|
|
handler foundation, but the general subscriber is not bootstrapped from
|
|
`apps/worker/src/index.ts`. That foundation is separate from the
|
|
`email-marketing` CRM-contact campaign plugin candidate in
|
|
`docs/feature-decisions/email-marketing-decision-report.md`.
|
|
|
|
The marketing-funnel ships a four-layer pipeline that lets external landing
|
|
pages, ads, and blog CTAs feed structured leads into the operator-side admin
|
|
inbox without an external CRM.
|
|
|
|
| Layer | Component |
|
|
|------------------|------------------------------------------------------------|
|
|
| 1. Data model | `Lead` + `LeadEvent` Prisma tables (`prisma/schema/marketingLeads.prisma`) |
|
|
| 2. Ingestion API | `POST /api/leads/capture` (`apps/web/app/routes/api.leads.capture.tsx`) |
|
|
| 3. SDK | `apps/web/public/embed/marketing.js` (origin-served mini-snippet; CDN-ready while A7/B2 remains backlog) |
|
|
| 4. Admin UI | `/admin/marketing-leads` (list) + `/admin/marketing-leads/:id` (detail) |
|
|
|
|
## Status machine
|
|
|
|
Leads move through seven states:
|
|
|
|
```
|
|
new → contacted → qualified → trial → customer
|
|
↘ churned
|
|
↘ disqualified
|
|
```
|
|
|
|
Transitions are recorded as `LeadEvent` rows of type `status-changed` for the
|
|
activity-timeline in the detail-view. Operators move leads forward (or to a
|
|
terminal state) from the detail page. The `score`, `notes`, and `convertedAt`
|
|
columns are operator-managed.
|
|
|
|
## Attribution
|
|
|
|
Every Lead carries first-touch attribution columns populated by the SDK on
|
|
the initial visit and persisted in localStorage so subsequent page visits
|
|
attribute to the same source:
|
|
|
|
| Column | Source |
|
|
|---------------|-----------------------------------------------------|
|
|
| `utmSource` | `?utm_source=…` |
|
|
| `utmMedium` | `?utm_medium=…` |
|
|
| `utmCampaign` | `?utm_campaign=…` |
|
|
| `utmTerm` | `?utm_term=…` |
|
|
| `utmContent` | `?utm_content=…` |
|
|
| `gclid` | `?gclid=…` (Google Ads click identifier) |
|
|
| `fbclid` | `?fbclid=…` (Meta Ads click identifier) |
|
|
| `referrer` | `document.referrer` |
|
|
| `landingPage` | `location.href` at the first visit |
|
|
| `firstSeenAt` | `new Date()` on initial insert (never overwritten) |
|
|
| `lastTouchAt` | `new Date()` on every re-capture |
|
|
|
|
`source` is operator-supplied (via the form's `data-source` attribute or the
|
|
SDK call's `source` field) and identifies the surface — `"landing-pro"`,
|
|
`"blog-cta"`, `"webinar-form"` — distinct from the paid-channel `utmSource`.
|
|
|
|
## Anti-spam (defence in depth)
|
|
|
|
The capture endpoint stacks four layers, each weakening different bot
|
|
classes; none of them is sufficient alone.
|
|
|
|
1. **Origin allowlist** — `MARKETING_ALLOWED_ORIGINS` (CSV with `*.host`
|
|
wildcards). 403 on miss. Defeats browser-based form-spam from
|
|
non-whitelisted pages; can be spoofed by server-to-server bots.
|
|
2. **Honeypot** — the SDK injects a hidden `website_url` field. When the
|
|
server sees it filled, it responds with a silent 204 and inserts nothing.
|
|
80%+ of crude form-fillers fall here.
|
|
3. **Sliding-window rate-limit** — 5/min per IP (configurable via
|
|
`MARKETING_RATE_LIMIT_PER_MIN`) and 1/min per email. The per-email limit
|
|
doubles as a re-capture dedupe so the LeadEvent timeline doesn't flood.
|
|
4. **Cloudflare Turnstile** (optional) — gated behind the
|
|
`marketing-funnel-turnstile` feature flag. When enabled, callers must
|
|
include a valid `turnstileToken`; verification hits the Cloudflare API.
|
|
|
|
Successful responses are intentionally identical for new and duplicate
|
|
emails (empty body, 200 OK) so bots cannot enumerate the customer list.
|
|
|
|
## API contract
|
|
|
|
### `POST /api/leads/capture`
|
|
|
|
Public, CORS-enabled. Accepts `application/json` or
|
|
`application/x-www-form-urlencoded`. Body:
|
|
|
|
```json
|
|
{
|
|
"email": "alice@example.com",
|
|
"firstName": "Alice",
|
|
"lastName": "Liddell",
|
|
"phone": "+44 …",
|
|
"company": "Acme",
|
|
"source": "landing-pro",
|
|
"attribution": {
|
|
"utmSource": "google",
|
|
"utmCampaign": "spring-launch",
|
|
"gclid": "EAIaIQ…",
|
|
"referrer": "https://google.com/",
|
|
"landingPage": "https://example.com/landing"
|
|
},
|
|
"turnstileToken": "0.OOOO…",
|
|
"website_url": ""
|
|
}
|
|
```
|
|
|
|
Responses:
|
|
|
|
| Status | Meaning |
|
|
|--------|--------------------------------------------------|
|
|
| 200 | Captured (or silently de-duplicated). Empty body.|
|
|
| 204 | Honeypot triggered or per-email rate-limit hit. |
|
|
| 400 | Invalid payload shape. |
|
|
| 403 | Origin not allowlisted or Turnstile failed. |
|
|
| 413 | Body exceeds 32 KiB. |
|
|
| 429 | Per-IP rate-limit exceeded. |
|
|
|
|
### `OPTIONS /api/leads/capture`
|
|
|
|
CORS preflight. Echoes the matched origin in `Access-Control-Allow-Origin`.
|
|
|
|
## SDK (`marketing.js`)
|
|
|
|
Drop the script tag onto every marketing page:
|
|
|
|
```html
|
|
<script src="https://cdn.example.com/embed/marketing.js"
|
|
data-api-url="https://app.example.com" defer></script>
|
|
```
|
|
|
|
The script self-bootstraps on `DOMContentLoaded`:
|
|
|
|
- Reads `?utm_source/medium/campaign/term/content`, `?gclid`, `?fbclid` from
|
|
the URL and merges them with the stored first-touch attribution.
|
|
- Persists merged attribution to `localStorage["__tcm_attr"]` so multi-page
|
|
flows hit the same lead source.
|
|
- Auto-binds any `<form data-toolctrl-lead>` on the page — submitting
|
|
POSTs to `/api/leads/capture` instead of the form's action.
|
|
|
|
Imperative API exposed as `window.toolctrlMarketing`:
|
|
|
|
```js
|
|
toolctrlMarketing.captureLead({
|
|
email: "alice@example.com",
|
|
firstName: "Alice",
|
|
source: "blog-cta",
|
|
});
|
|
|
|
toolctrlMarketing.bindFormById("lead-form");
|
|
toolctrlMarketing.bindFormsByAttr();
|
|
toolctrlMarketing.getAttribution(); // → merged first-touch object
|
|
toolctrlMarketing.resetAttribution(); // wipes localStorage cache
|
|
```
|
|
|
|
Custom events on bound forms:
|
|
|
|
- `tcm:captured` — `{ ok: boolean, status: number }`
|
|
- `tcm:error` — `{ error: string }`
|
|
|
|
The SDK is hand-minified to stay under the 3 KB gzipped budget (currently
|
|
~2 KB) and ships from `apps/web/public/embed/marketing.js`.
|
|
|
|
## Admin UI
|
|
|
|
- **List** at `/admin/marketing-leads` — filter by status (seven badge
|
|
shortcuts), source, or search by email/name/company. Paginated 25 per
|
|
page. Permission: `admin.marketing.leads.view`.
|
|
- **Detail** at `/admin/marketing-leads/:id` — profile + attribution
|
|
panels, status transition buttons (every state reachable via one click),
|
|
inline notes editor with history, convert-to-tenant action, and the
|
|
LeadEvent activity timeline. Permission: `admin.marketing.leads.manage`.
|
|
|
|
## CLI
|
|
|
|
```
|
|
toolctrl-cli leads list [--status new] [--limit 50]
|
|
toolctrl-cli leads export [--out leads.csv] [--status qualified]
|
|
toolctrl-cli leads delete --id <leadId> # DSGVO right-to-be-forgotten
|
|
```
|
|
|
|
## Configuration
|
|
|
|
| Env | Default | Purpose |
|
|
|--------------------------------|---------|-------------------------------------------------------------|
|
|
| `MARKETING_ALLOWED_ORIGINS` | (empty) | CSV of allowed origins for capture endpoint. Fail-closed. |
|
|
| `MARKETING_RATE_LIMIT_PER_MIN` | 5 | Per-IP sliding-window submission limit. |
|
|
| `TURNSTILE_SITE_KEY` | — | Cloudflare Turnstile site key (shared with form-widget). |
|
|
| `TURNSTILE_SECRET_KEY` | — | Cloudflare Turnstile secret (shared with form-widget). |
|
|
|
|
The `marketing-funnel-turnstile` feature flag must be enabled for Turnstile
|
|
to actually gate submissions; the secret alone is not enough.
|
|
|
|
## Rollback
|
|
|
|
- `MARKETING_ALLOWED_ORIGINS=` empties the allowlist → all browser captures
|
|
are rejected with 403 while leaving the admin UI usable for triage.
|
|
- Drop the `Lead` + `LeadEvent` tables to wipe captured data (no other
|
|
tables reference them today). Export with `toolctrl-cli leads export`
|
|
first.
|
|
|
|
---
|
|
|
|
# Phase 2 — Self-Service Conversion + Outbound Webhooks + Attribution
|
|
|
|
Phase 2 (task #79) extends Phase 1's inbox with three independently-shippable
|
|
layers:
|
|
|
|
1. **Self-service conversion deep-links** — a signed lead-token in the URL
|
|
pre-fills the registration form. The normal registration service creates
|
|
the User + Tenant, then records the Lead conversion against that Tenant:
|
|
`Lead.convertedToTenantId`, a `TenantAcquisition` snapshot, and the
|
|
marketing outbox event are written in one transaction.
|
|
2. **Outbound webhook foundation** — six pluggable adapters (Slack, Discord,
|
|
Mailchimp, HubSpot, Brevo, custom HTTPS) plus the
|
|
`apps/worker/src/jobs/marketing-outbound.ts` handler contract. General
|
|
marketing integration delivery is still a production follow-up because
|
|
that handler is not currently bootstrapped from `apps/worker/src/index.ts`.
|
|
3. **Attribution reports** — `/admin/marketing-funnels` aggregates leads
|
|
+ acquisitions by source, campaign, utm_source, conversionPath. Mirror
|
|
via `toolctrl-cli marketing funnel|attribution`.
|
|
|
|
## Signed lead-token
|
|
|
|
`signLeadToken({ leadId, planId, expiresInDays = 7 })` produces a token of
|
|
the shape `v1.<base64url(payload)>.<base64url(hmac-sha256)>`. The HMAC
|
|
key is `LEAD_CONVERSION_SECRET` (min 32 bytes). `verifyLeadToken(token)`
|
|
returns the payload when the signature matches AND the token has not
|
|
expired — otherwise `null`. Both functions fail-closed when the secret
|
|
env is missing.
|
|
|
|
Email pre-filled from a Lead is rendered read-only on the register form
|
|
to prevent lead-hijacking ("I use your token but with my email"). The signed
|
|
token is also submitted as a hidden field and verified again in the register
|
|
action. If email verification is enabled, the verified conversion context is
|
|
stored on the pending `Registration` row and replayed from `/verify/:token`
|
|
after the visitor sets a password. To change the email the visitor must
|
|
abandon the link and start a fresh signup.
|
|
|
|
## Atomic conversion service
|
|
|
|
`convertLeadToTenant({ leadId, tenant: { name, slug, inboundAddress }, ... })`
|
|
runs inside a single Prisma `$transaction`:
|
|
|
|
1. `tenant.create` (slug + inbound address pre-resolved by caller).
|
|
2. `lead.update` → `status` advances to `trial`/`customer`,
|
|
`convertedToTenantId` + `convertedAt` populated.
|
|
3. `leadEvent.create` of type `converted` with full conversion payload.
|
|
4. `tenantAcquisition.create` snapshot copying UTM/referrer/landingPage
|
|
from the Lead.
|
|
5. `publishOutboxEvent` with type `marketing.lead.convertedtotrial` or
|
|
`marketing.lead.convertedtocustomer` and dispatchKey
|
|
`lead-convert:<leadId>` (idempotent on retry).
|
|
|
|
Already-converted leads (`convertedToTenantId !== null`) short-circuit
|
|
and return the existing tenant.
|
|
|
|
For the self-service register flow, the already-created Tenant is linked via
|
|
`recordExistingTenantLeadConversion({ leadId, tenantId, ... })`. It reuses the
|
|
same conversion event/outbox payload and upserts the `TenantAcquisition` row
|
|
instead of creating a second Tenant.
|
|
|
|
## Admin manual override
|
|
|
|
`/admin/marketing-leads/<id>` exposes a "Create Tenant" form. Operator
|
|
can override the tenant name + supply an optional planId. The action
|
|
calls `convertLeadToTenant({ conversionPath: "admin-override", actorId })`.
|
|
A native confirm dialog guards accidental clicks.
|
|
|
|
## Outbound webhook adapters
|
|
|
|
`packages/marketing-outbound` ships six adapters under
|
|
`src/adapters/*.ts`:
|
|
|
|
| Adapter | Config keys (encrypted at rest) | Notes |
|
|
|---------|----------------------------------------------------------------|-------|
|
|
| slack | `webhookUrl`, `channel?` | Block-Kit message. |
|
|
| discord | `webhookUrl`, `username?` | Embed with color-coded event. |
|
|
| mailchimp | `apiKey`, `listId`, `status?`, `tags?` | PUT to lists/{id}/members/{md5(email)}. |
|
|
| hubspot | `accessToken` | POST /crm/v3/objects/contacts; 409 = idempotent. |
|
|
| brevo | `apiKey`, `listIds?` | POST /v3/contacts with `updateEnabled: true`. |
|
|
| webhook | `url`, `signingSecret?`, `headers?` | POSTs canonical envelope; optional HMAC-SHA256 sig. |
|
|
|
|
All adapters share a single `adapterFetch` helper that enforces:
|
|
|
|
- HTTPS only (rejects http: / localhost / RFC1918 / 169.254.* /
|
|
`metadata.google.internal` / `.internal` / `.local` / `.lan`).
|
|
- 10 s default timeout (AbortController).
|
|
- Returns structured `AdapterResult { ok, status?, errorTag? }` so the
|
|
worker can categorise failures consistently.
|
|
|
|
The `ctx.maskPii.email` toggle masks the local part of the email
|
|
(`patrick@example.com → ***@example.com`) before any payload leaves the
|
|
host. Use it for data-egress-restricted tenants.
|
|
|
|
## Apps/worker handler status
|
|
|
|
`apps/worker/src/jobs/marketing-outbound.ts` defines a pure handler contract
|
|
for marketing integration delivery, but it does not own or subscribe to
|
|
`outbox.marketing`. That queue is currently used by the email-sequence trigger
|
|
subscriber when `MARKETING_SEQUENCES_ENABLED=true`; this does not imply
|
|
Slack/Discord/Mailchimp/HubSpot/Brevo/custom webhook delivery is active.
|
|
Treat the adapters, handler, admin UI, and env flag as a shipped foundation
|
|
until a bootstrap module wires a distinct general marketing integration
|
|
subscriber into the worker runtime with explicit ownership tests.
|
|
|
|
The handler accepts only these event types today:
|
|
`marketing.lead.captured`, `marketing.lead.qualified`,
|
|
`marketing.lead.convertedtotrial`, and
|
|
`marketing.lead.convertedtocustomer`. New `marketing.lead.*` events require
|
|
an update to `MARKETING_LEAD_EVENT_TYPES` plus tests. For each accepted event:
|
|
|
|
1. Parse + validate envelope (`parseMarketingOutboundJob`).
|
|
2. Bail out when `MARKETING_WEBHOOKS_ENABLED=false`.
|
|
3. Fetch every enabled `MarketingIntegration` whose `triggerEvents`
|
|
includes the eventType.
|
|
4. For each integration: decrypt config, resolve adapter via
|
|
`resolveAdapter(type)`, call `adapter.send`. One integration's
|
|
failure does not affect siblings.
|
|
5. Persist diagnostics (`lastDeliveredAt`, `lastError`) on the
|
|
integration row so the admin UI can surface them.
|
|
|
|
## Admin integrations UI
|
|
|
|
`/admin/marketing-integrations` (Phase 05 #79). CRUD list with:
|
|
|
|
- Add form (type picker + name + trigger-events multi-select + config
|
|
JSON textarea).
|
|
- Per-integration enable/disable toggle.
|
|
- Delete with native confirm dialog.
|
|
- Last-delivered / last-error display.
|
|
- Config display masked (`***` for `webhookUrl`, `apiKey`,
|
|
`accessToken`, `signingSecret`).
|
|
|
|
Secrets are encrypted with the existing AES-256-GCM CryptoApi helper
|
|
(`apps/web/app/utils/api/.server/CryptoApi.ts`) keyed off `CRYPTO_SECRET`.
|
|
|
|
## Attribution reports
|
|
|
|
`/admin/marketing-funnels` (Phase 05 #79). Top-line cards (leads
|
|
captured, conversions, conversion rate) plus four breakdowns: campaign,
|
|
utm_source, source-label, conversionPath. Reads via `dbRead`
|
|
(replica-routed). CLI mirror:
|
|
|
|
- `toolctrl-cli marketing funnel --days 30`
|
|
- `toolctrl-cli marketing attribution --dimension campaign --days 30 --limit 25`
|
|
|
|
## Configuration (Phase 2)
|
|
|
|
| Env | Default | Purpose |
|
|
|------------------------------|---------|------------------------------------------------------------|
|
|
| `LEAD_CONVERSION_SECRET` | — | HMAC key for signed lead-tokens (min 32 bytes). |
|
|
| `LEAD_CONVERSION_ENABLED` | `true` | Set `false` to disable the conversion flow without DB rollback. |
|
|
| `MARKETING_WEBHOOKS_ENABLED` | `true` | Handler-level kill switch once the general marketing integration subscriber is bootstrapped. It does not start that subscriber by itself. |
|
|
|
|
## Webhook payload (custom adapter)
|
|
|
|
```json
|
|
{
|
|
"schema": "marketing.lead.v1",
|
|
"eventType": "marketing.lead.convertedtotrial",
|
|
"occurredAt": "2026-05-17T00:00:00.000Z",
|
|
"lead": {
|
|
"id": "lead_abc",
|
|
"email": "patrick@example.com",
|
|
"firstName": "Patrick",
|
|
"lastName": "Doe",
|
|
"company": "Acme",
|
|
"source": "landing-pro",
|
|
"utm": { "source": "google", "medium": "cpc", "campaign": "q2-launch" },
|
|
"referrer": "https://google.com",
|
|
"landingPage": "/pricing"
|
|
},
|
|
"conversion": {
|
|
"tenantId": "tenant_xyz",
|
|
"targetStatus": "trial",
|
|
"conversionPath": "self-service",
|
|
"planId": "plan_pro"
|
|
}
|
|
}
|
|
```
|
|
|
|
When `signingSecret` is set, the receiver verifies
|
|
`x-marketing-signature: sha256=<hex>` against
|
|
`hmac_sha256(body, signingSecret)`.
|
|
|
|
## Rollback (Phase 2)
|
|
|
|
- `LEAD_CONVERSION_ENABLED=false` — register loader ignores the token
|
|
query params; admin override button hidden.
|
|
- `MARKETING_WEBHOOKS_ENABLED=false` — once the general marketing
|
|
integration subscriber is bootstrapped, the worker handler short-circuits.
|
|
Today it is only a handler-level kill switch for the shipped foundation.
|
|
Outbox events accumulate; raise outbox retention if you plan a long
|
|
pause.
|
|
- Per-integration disable toggle in the admin UI — faulty integrations
|
|
can be parked without a deploy.
|
|
- `TenantAcquisition` + `MarketingIntegration` tables can be dropped
|
|
(no FK in to them from Tenant), but you'd lose attribution history.
|