toolctrl-labs/HiveGrid-Wireframes
9
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
3cb93a81bea50927864585d1fafe0af4c4772a09
HiveGrid-Wireframes/hivegrid-cloud/docs/features/marketing-funnel.md
T

18 KiB
Raw Blame History

Marketing-Funnel Overview

This page is the canonical overview for the shipped lead-capture and self-service conversion foundation. Email sequences are covered separately in docs/features/email-sequences.md.

This page also keeps the marketing-outbound boundary explicit: the packages/marketing-outbound adapters and apps/worker/src/jobs/marketing-outbound.ts handler are a shipped general handler foundation, but the general subscriber is not bootstrapped from apps/worker/src/index.ts. That foundation is separate from the email-marketing CRM-contact campaign plugin candidate in docs/feature-decisions/email-marketing-decision-report.md.

The marketing-funnel ships a four-layer pipeline that lets external landing pages, ads, and blog CTAs feed structured leads into the operator-side admin inbox without an external CRM.

Layer Component
1. Data model Lead + LeadEvent Prisma tables (prisma/schema/marketingLeads.prisma)
2. Ingestion API POST /api/leads/capture (apps/web/app/routes/api.leads.capture.tsx)
3. SDK apps/web/public/embed/marketing.js (origin-served mini-snippet; CDN-ready while A7/B2 remains backlog)
4. Admin UI /admin/marketing-leads (list) + /admin/marketing-leads/:id (detail)

Status machine

Leads move through seven states:

new → contacted → qualified → trial → customer
                                    ↘ churned
                                    ↘ disqualified

Transitions are recorded as LeadEvent rows of type status-changed for the activity-timeline in the detail-view. Operators move leads forward (or to a terminal state) from the detail page. The score, notes, and convertedAt columns are operator-managed.

Attribution

Every Lead carries first-touch attribution columns populated by the SDK on the initial visit and persisted in localStorage so subsequent page visits attribute to the same source:

Column Source
utmSource ?utm_source=…
utmMedium ?utm_medium=…
utmCampaign ?utm_campaign=…
utmTerm ?utm_term=…
utmContent ?utm_content=…
gclid ?gclid=… (Google Ads click identifier)
fbclid ?fbclid=… (Meta Ads click identifier)
referrer document.referrer
landingPage location.href at the first visit
firstSeenAt new Date() on initial insert (never overwritten)
lastTouchAt new Date() on every re-capture

source is operator-supplied (via the form's data-source attribute or the SDK call's source field) and identifies the surface — "landing-pro", "blog-cta", "webinar-form" — distinct from the paid-channel utmSource.

Anti-spam (defence in depth)

The capture endpoint stacks four layers, each weakening different bot classes; none of them is sufficient alone.

  1. Origin allowlist — MARKETING_ALLOWED_ORIGINS (CSV with *.host wildcards). 403 on miss. Defeats browser-based form-spam from non-whitelisted pages; can be spoofed by server-to-server bots.
  2. Honeypot — the SDK injects a hidden website_url field. When the server sees it filled, it responds with a silent 204 and inserts nothing. 80%+ of crude form-fillers fall here.
  3. Sliding-window rate-limit — 5/min per IP (configurable via MARKETING_RATE_LIMIT_PER_MIN) and 1/min per email. The per-email limit doubles as a re-capture dedupe so the LeadEvent timeline doesn't flood.
  4. Cloudflare Turnstile (optional) — gated behind the marketing-funnel-turnstile feature flag. When enabled, callers must include a valid turnstileToken; verification hits the Cloudflare API.

Successful responses are intentionally identical for new and duplicate emails (empty body, 200 OK) so bots cannot enumerate the customer list.

API contract

POST /api/leads/capture

Public, CORS-enabled. Accepts application/json or application/x-www-form-urlencoded. Body:

{
  "email": "alice@example.com",
  "firstName": "Alice",
  "lastName": "Liddell",
  "phone": "+44 …",
  "company": "Acme",
  "source": "landing-pro",
  "attribution": {
    "utmSource": "google",
    "utmCampaign": "spring-launch",
    "gclid": "EAIaIQ…",
    "referrer": "https://google.com/",
    "landingPage": "https://example.com/landing"
  },
  "turnstileToken": "0.OOOO…",
  "website_url": ""
}

Responses:

Status Meaning
200 Captured (or silently de-duplicated). Empty body.
204 Honeypot triggered or per-email rate-limit hit.
400 Invalid payload shape.
403 Origin not allowlisted or Turnstile failed.
413 Body exceeds 32 KiB.
429 Per-IP rate-limit exceeded.

OPTIONS /api/leads/capture

CORS preflight. Echoes the matched origin in Access-Control-Allow-Origin.

SDK (marketing.js)

Drop the script tag onto every marketing page:

<script src="https://cdn.example.com/embed/marketing.js"
        data-api-url="https://app.example.com" defer></script>

The script self-bootstraps on DOMContentLoaded:

  • Reads ?utm_source/medium/campaign/term/content, ?gclid, ?fbclid from the URL and merges them with the stored first-touch attribution.
  • Persists merged attribution to localStorage["__tcm_attr"] so multi-page flows hit the same lead source.
  • Auto-binds any <form data-toolctrl-lead> on the page — submitting POSTs to /api/leads/capture instead of the form's action.

Imperative API exposed as window.toolctrlMarketing:

toolctrlMarketing.captureLead({
  email: "alice@example.com",
  firstName: "Alice",
  source: "blog-cta",
});

toolctrlMarketing.bindFormById("lead-form");
toolctrlMarketing.bindFormsByAttr();
toolctrlMarketing.getAttribution();   // → merged first-touch object
toolctrlMarketing.resetAttribution(); // wipes localStorage cache

Custom events on bound forms:

  • tcm:captured — { ok: boolean, status: number }
  • tcm:error — { error: string }

The SDK is hand-minified to stay under the 3 KB gzipped budget (currently ~2 KB) and ships from apps/web/public/embed/marketing.js.

Admin UI

  • List at /admin/marketing-leads — filter by status (seven badge shortcuts), source, or search by email/name/company. Paginated 25 per page. Permission: admin.marketing.leads.view.
  • Detail at /admin/marketing-leads/:id — profile + attribution panels, status transition buttons (every state reachable via one click), inline notes editor with history, convert-to-tenant action, and the LeadEvent activity timeline. Permission: admin.marketing.leads.manage.

CLI

toolctrl-cli leads list [--status new] [--limit 50]
toolctrl-cli leads export [--out leads.csv] [--status qualified]
toolctrl-cli leads delete --id <leadId>        # DSGVO right-to-be-forgotten

Configuration

Env Default Purpose
MARKETING_ALLOWED_ORIGINS (empty) CSV of allowed origins for capture endpoint. Fail-closed.
MARKETING_RATE_LIMIT_PER_MIN 5 Per-IP sliding-window submission limit.
TURNSTILE_SITE_KEY — Cloudflare Turnstile site key (shared with form-widget).
TURNSTILE_SECRET_KEY — Cloudflare Turnstile secret (shared with form-widget).

The marketing-funnel-turnstile feature flag must be enabled for Turnstile to actually gate submissions; the secret alone is not enough.

Rollback

  • MARKETING_ALLOWED_ORIGINS= empties the allowlist → all browser captures are rejected with 403 while leaving the admin UI usable for triage.
  • Drop the Lead + LeadEvent tables to wipe captured data (no other tables reference them today). Export with toolctrl-cli leads export first.

Phase 2 — Self-Service Conversion + Outbound Webhooks + Attribution

Phase 2 (task #79) extends Phase 1's inbox with three independently-shippable layers:

  1. Self-service conversion deep-links — a signed lead-token in the URL pre-fills the registration form. The normal registration service creates the User + Tenant, then records the Lead conversion against that Tenant: Lead.convertedToTenantId, a TenantAcquisition snapshot, and the marketing outbox event are written in one transaction.
  2. Outbound webhook foundation — six pluggable adapters (Slack, Discord, Mailchimp, HubSpot, Brevo, custom HTTPS) plus the apps/worker/src/jobs/marketing-outbound.ts handler contract. General marketing integration delivery is still a production follow-up because that handler is not currently bootstrapped from apps/worker/src/index.ts.
  3. Attribution reports — /admin/marketing-funnels aggregates leads
    • acquisitions by source, campaign, utm_source, conversionPath. Mirror via toolctrl-cli marketing funnel|attribution.

Signed lead-token

signLeadToken({ leadId, planId, expiresInDays = 7 }) produces a token of the shape v1.<base64url(payload)>.<base64url(hmac-sha256)>. The HMAC key is LEAD_CONVERSION_SECRET (min 32 bytes). verifyLeadToken(token) returns the payload when the signature matches AND the token has not expired — otherwise null. Both functions fail-closed when the secret env is missing.

Email pre-filled from a Lead is rendered read-only on the register form to prevent lead-hijacking ("I use your token but with my email"). The signed token is also submitted as a hidden field and verified again in the register action. If email verification is enabled, the verified conversion context is stored on the pending Registration row and replayed from /verify/:token after the visitor sets a password. To change the email the visitor must abandon the link and start a fresh signup.

Atomic conversion service

convertLeadToTenant({ leadId, tenant: { name, slug, inboundAddress }, ... }) runs inside a single Prisma $transaction:

  1. tenant.create (slug + inbound address pre-resolved by caller).
  2. lead.update → status advances to trial/customer, convertedToTenantId + convertedAt populated.
  3. leadEvent.create of type converted with full conversion payload.
  4. tenantAcquisition.create snapshot copying UTM/referrer/landingPage from the Lead.
  5. publishOutboxEvent with type marketing.lead.convertedtotrial or marketing.lead.convertedtocustomer and dispatchKey lead-convert:<leadId> (idempotent on retry).

Already-converted leads (convertedToTenantId !== null) short-circuit and return the existing tenant.

For the self-service register flow, the already-created Tenant is linked via recordExistingTenantLeadConversion({ leadId, tenantId, ... }). It reuses the same conversion event/outbox payload and upserts the TenantAcquisition row instead of creating a second Tenant.

Admin manual override

/admin/marketing-leads/<id> exposes a "Create Tenant" form. Operator can override the tenant name + supply an optional planId. The action calls convertLeadToTenant({ conversionPath: "admin-override", actorId }). A native confirm dialog guards accidental clicks.

Outbound webhook adapters

packages/marketing-outbound ships six adapters under src/adapters/*.ts:

Adapter Config keys (encrypted at rest) Notes
slack webhookUrl, channel? Block-Kit message.
discord webhookUrl, username? Embed with color-coded event.
mailchimp apiKey, listId, status?, tags? PUT to lists/{id}/members/{md5(email)}.
hubspot accessToken POST /crm/v3/objects/contacts; 409 = idempotent.
brevo apiKey, listIds? POST /v3/contacts with updateEnabled: true.
webhook url, signingSecret?, headers? POSTs canonical envelope; optional HMAC-SHA256 sig.

All adapters share a single adapterFetch helper that enforces:

  • HTTPS only (rejects http: / localhost / RFC1918 / 169.254.* / metadata.google.internal / .internal / .local / .lan).
  • 10 s default timeout (AbortController).
  • Returns structured AdapterResult { ok, status?, errorTag? } so the worker can categorise failures consistently.

The ctx.maskPii.email toggle masks the local part of the email (patrick@example.com → ***@example.com) before any payload leaves the host. Use it for data-egress-restricted tenants.

Apps/worker handler status

apps/worker/src/jobs/marketing-outbound.ts defines a pure handler contract for marketing integration delivery, but it does not own or subscribe to outbox.marketing. That queue is currently used by the email-sequence trigger subscriber when MARKETING_SEQUENCES_ENABLED=true; this does not imply Slack/Discord/Mailchimp/HubSpot/Brevo/custom webhook delivery is active. Treat the adapters, handler, admin UI, and env flag as a shipped foundation until a bootstrap module wires a distinct general marketing integration subscriber into the worker runtime with explicit ownership tests.

The handler accepts only these event types today: marketing.lead.captured, marketing.lead.qualified, marketing.lead.convertedtotrial, and marketing.lead.convertedtocustomer. New marketing.lead.* events require an update to MARKETING_LEAD_EVENT_TYPES plus tests. For each accepted event:

  1. Parse + validate envelope (parseMarketingOutboundJob).
  2. Bail out when MARKETING_WEBHOOKS_ENABLED=false.
  3. Fetch every enabled MarketingIntegration whose triggerEvents includes the eventType.
  4. For each integration: decrypt config, resolve adapter via resolveAdapter(type), call adapter.send. One integration's failure does not affect siblings.
  5. Persist diagnostics (lastDeliveredAt, lastError) on the integration row so the admin UI can surface them.

Admin integrations UI

/admin/marketing-integrations (Phase 05 #79). CRUD list with:

  • Add form (type picker + name + trigger-events multi-select + config JSON textarea).
  • Per-integration enable/disable toggle.
  • Delete with native confirm dialog.
  • Last-delivered / last-error display.
  • Config display masked (*** for webhookUrl, apiKey, accessToken, signingSecret).

Secrets are encrypted with the existing AES-256-GCM CryptoApi helper (apps/web/app/utils/api/.server/CryptoApi.ts) keyed off CRYPTO_SECRET.

Attribution reports

/admin/marketing-funnels (Phase 05 #79). Top-line cards (leads captured, conversions, conversion rate) plus four breakdowns: campaign, utm_source, source-label, conversionPath. Reads via dbRead (replica-routed). CLI mirror:

  • toolctrl-cli marketing funnel --days 30
  • toolctrl-cli marketing attribution --dimension campaign --days 30 --limit 25

Configuration (Phase 2)

Env Default Purpose
LEAD_CONVERSION_SECRET — HMAC key for signed lead-tokens (min 32 bytes).
LEAD_CONVERSION_ENABLED true Set false to disable the conversion flow without DB rollback.
MARKETING_WEBHOOKS_ENABLED true Handler-level kill switch once the general marketing integration subscriber is bootstrapped. It does not start that subscriber by itself.

Webhook payload (custom adapter)

{
  "schema": "marketing.lead.v1",
  "eventType": "marketing.lead.convertedtotrial",
  "occurredAt": "2026-05-17T00:00:00.000Z",
  "lead": {
    "id": "lead_abc",
    "email": "patrick@example.com",
    "firstName": "Patrick",
    "lastName": "Doe",
    "company": "Acme",
    "source": "landing-pro",
    "utm": { "source": "google", "medium": "cpc", "campaign": "q2-launch" },
    "referrer": "https://google.com",
    "landingPage": "/pricing"
  },
  "conversion": {
    "tenantId": "tenant_xyz",
    "targetStatus": "trial",
    "conversionPath": "self-service",
    "planId": "plan_pro"
  }
}

When signingSecret is set, the receiver verifies x-marketing-signature: sha256=<hex> against hmac_sha256(body, signingSecret).

Rollback (Phase 2)

  • LEAD_CONVERSION_ENABLED=false — register loader ignores the token query params; admin override button hidden.
  • MARKETING_WEBHOOKS_ENABLED=false — once the general marketing integration subscriber is bootstrapped, the worker handler short-circuits. Today it is only a handler-level kill switch for the shipped foundation. Outbox events accumulate; raise outbox retention if you plan a long pause.
  • Per-integration disable toggle in the admin UI — faulty integrations can be parked without a deploy.
  • TenantAcquisition + MarketingIntegration tables can be dropped (no FK in to them from Tenant), but you'd lose attribution history.
Reference in new issue View Git Blame Copy Permalink