Files
HiveGrid-Wireframes/hivegrid-cloud/docs/improvement-plan/05-business-features/marketing-funnel-phase-1.md
T

17 KiB

Marketing-Funnel — Phase 1: Lead-Capture + Ingestion-API + Admin-Inbox

Status: in_progress (core shipped) Task-ID: #78 Phase: 05 Estimated Effort: 1 Woche Owner: unassigned

Problem

Die Boilerplate hat keinen Marketing-Funnel — Leads von externen Landing-Pages, Ads oder Marketing-Sites haben keinen strukturierten Eingang ins System. Aktuell: Marketing-Page leitet zu Generic-Signup, alle UTMs und Attribution-Daten gehen verloren. Use-Cases die heute scheitern:

  • Marketing-Landing-Page mit Lead-Capture-Form → "Mailchimp / Brevo / HubSpot bitte selbst integrieren"
  • Performance-Marketing: Ads-Conversion-Tracking ohne Lead-Storage
  • Lead-Scoring + Qualification-Workflow durch Sales-Team
  • Attribution-Reporting (welcher Channel bringt welche Conversions)
  • Outbound-Sales: Liste der Interessenten ohne CRM zu kopieren

Boilerplate-Käufer wollen einen sofort-startbaren Marketing-Funnel, nicht "bauen sich selbst CRM-Integration".

Depends on

  • 05-business-features/feature-flags.md (#53) — Turnstile-Layer hinter Flag für Gradual-Rollout
  • 03-scaling-infrastructure/shared-store-package.md (#37) — Rate-Limit pro IP + pro Email via SharedStore
  • 02-observability/sentry-wiring-api-worker.md (#11) — Failed-Captures landen in Sentry
  • Existierende AuditEvent-Infrastruktur — vorhanden, wird konsumiert
  • 03-scaling-infrastructure/asset-cdn.md (#46) — marketing.js-SDK wird via CDN ausgeliefert

Blocks

  • 05-business-features/marketing-funnel-phase-2.md (#79) — Self-Service-Conversion baut auf Lead-Model
  • 05-business-features/marketing-funnel-phase-3.md (#80) — Email-Sequences triggern auf Lead-Lifecycle-Events

Approach

Vier-Schicht-Architektur:

  1. Lead-Datenmodell — Lead-Tabelle als zentraler Eintrag mit Status-Machine + Attribution-Daten.
  2. Ingestion-Endpoint — POST /api/leads/capture mit CORS-Whitelist, Rate-Limit, Anti-Spam.
  3. Marketing-SDK (marketing.js) — minimaler JS-Snippet für Marketing-Pages mit Auto-UTM-Capture + Form-Hook.
  4. Admin-Inbox — DataTable mit Filter/Search, Detail-View mit Activity-Timeline + Inline-Notes.

Begründung der Architektur-Wahl:

  • Eigene Lead-Tabelle statt Tenant.User-Misuse: Leads haben fundamentally andere Lifecycle als Tenants/Users. Misch-Schema wäre Footgun. Saubere Trennung erlaubt später spezialisierte Marketing-Logic (Scoring, Sequences) ohne User-Schema zu touchen.
  • Public-Capture-Endpoint mit CORS-Whitelist: Marketing-Domains sind bekannt (toolctrl.de, marketing-pages.toolctrl.de, externe Customer-Domains). Whitelist via Env reduziert DOS-Surface.
  • Mini-SDK statt Recommendation "macht ein Fetch": Auto-UTM-Capture + Referrer + viewport-Stuff ist boilerplate-code den jeder Marketer falsch macht. SDK gibt konsistente Datenqualität.
  • Anti-Spam-Defense-in-Depth: Honeypot (free, 80% Coverage) + Rate-Limit (Server-side) + optional Turnstile (90%+ Coverage gegen sophisticated Bots). Kein einzelner Layer reicht.
  • Status-Machine (new → contacted → qualified → trial → customer → churned/disqualified): klare Sales-Funnel-Stages, ermöglicht Funnel-Reports in Phase 2/3.

Implementation Steps

  1. Prisma-Lead-Model — Lead { id, email (unique-per-status? oder pure unique mit dedup-logic), firstName?, lastName?, source, campaign?, utmSource?, utmMedium?, utmCampaign?, utmContent?, utmTerm?, referrer?, payload Json, capturedAt, ipAddress?, userAgent?, status LeadStatus (enum: new|contacted|qualified|trial|customer|churned|disqualified) default new, assignedToAdminUserId?, notes Text default "", convertedToTenantId?, convertedAt? }. Indexes: email, status, capturedAt DESC, (source, capturedAt), assignedToAdminUserId. Migration.
    • Files: prisma/schema.prisma, prisma/migrations/<ts>_marketing_leads/migration.sql
    • Commit: 05/marketing-phase-1: prisma lead model
  2. Lead-Capture-API — apps/web/app/routes/api.leads.capture.tsx action. Validation via Zod (email pflicht, sonstige optional). CORS-Check: Request-Origin gegen MARKETING_DOMAINS env (comma-separated). Honeypot-Field-Check (Field-Name botfield — wenn gefüllt: silent 200 ohne DB-Insert). Optional Turnstile-Token via Cloudflare-Verify-API. Rate-Limit via SharedStore: max 5/min/IP, max 3/min/Email. Bei Duplicate-Email innerhalb 24h: kein neuer Insert, sondern Update von payload und capturedAt (Dedup mit Activity-Append). Insert mit Auto-UTM-Population. Returnt { ok: true, leadId } für SDK-Tracking.
    • Files: apps/web/app/routes/api.leads.capture.tsx, apps/web/app/utils/leads/captureLead.server.ts, apps/web/app/utils/leads/originCheck.ts, Tests
    • Commit: 05/marketing-phase-1: lead capture api with anti-spam
  3. packages/marketing-sdk-js/ — neuer Workspace. src/index.ts exportiert init({ apiUrl, formSelector? }), captureFormSubmission(formData), trackPageView(). Auto-Init: bei Script-Load liest data-api-url aus Script-Tag, attached an Forms mit data-marketing-form Attribut, captured Form-Submit → POST an /api/leads/capture. Auto-UTM aus URL-Params bei Page-Load → localStorage gecached für nachfolgende Sessions. Build-Output: minified marketing.js <5KB gzipped via esbuild.
    • Files: packages/marketing-sdk-js/src/index.ts, packages/marketing-sdk-js/build.config.ts, Tests, README
    • Commit: 05/marketing-phase-1: marketing-sdk-js package
  4. CDN-Deploy-Step für marketing.js — CI-Step erweitert: nach Build wird marketing.js zum Asset-CDN deployed unter versioniertem Pfad (marketing.v1.js) + symlinked als marketing.js für latest. SRI-Hash generieren, in Doku einbetten.
    • Files: .github/workflows/deploy.yml oder vergleichbarer CI-Step, apps/web/app/utils/marketing/getSdkSriHash.server.ts
    • Commit: 05/marketing-phase-1: ci step for marketing-sdk cdn deploy
  5. Admin-Inbox /admin/marketing/leads — Route mit Loader (paginated, mit Filter-Query-Params: status, source, search-by-email, date-range, assignedTo). DataTable-Component: Email, Name, Source, UTMs, Status (Badge), CapturedAt, AssignedTo, Actions. Bulk-Actions: assign-to-me, mark-disqualified, export-csv. Detail-View per Click → Drawer/Modal/Sub-Route mit Lead-Detail.
    • Files: apps/web/app/routes/admin.marketing.leads._index.tsx, Components
    • Commit: 05/marketing-phase-1: admin leads inbox
  6. Lead-Detail-View — /admin/marketing/leads/$id: shows alle Fields, Edit-Form für Status + AssignedTo + Notes, Activity-Timeline (alle AuditEvents zum Lead chronologisch). Inline-Note-Append-Action.
    • Files: apps/web/app/routes/admin.marketing.leads.$id.tsx, Action für Update
    • Commit: 05/marketing-phase-1: lead detail view with activity timeline
  7. Audit-Logging — jede Status-Mutation, Assignment, Notes-Edit schreibt AuditEvent mit entityType=lead, entityId=lead.id, actorUserId, action, metadata (old/new values). Verwendung des existing Audit-Helpers.
    • Files: apps/web/app/utils/leads/leadAudit.server.ts, Integration in alle Action-Handlers
    • Commit: 05/marketing-phase-1: audit trail for lead mutations
  8. CSV-Export-Action — /admin/marketing/leads/export.csv action: liefert gefilterte Leads als CSV-Download. Limit 50k Rows; bei mehr Background-Job-Hinweis.
    • Files: apps/web/app/routes/admin.marketing.leads.export[.]csv.tsx
    • Commit: 05/marketing-phase-1: csv export for leads
  9. Doku — docs/features/marketing-funnel.md (Phase-1-Section): Lead-Schema-Erklärung, API-Reference für /api/leads/capture, SDK-Usage mit Code-Beispiel, CORS-Setup, Anti-Spam-Layer, Admin-Inbox-Walkthrough, AuditEvent-Format.
    • Files: docs/features/marketing-funnel.md
    • Commit: 05/marketing-phase-1: docs

Test Requirements

  • Unit-Tests:
    • captureLead.test.ts — valid Lead-Insert, Duplicate-Dedup-Logic, UTM-Population, Status-Default
    • originCheck.test.ts — Whitelist-Matches, Wildcards, Reject
    • Honeypot-Validation: gefüllter Honeypot → silent 200, kein Insert
    • Rate-Limit: 6. Request innerhalb 1min/IP → 429
    • Turnstile-Mock: invalid token → 403
  • Integration-Tests:
    • End-to-End Capture: Request → DB-Row mit allen Feldern → AuditEvent geschrieben
    • Duplicate-Email innerhalb 24h: erste Capture inserts, zweite updated payload+capturedAt, schreibt zweites AuditEvent
    • CSV-Export: 100 Test-Leads → korrekte CSV-Struktur, alle Filter-Params respektiert
  • E2E-Tests (Playwright):
    • SDK-Test: Test-HTML-Page mit eingebundenem marketing.js + Marketing-Form → submit → Lead in Admin-Inbox sichtbar
    • Admin-Inbox: Filter nach Status, Search nach Email, Bulk-Disqualify, CSV-Download funktioniert
    • Detail-View: Status ändern → AuditEvent in Timeline; Note hinzufügen → in Timeline
    • Origin-Block: SDK auf nicht-whitelisteter Domain → API liefert CORS-Error, sichtbar in Test-Console
  • Coverage-Target: 85% für packages/marketing-sdk-js/, 85% für captureLead.server.ts + originCheck.ts, 70% UI
  • Test-Files: neben Source-Files; E2E unter apps/web/tests/e2e/marketing-leads.spec.ts + Test-HTML-Page apps/web/tests/fixtures/marketing-test-page.html

NICHT als completed markieren bevor alle Tests grün sind.

Acceptance Criteria

  • Prisma-Model Lead mit allen Feldern + Indexes + Migration
  • /api/leads/capture mit CORS, Honeypot, Turnstile (opt-in), Rate-Limit
  • packages/marketing-sdk-js/ bauen + minified <5KB gzipped
  • CI-Step deployed marketing.js zum CDN mit SRI-Hash
  • Admin-Inbox mit Filter/Search/Pagination/Bulk-Actions
  • Lead-Detail-View mit Activity-Timeline + Inline-Notes
  • Audit-Logging jeder Status- + Assignment- + Notes-Mutation
  • CSV-Export mit Filter-Respekt + Row-Limit
  • docs/features/marketing-funnel.md Phase-1-Section komplett
  • .env.example enthält MARKETING_DOMAINS, TURNSTILE_SITE_KEY, TURNSTILE_SECRET_KEY, MARKETING_SDK_CDN_URL
  • pnpm lint, pnpm typecheck, pnpm test grün
  • Coverage-Gates erfüllt
  • Atomic Commits gepushed auf agent/05-marketing-phase-1
  • Detail-MD Progress-Log aktualisiert
  • PROGRESS.md Status auf completed

Rollback Strategy

  • Feature-Flag-Disable: marketing-funnel-enabled global. Wenn off, /api/leads/capture liefert 404; Admin-UI versteckt.
  • API-Disable: Env LEADS_CAPTURE_ENABLED=false deaktiviert nur Endpoint, lässt Admin-Inbox sichtbar (für Read-Only-Modus während Daten-Migration).
  • CDN-SDK-Versioning: alte SDK-Version bleibt unter marketing.v1.js; Customer-Embeds pinnen Version → neue Bugs betreffen nur neue Embeds.
  • DB-Rollback: Lead-Tabelle gefahrlos droppbar (kein anderer Code referenziert) — aber Daten-Verlust bei Rollback nach Production-Use. Vor Rollback: Daten-Export.

Security Notes

  • PII in Leads: Email + Name + IP sind PII. Tenant-Isolation entfällt (Leads sind global Sales-side). Aber: Read-Permission strikt auf role=admin oder neue Role role=sales. CSV-Export ist Audit-Event-pflichtig (wer hat wann was exportiert).
  • Honeypot-Field-Naming: Field-Name botfield ist generisch — Bots könnten lernen. Field-Name aus Env (HONEYPOT_FIELD_NAME) zufällig pro Deployment.
  • CORS-Whitelist als Defense: echter Bot-Schutz kommt von Turnstile, CORS schützt nur Browser-Origin-Header-Vertrauende-Requests. Server-to-Server-Bots umgehen CORS — daher Rate-Limit zwingend.
  • Email-Enumeration: Capture-Endpoint sollte nicht zwischen "Lead existiert schon" und "neuer Lead" unterscheiden (Dedup-Logic ist server-internal, Response-Code identisch). Sonst kann Bot Email-Listen verifizieren.
  • GDPR/DSGVO: Lead-Capture braucht Consent-Mechanismus. SDK-API soll consentGiven-Boolean unterstützen; Marketing-Page-Operator verantwortet Consent-Banner. Lead-Right-to-be-Forgotten: Delete-Action in Admin-UI vorhanden, schreibt AuditEvent für Audit.
  • IP-Logging: wir loggen ipAddress für Rate-Limit + Abuse-Detection. DSGVO-Doku-Update notwendig (Phase 10 dsgvo-bulk-ops.md referenziert das).

Subagent Recommendations

  • Planning: Plan-Subagent mit Detail-MD-Input.
  • Research: Explore-Subagent: "Existing Mailchimp/Brevo-Integration-Versuche im Codebase prüfen — gibt es schon Code-Pfade die wir wiederverwenden können oder ablösen?"
  • Implementation Step 1-2 (Model + API): general-purpose-Subagent.
  • Implementation Step 3-4 (SDK + CDN-Deploy): general-purpose-Subagent mit Bundle-Size-Budget-Focus.
  • Implementation Step 5-6 (Admin-UIs): general-purpose-Subagent + subframe:design für Inbox-Layout.
  • Implementation Step 7-8 (Audit + Export): general-purpose-Subagent.
  • Tests: separater general-purpose-Subagent inkl. SDK-E2E-Setup.
  • Verification: gsd-verifier.
  • Code-Review: gsd-code-reviewer + security-review (PII + CORS + Anti-Spam).

Staging-Deployment-Steps

  1. Branch agent/05-marketing-phase-1 pushen → Coolify-Auto-Deploy
  2. Asset-CDN für marketing.js deployed (CI-Step)
  3. Migration pnpm prisma migrate deploy
  4. Turnstile-Test-Keys von Cloudflare in Staging-Env setzen
  5. MARKETING_DOMAINS=marketing.staging.toolctrl.de,test.toolctrl.de setzen
  6. Staging-Test:
    • Test-Landing-Page mit eingebundenem marketing.js deployen
    • Form auf Page submitten → Lead in /admin/marketing/leads sichtbar
    • UTM-Test: URL mit ?utm_source=test&utm_campaign=phase1, danach submit → UTMs in Lead-Row
    • CORS-Block: Form aus nicht-whitelisteter Domain → Reject sichtbar in Devtools
    • Honeypot: programmatischer Submit mit gefülltem botfield → 200 OK aber kein Lead
    • Rate-Limit: 10 schnelle Submits von gleicher IP → 6.+ wird 429
    • Admin-Inbox: Filter + Search + Bulk-Disqualify + CSV-Download
    • Audit-Timeline: Status-Änderung → Eintrag erscheint sofort
  7. SDK-Bundle-Size-Verify via curl ... | gzip -c | wc -c < 5120

Deviations from Plan

  • Admin sidebar nav entry not yet wired into AdminSidebar.tsx — resolved 2026-05-20 f4c2f908: added UI_NAV_IDS.adminMarketing collapsible group under the market segment exposing Leads / Funnels / Integrations / Email-sequences sub-items, all gated on the existing admin.marketing.* permissions; added the four route slugs to canonicalAdminRouteSlugs so the sidebar route-coverage test stays green.
  • CSV export route on the admin UI — resolved 2026-05-20 89c79e9f: list loader now branches on ?export=csv and streams a 50k-row-capped CSV byte-compatible with the operator-side toolctrl-cli leads export (same 20-column ordering, same RFC-4180 escaping). Each export writes an admin.marketing.leads.csv_export audit-event with the filter snapshot + pii/gdpr regulatory scope so SIEM/audit-trail readers can reconstruct who exported what.
  • packages/marketing-sdk-js/ workspace not split out — the SDK lives as a single hand-minified file at apps/web/public/embed/marketing.js to mirror the form-widget pattern (#58). When a build-step + npm-publish flow is needed (e.g. for first-party customer sites embedding it from package managers), promote to a package.
  • CDN deploy CI step + SRI hash deferred — the file ships from the apps/web origin today; CDN setup belongs to the asset-CDN task (#46).
  • Audit-Event integration replaced with LeadEvent rows (status-changed, note-added, captured, re-captured, converted). The detail-view timeline reads from LeadEvent so no separate AuditEvent join was needed. Cross-system audit (e.g. exporting to SIEM) is a follow-up.
  • Drag-drop form-builder UI explicitly out-of-scope for Phase 1.
  • E2E Playwright + integration tests deferred — unit-test coverage spans origin matcher, capture endpoint response shapes (8 cases), SDK size+behavior (8 cases). Coverage gates are not yet on.

Progress Log

  • 2026-05-17 14:00 — 7241dd1c — Lead + LeadEvent prisma model + migration with status enum + first-touch attribution
  • 2026-05-17 14:00 — cfcd8fca — capture endpoint with origin allowlist + honeypot + rate-limit + Turnstile (19 unit tests)
  • 2026-05-17 14:03 — 53aa5f52 — hand-minified marketing.js SDK (~2KB gz, 8 size+behavior tests)
  • 2026-05-17 14:22 — 1f62b162 — admin /admin/marketing-leads list + detail view + status-machine + notes + timeline
  • 2026-05-17 14:25 — be6c8407 — toolctrl-cli leads {list,export,delete}
  • 2026-05-17 14:29 — 0c17d998 — env vars + docs/features/marketing-funnel.md
  • 2026-05-20 — f4c2f908 — admin sidebar wiring (Marketing funnel collapsible group: Leads / Funnels / Integrations / Email sequences) + en/de i18n keys + canonical route slugs
  • 2026-05-20 — 89c79e9f — admin-UI CSV-export (?export=csv on list loader, 50k cap, audit-logged, header download button) + serializeLeadsCsv helper with 9 unit tests