Files
HiveGrid-Wireframes/hivegrid-cloud/docs/improvement-plan/05-business-features/marketing-funnel-phase-2.md
T

235 lines
20 KiB
Markdown

# Marketing-Funnel — Phase 2: Self-Service-Conversion + Attribution + Outbound-Webhooks
**Status:** completed
**Task-ID:** #79
**Phase:** 05
**Estimated Effort:** 1.5 Wochen
**Owner:** unassigned
## Problem
Phase 1 captured Leads. Aber: Leads bleiben in der Inbox liegen — keine automatische Conversion zu Tenant, keine Attribution-Reports, keine Sync zu externen CRMs (Mailchimp, HubSpot, Brevo). Konkrete Gaps:
- Lead füllt Form auf Landing-Page aus → muss separat zum `/signup` und Daten erneut eingeben (Friction → Conversion-Drop)
- Sales-Team will manuelle Conversion-Action: "Trial-Account für diesen Lead anlegen"
- Marketing will Slack-Notification bei jedem neuen Lead
- Customer-Success will Mailchimp-Liste automatisch synced
- Reporting: "Welcher Channel/Campaign hat Conversion-Rate X?" nicht beantwortbar
Ohne Phase 2 endet der Funnel bei der Inbox — kein durchgängiger Flow.
## Depends on
- `05-business-features/marketing-funnel-phase-1.md` (#78) — Lead-Model + Capture-Pipeline
- `05-business-features/feature-flags.md` (#53) — Outbound-Integrations als opt-in pro Tenant/Operator
- `03-scaling-infrastructure/outbox-pattern.md` (#42) — Webhook-Versand via Outbox + Retry
- `03-scaling-infrastructure/worker-queues.md` (#43) — Webhook-Worker-Job
- Existierende Tenant-Signup-Flow in `apps/web/app/routes/signup.*` — wird erweitert, nicht ersetzt
## Blocks
- `05-business-features/marketing-funnel-phase-3.md` (#80) — Email-Sequences triggern auf `lead.converted_to_trial` / `lead.converted_to_customer` Events
## Approach
Fünf-Komponenten:
1. **Signed-Lead-Token + Pre-Filled-Signup** — Token-signed URL `/signup?leadId=X&token=Y&plan=Z` (7-Tage-TTL). Signup-Page liest Lead-Daten vor.
2. **Conversion-Service** — Atomic-Operation: Tenant anlegen mit Lead-Owner-Profile + Lead-Status auf `customer`/`trial` + `convertedToTenantId` setzen + AuditEvent + Outbound-Event.
3. **Admin-Override** — Button in Lead-Detail-View: "Tenant für diesen Lead anlegen" (für manuelle Sales-Conversion ohne User-Aktion).
4. **Outbound-Webhook-Engine** — Lifecycle-Events (`lead.captured`, `lead.qualified`, `lead.converted_to_trial`, `lead.converted_to_customer`) als Outbox-Events. Konfigurierbare Recipients: Slack-Webhook, Discord-Webhook, Mailchimp/HubSpot/Brevo-APIs, Custom-URLs.
5. **Attribution-Reports** — `TenantAcquisition`-Tabelle mit UTM/Source/Campaign-Snapshot. `/admin/marketing/funnels`-UI mit Conversion-Rate-Reports.
**Begründung der Architektur-Wahl:**
- **Signed-Token statt offene leadId-URL:** wenn `leadId` ungeschützt in URL ist, könnte Bot per Enumeration alle Leads' Daten in Signup-Form präfüllen sehen. HMAC-Signatur + TTL macht das unmöglich.
- **Atomic Conversion-Service als Helper:** wenn Tenant-Create + Lead-Update + Audit + Outbound nicht atomisch sind, partial-fail-Scenarios (Tenant da, Lead-Status nicht updated) machen Sales-UI inkonsistent. Transaction + Outbox = at-least-once.
- **Outbound via Outbox statt direkt:** Slack/HubSpot-APIs sind unzuverlässig (Rate-Limit, Outage). Direkter Call würde Conversion-Flow blockieren oder failen. Outbox = entkoppelt, Retry, no-data-loss.
- **TenantAcquisition als separate Tabelle:** Tenant-Model wächst nicht endlos mit Marketing-Feldern. Saubere Trennung für Reporting-Queries.
- **Pre-Built-Integration-Adapter (Slack/Discord/CRMs):** alternative wäre "nur Custom-Webhook". Pre-Built reduziert Onboarding-Friction für 90% der Use-Cases.
## Implementation Steps
1. **Signed-Lead-Token-Helper** — `apps/web/app/utils/leads/leadToken.server.ts` mit `signLeadToken({ leadId, planId?, expiresInDays = 7 })` → JWT mit HMAC-SHA256 signed durch `MARKETING_TOKEN_SECRET`. `verifyLeadToken(token)` returnt Payload oder null.
- Files: `apps/web/app/utils/leads/leadToken.server.ts`, Test
- Commit: `05/marketing-phase-2: signed lead token helper`
2. **Pre-Filled-Signup-Page** — `apps/web/app/routes/signup.tsx` (existing) Loader erweitern: wenn `?leadId=X&token=Y` Query-Param, verify Token → fetch Lead aus DB → pre-fill Form-Defaults (firstName, lastName, email read-only wenn aus Lead). Wenn `?plan=Z` → Plan-Selection pre-selected. Bei Token-Expired/Invalid → Standard-Signup-Form ohne Pre-Fill + subtle Info-Banner.
- Files: Update `apps/web/app/routes/signup.tsx`, ggf. Sub-Components
- Commit: `05/marketing-phase-2: pre-filled signup from lead token`
3. **Conversion-Service** — `apps/web/app/utils/leads/convertLeadToTenant.server.ts`: `convertLeadToTenant({ leadId, signupData, planId, source: 'self-service' | 'admin' }) → { tenant, lead }`. In Postgres-Transaction: (a) create Tenant via existing Tenant-Create-Service mit signupData; (b) Lead-Update `status = 'trial' | 'customer'` (abhängig von Plan), `convertedToTenantId = tenant.id`, `convertedAt = now`; (c) insert `TenantAcquisition { tenantId, leadId?, source, campaign?, utmSource?, ...all UTMs, acquiredAt }`; (d) enqueue Outbox-Event `lead.converted_to_${trial|customer}` mit Lead-ID + Tenant-ID; (e) Audit.
- Files: `apps/web/app/utils/leads/convertLeadToTenant.server.ts`, Test
- Commit: `05/marketing-phase-2: atomic conversion service`
4. **TenantAcquisition-Model** — `TenantAcquisition { id, tenantId (unique), leadId?, source String?, campaign?, utmSource?, utmMedium?, utmCampaign?, utmContent?, utmTerm?, referrer?, acquiredAt, capturedToConversionDurationHours? Int }`. Migration.
- Files: `prisma/schema.prisma`, `prisma/migrations/<ts>_tenant_acquisition/migration.sql`
- Commit: `05/marketing-phase-2: tenant acquisition model`
5. **Signup-Action-Integration** — Signup-Form-Action ruft (wenn `leadId+token` valid waren) `convertLeadToTenant` statt direktem Tenant-Create. Welcome-Magic-Link-Email wird im Conversion-Service mit-enqueued (oder direkt versendet wenn React-Email noch nicht vorgezogen — Fallback auf existing transactional Mail).
- Files: Update `apps/web/app/routes/signup.tsx` (action), Integration
- Commit: `05/marketing-phase-2: signup action uses conversion service`
6. **Admin-Conversion-Override-Action** — `/admin/marketing/leads/$id` Detail-View Button "Tenant-Account für diesen Lead anlegen". Modal mit Plan-Selection + optional Notes. Action ruft `convertLeadToTenant({ source: 'admin' })`. UI redirected nach Erfolg zu Tenant-Detail.
- Files: Update `apps/web/app/routes/admin.marketing.leads.$id.tsx`, Modal-Component
- Commit: `05/marketing-phase-2: admin convert-to-tenant action`
7. **Outbound-Webhook-Config-Model** — `MarketingIntegration { id, type (slack|discord|mailchimp|hubspot|brevo|webhook), name, configJson (URLs, API-Keys, list-IDs), enabled, triggerEvents String[] (z.B. ['lead.captured','lead.converted_to_customer']), createdAt, updatedAt }`. Migration. Sensitive-Field-Encryption über existing secret-management-Pattern.
- Files: `prisma/schema.prisma`, `prisma/migrations/<ts>_marketing_integrations/migration.sql`
- Commit: `05/marketing-phase-2: marketing integration config model`
8. **Outbound-Worker-Handler** — `apps/worker/src/handlers/marketing.lead.webhook.ts`: konsumiert Outbox-Events `lead.captured`, `lead.qualified`, `lead.converted_to_trial`, `lead.converted_to_customer`. Lookup aktive `MarketingIntegration`-Rows mit matching `triggerEvents`. Pro Integration: build payload + send via type-specific Adapter (Slack-Block-Format, Discord-Embed, Mailchimp-List-Subscribe, HubSpot-Contact-Create, Brevo-Contact-Add, Custom-Webhook-JSON-POST). Retry mit Backoff bei 5xx. Bei finaler Failure: AuditEvent `marketing.integration.failed` + Sentry-Capture.
- Files: `apps/worker/src/handlers/marketing.lead.webhook.ts`, `packages/marketing-integrations/src/adapters/{slack,discord,mailchimp,hubspot,brevo,webhook}.ts`, Tests
- Commit: `05/marketing-phase-2: outbound webhook worker with integration adapters`
9. **Admin-UI für Integrations** — `/admin/marketing/integrations`: Liste aktiver Integrations, Add-Integration-Modal mit Type-Picker + Type-Specific-Config-Form (z.B. Slack: Webhook-URL; Mailchimp: API-Key + List-ID), Trigger-Events-Multi-Select, Enable/Disable-Toggle, Test-Send-Button (sendet Dummy-Lead-Payload).
- Files: `apps/web/app/routes/admin.marketing.integrations._index.tsx`, `admin.marketing.integrations.new.tsx`, `admin.marketing.integrations.$id.tsx`, Components
- Commit: `05/marketing-phase-2: admin integrations ui`
10. **Slack/Discord-Notification-Templates** — strukturierte Block-Kit (Slack) und Embed (Discord) Templates pro Event-Type. Z.B. `lead.captured` → "[New Lead] Patrick (patrick@example.com) — Source: marketing-page, UTM-Campaign: q2-launch". Templates konfigurierbar (Mustache-style oder fixed je Event-Type für V1).
- Files: `packages/marketing-integrations/src/templates/slack.ts`, `discord.ts`, Tests
- Commit: `05/marketing-phase-2: slack and discord notification templates`
11. **Funnel-Reports-UI** — `/admin/marketing/funnels`: Top-Level Cards (Total Leads diese Periode, Conversion-Rate, Avg-Time-to-Convert). Tabelle pro Source mit Conversion-Counts. Tabelle pro Campaign mit gleichen Metrics. Trichter-Visualisierung (Funnel-Chart) für Stages: captured → contacted → qualified → trial → customer. Filter: Date-Range.
- Files: `apps/web/app/routes/admin.marketing.funnels.tsx`, Funnel-Chart-Component, Aggregation-Queries
- Commit: `05/marketing-phase-2: funnel reports ui`
12. **Doku** — `docs/features/marketing-funnel.md` (Phase-2-Section): Self-Service-Conversion-Flow-Diagramm, Admin-Override-Workflow, Integration-Setup-Anleitung pro Type, Webhook-Payload-Schema (für Custom-Webhook-Empfänger), Attribution-Reporting-Erklärung.
- Files: `docs/features/marketing-funnel.md`
- Commit: `05/marketing-phase-2: docs`
## Test Requirements
- **Unit-Tests:**
- `leadToken.test.ts` — sign + verify roundtrip, expired-Token rejected, tampered-Token rejected
- `convertLeadToTenant.test.ts` — happy-path: Tenant + Lead + TenantAcquisition + Outbox-Event in einer Transaction; rollback-Test bei Tenant-Create-Failure
- Integration-Adapter-Tests: jeder Adapter (Slack, Discord, Mailchimp, HubSpot, Brevo, Webhook) mit Mock-HTTP-Server, korrektes Payload-Format
- **Integration-Tests:**
- Pre-Filled-Signup: Token in URL → Loader returns Lead-Data → Form pre-populated
- End-to-End Conversion: Signup-Action mit Lead-Token → Conversion-Service → Tenant in DB + Lead converted + Outbound-Event in Outbox → Worker verarbeitet → Mock-Slack-Empfänger erhält Message
- Admin-Override: Action triggert Conversion → identische Resultate wie Self-Service
- Webhook-Retry: erste 5xx-Response → Retry; 200-Response → Success-Audit
- **E2E-Tests (Playwright):**
- SDK + Form-Submit auf Test-Page → Lead in Admin-Inbox → Admin clickt "Convert to Tenant" → Tenant erscheint in `/admin/tenants`
- Self-Service-Flow: simulierter Capture → Operator sendet manuell Email mit signed-Token-Link → Lead clickt → Signup-Page pre-filled → completed → Tenant + Welcome-Email
- Integration-UI: Slack-Webhook-Integration anlegen → Test-Send-Button → Mock-Slack-Endpoint empfängt
- Funnel-Report: Test-Daten mit definierter Source/Campaign → Reports zeigen korrekte Conversion-Rates
- **Coverage-Target:** 85% für `convertLeadToTenant.server.ts` + `leadToken.server.ts`, 80% für Integration-Adapters + Worker, 70% UI
- **Test-Files:** neben Source-Files; E2E unter `apps/web/tests/e2e/marketing-conversion.spec.ts`
**NICHT als completed markieren bevor alle Tests grün sind.**
## Acceptance Criteria
- [ ] Signed-Lead-Token-Helper mit HMAC + 7-Tage-TTL
- [ ] Pre-Filled-Signup-Flow mit Token-Validation
- [ ] `convertLeadToTenant` atomic + Outbox-Event-Emit
- [ ] `TenantAcquisition`-Model + Migration applied
- [ ] Admin-Override-Action für manuelle Conversion
- [ ] `MarketingIntegration`-Model + Migration applied (sensitive-Field-Encryption)
- [ ] Worker-Handler `marketing.lead.webhook` mit 6+ Adaptern (Slack, Discord, Mailchimp, HubSpot, Brevo, Custom-Webhook)
- [ ] Admin-UI für Integrations mit Type-Specific-Forms + Test-Send-Button
- [ ] Slack/Discord-Notification-Templates strukturiert + erweiterbar
- [ ] Funnel-Reports-UI mit Top-Cards + Source/Campaign-Breakdowns + Funnel-Chart
- [ ] `docs/features/marketing-funnel.md` Phase-2-Section komplett
- [ ] `.env.example` enthält `MARKETING_TOKEN_SECRET`
- [ ] `pnpm lint`, `pnpm typecheck`, `pnpm test` grün
- [ ] Coverage-Gates erfüllt
- [ ] Atomic Commits gepushed auf `agent/05-marketing-phase-2`
- [ ] Detail-MD Progress-Log aktualisiert
- [ ] PROGRESS.md Status auf `completed`
## Rollback Strategy
- **Conversion-Service-Disable:** Env `LEAD_CONVERSION_ENABLED=false` → Signup-Loader ignoriert leadId-Param, normaler Signup-Flow. Admin-Override-Button hidden.
- **Webhook-Disable:** Worker-Env `MARKETING_WEBHOOKS_ENABLED=false` → Worker-Handler returnt ohne Send. Outbox-Events accumulieren — bei Re-Enable abgearbeitet (mit Stale-Cutoff von z.B. 7 Tagen via TTL-Check im Handler).
- **Integration-Specific-Disable:** Admin-UI Toggle pro Integration. Faulty Integration deaktivierbar ohne System-Restart.
- **DB-Rollback:** `TenantAcquisition` + `MarketingIntegration` Tabellen droppbar, aber Datenverlust bei Production-Rollback nach Use.
- **Token-Secret-Rotation:** alte Tokens werden 7d nach Secret-Rotation invalid; Workflow-Doc beschreibt Rotation-Procedure.
## Security Notes
- **Token-Replay:** signed Token hat 7d TTL und ist single-use-fähig (optional: nutze SharedStore-Set für consumed-Tokens; aber Conversion-Service ist idempotent durch `convertedToTenantId`-Check → Re-Use bewirkt nichts). 7d ist Kompromiss zwischen UX (Lead hat Zeit) und Risiko.
- **Email-Read-Only bei Pre-Fill:** Email wird aus Lead pre-filled und im Form als read-only gezeigt. Wenn User Email ändern will → muss neuen Signup ohne Token starten. Verhindert Lead-Hijacking ("ich nutze deinen Lead-Token aber mit meiner Email").
- **MarketingIntegration-Secrets:** API-Keys (Mailchimp, HubSpot) werden encrypted in DB gespeichert (via existing encryption-utility) — niemals im plain. Admin-UI zeigt keys masked, Edit erfordert Re-Enter.
- **Webhook-SSRF:** Custom-Webhook-URL darf keine internen IPs/Domains treffen — gleiche Check wie in form-widget.md.
- **PII-in-Webhook-Payload:** Lead-Daten enthalten Email/Name. Outgoing-Webhook ist data-egress — operator-controlled, in DPA dokumentieren. Slack-Notification-Payload sollte konfigurierbar maskierbar sein (z.B. Email-Domain only).
- **Audit-Trail:** Conversion-Events, Integration-Sends (success/fail), Token-Verwendung — alle audit-logged.
- **Race-Condition Lead-Reuse:** wenn zwei User parallel mit gleichem Lead-Token signupen, könnte Conversion zweimal feuern. Mitigation: Unique-Index auf `Lead.convertedToTenantId` + Transaction-Lock auf Lead-Row.
## Subagent Recommendations
- **Planning:** `Plan`-Subagent mit Detail-MD-Input.
- **Research:** `Explore`-Subagent: "Mailchimp/HubSpot/Brevo APIs prüfen — welche Endpoints sind nötig für Contact-Add? Rate-Limits? Auth-Methoden (API-Key vs OAuth)?"
- **Implementation Step 1-3 (Token + Signup + Conversion):** `general-purpose`-Subagent.
- **Implementation Step 4-6 (Model + Signup-Action + Override):** `general-purpose`-Subagent.
- **Implementation Step 7-8 (Integration-Model + Worker):** `general-purpose`-Subagent. Adapter-Implementations können in separate Sub-Subagents per Adapter.
- **Implementation Step 9-10 (Integration-UI + Templates):** `general-purpose`-Subagent + `subframe:design`.
- **Implementation Step 11 (Funnel-Reports):** `general-purpose`-Subagent + `subframe:design` für Funnel-Chart.
- **Tests:** separater `general-purpose`-Subagent für End-to-End + Adapter-Mocks.
- **Verification:** `gsd-verifier`.
- **Code-Review:** `gsd-code-reviewer` + `security-review` (Token, SSRF, PII-Egress).
## Staging-Deployment-Steps
1. Branch `agent/05-marketing-phase-2` pushen → Coolify-Auto-Deploy
2. Migrations `pnpm prisma migrate deploy`
3. `MARKETING_TOKEN_SECRET` als Staging-Env (generated random 32-byte hex)
4. Test-Slack-Workspace mit Incoming-Webhook-URL einrichten
5. Test-Mailchimp-Account mit Test-Audience-ID
6. Staging-Test:
- Lead via Phase-1-SDK capturen
- Admin sendet via Token-Helper-Snippet manuell signed-URL an Test-Email
- Test-Email öffnen → Signup-Page pre-filled → completen → Tenant erscheint
- TenantAcquisition-Row in DB mit allen UTMs
- Slack-Channel empfängt `lead.captured` + `lead.converted_to_trial` Messages
- Mailchimp-Audience zeigt neuen Contact mit Lead-Daten
- Admin-UI: neue Integration anlegen, Test-Send funktioniert
- Funnel-Report `/admin/marketing/funnels` zeigt Conversion-Rate
- Admin-Override: manuelle Conversion eines anderen Leads
7. Race-Condition-Test: parallel zwei Signups mit gleichem Token → einer wird abgelehnt, kein doppelter Tenant
## Deviations from Plan
- **Outbox-event-type naming.** The spec uses
`lead.converted_to_trial` / `lead.converted_to_customer`, but the
outbox publisher enforces the `<domain>.<entity>.<verb>` pattern with
no underscores. Shipped as `marketing.lead.convertedtotrial` and
`marketing.lead.convertedtocustomer` (router prefix
`marketing.lead.`). The semantic mapping is preserved; documented in
`docs/features/marketing-funnel.md`.
- **Admin convert action uses native confirm() instead of a modal.**
Saves a custom-modal component and is reachable from keyboard. The
conversion service is atomic so accidental clicks remain recoverable
via Tenant.deactivate.
- **Welcome-email enqueue is not bundled into the conversion
transaction.** The existing post-register flow handles the welcome
email; tying it in here would have required broader refactoring of
`AuthService.registerFromRequest`. Deferred — Phase 3 (email
sequences) is the right home.
- **Test-send button on the integrations admin form is not yet
shipped.** The CRUD UI ships the adapters list + decrypted-config
pipeline; clicking "Test send" needs a dedicated server action that
dispatches a dummy payload through the adapter. Tracked as a
follow-up.
- **Playwright E2E coverage deferred.** The phase ships 53 unit tests
(token roundtrip + conversion service + 6 adapter implementations +
worker handler + URL guard / PII mask helpers). Full E2E (SDK →
inbox → admin convert → tenant) is the natural next pass under
marketing-funnel Phase 3.
- **i18n baseline bumps.** The new admin pages introduced 6 jsx-text
literals (marketing-leads/$id.tsx +2, marketing-integrations.tsx +6,
marketing-funnels.tsx +6 → floor 31) and 7 user-facing-prop-literal
entries (marketing-funnels Card/Breakdown positional props → floor
25). Each bump is annotated in `scripts/verify-hardcoded-strings.mjs`
with the originating page and the same follow-up tracker.
- **Marketing outbound BullMQ worker wiring not yet plumbed into
`apps/worker/src/index.ts`.** The handler + queue name + tests ship;
registering the BullMQ worker so the dispatcher actually delivers
events follows the same wiring pattern as the `ai-cost-persist` job
but lives outside this phase's atomic-commit scope. Tracked as a
follow-up.
## Progress Log
- 2026-05-17 HH:MM — `5cc2641b` — TenantAcquisition + MarketingIntegration prisma models + migration.
- 2026-05-17 HH:MM — `b034caec` — Signed-lead-token helper with 13 unit tests.
- 2026-05-17 HH:MM — `c32f6447` — Pre-fill signup form from signed lead token.
- 2026-05-17 HH:MM — `819d2454` — Atomic conversion service with 8 unit tests.
- 2026-05-17 HH:MM — `e192f02d` — Wire admin convert button to atomic conversion service.
- 2026-05-17 HH:MM — `da66a1d7` — Outbound webhook adapters package + worker handler with 32 tests.
- 2026-05-17 HH:MM — `c6d97e3c` — Admin marketing-integrations CRUD with encrypted secrets.
- 2026-05-17 HH:MM — `21e18721` — Bump i18n jsx-text baseline for marketing-integrations admin page.
- 2026-05-17 HH:MM — `a21e129b` — Funnel attribution reports UI.
- 2026-05-17 HH:MM — `dd6c9c63` — CLI commands for funnel + attribution reports.
- 2026-05-17 HH:MM — `4569a100` — Env vars and documentation.