214 lines
19 KiB
Markdown
214 lines
19 KiB
Markdown
# Email-Sequence Engine (Marketing-Funnel Phase 3)
|
||
|
||
**Status:** Phase 05 Task #80 — the sequence data model, pure engine foundation, admin UI/CLI, local E2E readiness seed fixtures, unsubscribe route, open/click tracking routes, worker bootstrap contracts, delayed-step enqueueing foundation, bundled template renderer, `@toolctrl/mail` Mailjet adapter, signed Mailjet bounce webhook handling, unit-tested stale-lead/trial-expiring cron runner functions, and live admin statistics views are in place. This is shipped foundation/admin-preview work, not an end-to-end production email-delivery claim for marketing sequences. The worker entrypoints are default-safe: `MARKETING_SEQUENCES_ENABLED` must be explicitly set to `true` before sequence subscribers do work; entrypoint tests prove the sequence workers and readiness probes stay disabled while the flag is unset. Keep it unset or `false` in production until production/demo seed rollout evidence, statistics aggregation cron, Playwright E2E coverage, delivery webhook production evidence, and production scheduled-job registration/deploy evidence are complete.
|
||
|
||
## What ships
|
||
|
||
| Component | Package | Notes |
|
||
| ------------------------- | ------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------- |
|
||
| Data model | `prisma/schema/emailSequences.prisma` | `EmailSequence`, `EmailSequenceRun`, `EmailSequenceStepDelivery` |
|
||
| Engine | `@toolctrl/email-sequences` | pure-logic — types, conditions, engine, step-runner |
|
||
| Template definitions | `@toolctrl/email-templates` | welcome / trial-reminder / re-activation / generic-step renderer used by the worker runtime |
|
||
| Trigger worker contract | `apps/worker/src/jobs/email-sequence-trigger.ts` | consumes matching outbox events, persists trigger context, and starts runs idempotently |
|
||
| Scheduler worker contract | `apps/worker/src/jobs/email-sequence-scheduler.ts` | step-runner foundation; runtime enqueues idempotent delayed jobs and sends through `@toolctrl/mail` when Mailjet env is configured |
|
||
| Unsubscribe route | `apps/web/app/routes/unsubscribe.$token.tsx` | RFC-8058 one-click (GET + POST) |
|
||
| Tracking — open | `apps/web/app/routes/email/track-open.$deliveryId[.gif].tsx` | 1×1 transparent GIF |
|
||
| Tracking — click | `apps/web/app/routes/email/track-click.$deliveryId.tsx` | 302-redirect + first-touch update |
|
||
| Provider bounce webhook | `POST /v1/email-sequences/mailjet/webhooks` | Signed Mailjet ingress marks correlated sequence deliveries as bounced |
|
||
| Local readiness seed | `scripts/seed-e2e-readiness.ts` | deterministic welcome / trial-expiring / stale-lead / customer-welcome definitions for local E2E fixtures |
|
||
|
||
This page claims active ownership of the email-sequence subscriber on
|
||
`outbox.marketing`.
|
||
It does not classify Email Sequences as the `email-marketing` CRM-contact
|
||
campaign plugin candidate from
|
||
`docs/feature-decisions/email-marketing-decision-report.md` or
|
||
`docs/feature-manifests/plugin-candidates.json`. Treat Email Sequences as
|
||
foundation/admin-preview work until the production follow-ups below land.
|
||
The generic marketing integration handler/adapters for Slack, Discord,
|
||
Mailchimp, HubSpot, Brevo, and custom webhooks are documented in
|
||
`docs/features/marketing-funnel.md`; as of this status reconciliation that
|
||
handler is pure foundation and the general subscriber is not bootstrapped from `apps/worker/src/index.ts`.
|
||
|
||
## Architecture
|
||
|
||
The diagram below shows the sequence flow. The delayed-job producer, bundled template renderer, and `@toolctrl/mail` Mailjet adapter exist, but missing Mailjet configuration still fails closed and the remaining production follow-ups below must be completed before this path should be treated as approved production delivery.
|
||
|
||
```
|
||
┌──────────────────────────────┐
|
||
lead.captured / tenant.signup outbox event
|
||
│
|
||
▼
|
||
┌──────────────────────────────┐
|
||
│ email-sequence-trigger │ apps/worker/src/jobs
|
||
│ - lookup matching seqs │
|
||
│ - createRun (idempotent) │
|
||
│ - records pending step │
|
||
└──────────────┬───────────────┘
|
||
│ delayed enqueue
|
||
▼
|
||
┌──────────────────────────────┐
|
||
│ email-sequence-scheduler │
|
||
│ - runStep │
|
||
│ ├── conditions │
|
||
│ ├── renderTemplate │ bundled sequence templates
|
||
│ ├── sendMail │ @toolctrl/mail Mailjet adapter
|
||
│ ├── markDelivery │
|
||
│ └── advanceSequenceRun │
|
||
└──────────────────────────────┘
|
||
```
|
||
|
||
## Trigger events (per spec)
|
||
|
||
- `lead.captured` — `marketing.lead.captured` outbox prefix is normalised
|
||
- `lead.qualified`
|
||
- `lead.converted_to_trial` / `lead.converted_to_customer`
|
||
- `lead.stale_60d` — emitted by the stale-lead cron runner once scheduled in the worker runtime
|
||
- `tenant.signup`
|
||
- `tenant.trial_expiring`
|
||
- `tenant.cancelled`
|
||
- `custom` — operator-defined
|
||
|
||
## Step-condition DSL
|
||
|
||
```jsonc
|
||
{
|
||
"path": "lead.status",
|
||
"op": "eq | neq | in | not_in | exists | not_exists",
|
||
"value": "qualified", // string | number | boolean | array (for in / not_in)
|
||
}
|
||
```
|
||
|
||
Per-step `conditions` are ANDed; sequence-level `conditions` gate the whole sequence at trigger-time.
|
||
|
||
## Unsubscribe + List-Unsubscribe
|
||
|
||
When production delivery is wired, every sequence email must inject:
|
||
|
||
```
|
||
List-Unsubscribe: <https://app.example.com/unsubscribe/<token>>, <mailto:unsubscribe+<token>@example.com>
|
||
List-Unsubscribe-Post: List-Unsubscribe=One-Click
|
||
```
|
||
|
||
The token is `u1.<base64url(payload)>.<base64url(hmac-sha256)>` signed with `MARKETING_TOKEN_SECRET` (falls back to `LEAD_CONVERSION_SECRET`). Tokens never expire — secret rotation is the kill-switch.
|
||
|
||
## Open / click tracking
|
||
|
||
- Open: HTML `<img src="https://app.example.com/email/track-open/<deliveryId>.gif">` — sets `openedAt` on first fetch.
|
||
- Click: Every anchor URL is rewritten to `https://app.example.com/email/track-click/<deliveryId>?u=<encoded>` — sets `clickedAt` on first hit, 302s to the original target. Only `http:` / `https:` schemes accepted (no `javascript:`, `data:` etc.).
|
||
|
||
### Caveats
|
||
|
||
- Apple Mail Privacy Protection pre-fetches images → false-positive opens.
|
||
- Image-blocking mail clients (Outlook default) suppress the pixel → false-negative opens.
|
||
- Click-tracking introduces an extra hop — bots and security scanners may pre-resolve URLs, inflating clicks.
|
||
|
||
The admin stats dashboard documents these as approximations. It currently computes live per-sequence statistics from delivery rows; a persisted aggregation cron remains a follow-up for larger datasets.
|
||
|
||
## Provider bounce webhooks
|
||
|
||
`POST /v1/email-sequences/mailjet/webhooks` accepts signed Mailjet provider events using `MAILJET_WEBHOOK_SECRET`. Bounce and rejection events correlate against `EmailSequenceStepDelivery.providerMessageId`, set `status=bounced`, record `bouncedAt` / `errorMessage`, and fail the related `EmailSequenceRun`. Unknown provider message ids and non-bounce events return `202` without echoing provider payloads to avoid retry storms.
|
||
|
||
## Templates
|
||
|
||
Minimal React-Email-style template definitions exist. The worker runtime turns these definitions into a `@toolctrl/mail` send payload, but the remaining production follow-ups below still need to land before marketing sequences should be cited as approved production mail end to end.
|
||
|
||
| Template | Trigger example | Variables |
|
||
| -------------------- | ---------------------------------- | -------------------------------------------------------------------------------------- |
|
||
| `WelcomeEmail` | `lead.captured` day-0 | `recipientName`, `productName`, `ctaUrl`, `ctaLabel?`, `unsubscribeUrl`, `senderName?` |
|
||
| `TrialReminderEmail` | `tenant.trial_expiring` | `recipientName?`, `productName`, `daysRemaining`, `upgradeUrl`, `unsubscribeUrl` |
|
||
| `ReActivationEmail` | `lead.stale_60d` | `recipientName?`, `productName`, `returnUrl`, `highlight?`, `unsubscribeUrl` |
|
||
| `GenericStepEmail` | onboarding-drip / operator-defined | `headline`, `bodyParagraphs[]`, `ctaUrl?`, `ctaLabel?`, `unsubscribeUrl` |
|
||
|
||
The full React-Email migration tracked under #75 is no longer the blocker here; the worker runtime renders the bundled sequence templates and sends through `@toolctrl/mail` when `MAILJET_API_KEY` and `MAILJET_FROM_EMAIL` are configured together. If either key is absent, step delivery fails closed with `email_sequence_mail_sender_unavailable`.
|
||
|
||
## Admin UI
|
||
|
||
`/admin/email-sequences` (permission `admin.marketing.leads.view`):
|
||
|
||
- **List view** — `index.tsx`. Name, trigger event, enabled badge, step count, per-status run counts
|
||
(active / completed / unsubscribed) aggregated from `EmailSequenceRun.groupBy` against `dbRead`.
|
||
- **Detail / edit** — `$id.tsx`. Edit `name`, `description`, `triggerEvent`, `stepsJson` (validated by
|
||
`parseSequenceSteps` before save), `conditionsJson`. Inline toggle for `isEnabled`.
|
||
- **Create** — `new.tsx`. Same form pre-seeded with a single-step template; new sequences are created
|
||
with `isEnabled=false` so the operator can review the JSON before turning the trigger on.
|
||
- **Stats** — `$id/stats.tsx`. Per-step `scheduled` / `sent` / `opened` / `clicked` / `bounced` / `skipped`
|
||
counters with computed open-rate and click-rate (denominator = sent count, "—" when sent=0).
|
||
Sequence-level enrollment + completion + unsubscribe + failure totals.
|
||
|
||
No drag-drop step builder — operators paste step + condition JSON directly into the form.
|
||
|
||
## CLI
|
||
|
||
`toolctrl-cli sequences <subcommand>` exposes operator commands:
|
||
|
||
| Subcommand | Purpose | `--json` |
|
||
| -------------------------------------- | -------------------------------------------------------------- | -------- |
|
||
| `list` | Print every sequence (id, name, trigger, on/off, step count) | yes |
|
||
| `enable <id>` | Flip `isEnabled` to true | yes |
|
||
| `disable <id>` | Flip `isEnabled` to false | yes |
|
||
| `start-run <id> --leadId / --tenantId` | Manually enrol a target (skips outbox trigger) | yes |
|
||
| `stats` | Per-sequence active / completed / unsubscribed / failed counts | yes |
|
||
|
||
All commands respect `MARKETING_SEQUENCES_ENABLED=false` indirectly where the worker contracts are wired — when false, `worker-mail` does not subscribe to the sequence trigger/scheduler queues, so outbox or BullMQ work can accumulate and replay after re-enable. The CLI itself never refuses to inspect DB state. In production the recommended value remains `false` until the remaining production follow-ups land and the Mailjet env pair is configured; otherwise runs, delivery rows, and delayed jobs can be created without an operator-approved delivery path.
|
||
|
||
## Cron triggers
|
||
|
||
Two unit-tested cron runner functions exist for producing outbox events that flow into the sequence trigger handler. Production daily scheduler registration is still follow-up unless the deployment runbook for that environment proves it.
|
||
|
||
- `apps/worker/src/jobs/lead-stale-trigger.ts` — finds `Lead` rows with
|
||
`firstSeenAt < now - 60d AND status IN (new, contacted)` and emits `lead.stale_60d`.
|
||
- `apps/worker/src/jobs/trial-expiring-trigger.ts` — finds tenant subscriptions whose trial ends within
|
||
3 days and emits `tenant.trial_expiring` with `daysRemaining` in the payload.
|
||
|
||
Both use date-keyed dispatch keys (`lead.stale_60d.<id>.<YYYY-MM-DD>` /
|
||
`tenant.trial_expiring.<id>.<YYYY-MM-DD>`) so re-runs on the same day are idempotent. The next code blocker for production scheduler registration is the trial source contract: `runTrialExpiringTriggerCron` needs a canonical `trialEndsAt`, but the current subscription schema does not expose one directly. Treat deploy-specific cron evidence, alerting, and production schedule verification as remaining operator/runtime follow-up work unless the deployment runbook for that environment proves otherwise.
|
||
|
||
## Env
|
||
|
||
| Var | Default | Effect |
|
||
| ----------------------------- | ---------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
|
||
| `MARKETING_TOKEN_SECRET` | (falls back to `LEAD_CONVERSION_SECRET`) | HMAC key for unsubscribe tokens |
|
||
| `MARKETING_SEQUENCES_ENABLED` | unset / `false` in production examples | Opt-in switch — trigger, scheduler, and sequence cron producers only do work when this is exactly `true`. Keep this unset or `false` in production until the remaining production follow-ups are approved. |
|
||
| `MAILJET_API_KEY` | unset | Required with `MAILJET_FROM_EMAIL` for sequence step delivery through `@toolctrl/mail`; missing configuration fails closed. |
|
||
| `MAILJET_FROM_EMAIL` | unset | Required with `MAILJET_API_KEY` for sequence step delivery through `@toolctrl/mail`; missing configuration fails closed. |
|
||
| `MAILJET_FROM_NAME` | unset | Optional display name passed to `@toolctrl/mail`. |
|
||
| `MAILJET_WEBHOOK_SECRET` | unset | Required on the API runtime for signed sequence bounce webhook ingestion at `/v1/email-sequences/mailjet/webhooks`. |
|
||
|
||
## Troubleshooting
|
||
|
||
- Unsubscribe link 404s → check `MARKETING_TOKEN_SECRET` matches between mail send time and verify time.
|
||
- Sequence does not start → confirm outbox dispatcher is running + `EmailSequence.isEnabled = true` + `triggerEvent` matches the (normalised) outbox event-type.
|
||
- Steps stuck pending → check the scheduler worker logs, the `email-sequence.step` queue depth, Redis connectivity, and idempotent job ids (`email-sequence-step:<runId>:<stepIndex>`).
|
||
- Open-rate looks impossible (>100%) → Apple Mail Privacy Protection pre-fetch; combine with click-rate for a more reliable engagement signal.
|
||
- Hard bounce → run transitions to `failed` and no further steps fire. The Lead row's `unsubscribedAt` is NOT touched (Phase 1 lead-model owns that semantic; bounce is per-run only).
|
||
- Sequence isn't sending — use this quick diagnostic checklist:
|
||
1. `toolctrl-cli sequences list --json` → confirm `isEnabled=true` on the matching row.
|
||
2. `triggerEvent` matches the normalised outbox event-type. Outbox emits
|
||
`marketing.lead.captured` but sequences store `lead.captured` — the trigger handler
|
||
normalises both directions. Manually configured sequences with a `marketing.`-prefixed
|
||
trigger will never match.
|
||
3. The permanent `worker-mail` service is running with `WORKER_QUEUE_GROUP=mail`; the bootstrap helper short-circuits on other groups.
|
||
4. `MAILJET_API_KEY` and `MAILJET_FROM_EMAIL` are both configured on `worker-mail`. If either is missing, step delivery fails closed with `email_sequence_mail_sender_unavailable`.
|
||
5. `outbox.marketing` BullMQ queue depth > 0 means the dispatcher has already
|
||
handed sequence trigger work to Redis; check `worker-mail`,
|
||
`MARKETING_SEQUENCES_ENABLED`, and sequence subscriber errors next.
|
||
Pending `marketing.*` rows in the database outbox instead mean the
|
||
dispatcher still needs to be running on the `general` group.
|
||
- Trial-expiring cron emits nothing — verify the subscription store actually has rows with a
|
||
`trialEndsAt` inside the horizon (default 3 days). Bare Stripe-managed trials may not have a
|
||
cached `trialEndsAt` column; the cron skips them silently.
|
||
- Stale-lead cron over-eager — bump `MARKETING_STALE_LEAD_DAYS` (or pass a non-default
|
||
`thresholdDays` to the cron when wiring) and re-run.
|
||
|
||
## Status
|
||
|
||
Detail spec: `docs/improvement-plan/05-business-features/marketing-funnel-phase-3.md`
|
||
|
||
Open follow-ups:
|
||
|
||
- Promote the four local readiness sequence definitions into deployable demo/production seed data, with environment-specific owner approval.
|
||
- Add a statistics aggregation cron and have the dashboard read persisted aggregates instead of live groupings.
|
||
- Add provider delivery/open/click webhook reconciliation beyond the in-repo open/click tracking routes if operators choose Mailjet-native engagement events.
|
||
- Add Playwright E2E coverage for sequence create/edit, enrollment visibility, tracking, and unsubscribe flows.
|
||
- Wire/verify stale-lead/trial-expiring scheduled-job registration after choosing the canonical trial-end data source, then add deployment evidence, alerting, and runbook verification.
|