258 lines
13 KiB
Markdown
258 lines
13 KiB
Markdown
# Asset CDN Operations Runbook
|
|
|
|
When A7/B2/#51 is reactivated, front the boilerplate's static assets
|
|
(`/assets/*`, `/fonts/*`, `/images/*`) with a CDN so the origin no longer pays
|
|
the bandwidth + TLS handshake cost for every visitor and so users far from the
|
|
Frankfurt DC see sub-second first paints.
|
|
|
|
**Default provider:** Bunny.net (EU-based, cheap, simple).
|
|
**Documented alternative:** Cloudflare.
|
|
|
|
Related task: [`docs/improvement-plan/03-scaling-infrastructure/asset-cdn.md`](../improvement-plan/03-scaling-infrastructure/asset-cdn.md).
|
|
|
|
**Rollout status:** CDN provisioning and `ASSET_CDN_URL` flips remain parked
|
|
with A7/B2/#51. The deploy workflow already includes a no-op-safe Bunny purge
|
|
step for future activation; keep this runbook as the reactivation guide until
|
|
`docs/improvement-plan/OPERATOR-CHECKLIST.md` and
|
|
`docs/improvement-plan/PROGRESS.md` move #51 out of backlog.
|
|
|
|
## Architecture
|
|
|
|
```
|
|
Browser
|
|
│
|
|
├─→ cdn.{domain}/assets/app-3f4a1b.js ──→ Bunny edge (cache hit: ~1 ms)
|
|
│ │
|
|
│ └─ origin pull on miss:
|
|
│ app.{domain}/assets/app-3f4a1b.js
|
|
│
|
|
└─→ app.{domain}/ (HTML, API, /healthz) ──→ origin server (no CDN detour)
|
|
```
|
|
|
|
Only static directories travel through the CDN. HTML, API responses, and
|
|
`/healthz`/`/readyz` go direct to origin so authenticated content and
|
|
liveness probes are never edge-cached.
|
|
|
|
## Cache strategy
|
|
|
|
| Path pattern | `Cache-Control` | Why |
|
|
| ----------------------------- | ------------------------------------------ | --------------------------------------------------------------------- |
|
|
| `/assets/*` (Vite-hashed) | `public, max-age=31536000, immutable` | Filenames embed a content hash — safe to cache forever. |
|
|
| `/fonts/*`, `/images/*` | `public, max-age=86400` | Static brand assets without hash; 1-day TTL is the longest safe value.|
|
|
| HTML documents (loader/SSR) | `no-store` (unless loader overrides) | May contain CSRF tokens, session-derived markup, or feature flags. |
|
|
| `/api/*`, `/healthz`, `/readyz` | bypass cache (edge rule) | Authenticated and/or liveness-critical — caching corrupts behaviour. |
|
|
|
|
The headers above are set by `server/server.ts` (`getStaticHeaders`) for the
|
|
static directories and by `apps/web/app/entry.server.tsx` for the SSR HTML
|
|
response. Loaders may set their own `Cache-Control` to opt out of `no-store`
|
|
(e.g. marketing landing pages choosing `public, max-age=300`).
|
|
|
|
## Provider comparison
|
|
|
|
| Concern | Bunny.net (default) | Cloudflare (alternative) |
|
|
| --------------------- | --------------------------------------- | ------------------------------------------ |
|
|
| HQ | Slovenia (EU) | United States |
|
|
| Pricing baseline | ~€1-5/month for small SaaS | Free tier sufficient for small sites |
|
|
| DSGVO posture | EU jurisdiction, simple DPA | Requires Schrems-II analysis for EU users |
|
|
| DNS migration | Not required — single CNAME suffices | Free tier requires full DNS migration |
|
|
| Image optimisation | Bunny Optimizer add-on (~$9.50/mo) | Cloudflare Images ($5/mo + per-image fee) |
|
|
| Cache purge API | Simple POST with `AccessKey` header | Token-scoped API; per-zone purge endpoints |
|
|
| Origin-shield option | Yes (single optional PoP in front of origin) | Yes (Argo Tunnel / Origin Rules) |
|
|
|
|
**Choose Bunny.net unless** the operator already runs all DNS through
|
|
Cloudflare and wants a single console — then Cloudflare is the lower-friction
|
|
choice.
|
|
|
|
## Bunny.net setup (default, when A7/B2/#51 is reactivated)
|
|
|
|
Do not execute these steps while A7/B2/#51 remains parked. They are the
|
|
reactivation checklist for the future CDN rollout. None of this is automated by
|
|
the deploy pipeline.
|
|
|
|
1. **Create the Bunny account.**
|
|
Sign up at <https://bunny.net>, add a payment method, generate an
|
|
*account-level API key* in *Account → API*. Store it as the `BUNNY_API_KEY`
|
|
secret in the Gitea / Coolify environment.
|
|
|
|
2. **Create a pull zone per environment.**
|
|
- Staging pull zone:
|
|
- Name: `cdn-staging-{project}`
|
|
- Origin URL: `https://staging.{domain}` (the app origin)
|
|
- Pricing tier: *Standard Network* (high volume HDD/Volume can be selected later)
|
|
- Production pull zone:
|
|
- Name: `cdn-{project}`
|
|
- Origin URL: `https://app.{domain}`
|
|
Note the numeric **pull-zone ID** from the URL — store it as
|
|
`BUNNY_PULL_ZONE_ID` (one per environment).
|
|
|
|
3. **Add a custom hostname.**
|
|
In the pull zone, *Hostnames → Add Custom Hostname*:
|
|
- Staging: `cdn-staging.{domain}`
|
|
- Production: `cdn.{domain}`
|
|
Create the matching DNS records at your DNS provider:
|
|
```
|
|
cdn-staging.{domain} CNAME cdn-staging-{project}.b-cdn.net
|
|
cdn.{domain} CNAME cdn-{project}.b-cdn.net
|
|
```
|
|
Wait until the records resolve (`dig cdn-staging.{domain}`), then enable
|
|
*Force SSL* — Bunny provisions a Let's Encrypt certificate automatically
|
|
(~5 minutes).
|
|
|
|
4. **Configure edge rules.**
|
|
- **Bypass cache for `/api/*`** (matching: URL path starts with `/api/`).
|
|
- **Bypass cache for `/healthz`, `/readyz`** (URL exact match).
|
|
- **Force cache for `/assets/*`** (URL path starts with `/assets/`,
|
|
`Override Cache Time = 1 year`) — defensive, since the origin already
|
|
emits `immutable`, but protects against future regressions.
|
|
|
|
5. **Enable origin shield (recommended).**
|
|
*Configuration → Origin Shield → Enabled* with the closest PoP to the
|
|
origin DC (Frankfurt for Hetzner FRA). Reduces origin load when the edge
|
|
cold-misses.
|
|
|
|
6. **Set the runtime env vars in Coolify.**
|
|
|
|
| Env var | Where to set | Example |
|
|
| --------------------- | ------------------------------------- | ------------------------------------ |
|
|
| `ASSET_CDN_URL` | Web app (build-time + runtime) | `https://cdn-staging.{domain}` |
|
|
| `BUNNY_PULL_ZONE_ID` | Gitea secret + Coolify env (deploy) | `<pull-zone-id>` |
|
|
| `BUNNY_API_KEY` | Gitea secret + Coolify env (deploy) | `<bunny-api-key>` |
|
|
|
|
`ASSET_CDN_URL` must be present at **build time** — Vite bakes the CDN
|
|
origin into asset URLs via `apps/web/vite.config.ts` (`resolveAssetBase`).
|
|
|
|
7. **Trigger a deploy** and run the smoke tests below.
|
|
|
|
## Smoke tests (when A7/B2/#51 is reactivated)
|
|
|
|
Do not run these as active acceptance checks while CDN rollout remains parked.
|
|
|
|
```bash
|
|
# 1. Asset is served via Bunny
|
|
curl -sI https://cdn-staging.{domain}/assets/<hash>.js | head -10
|
|
# Expect: HTTP/2 200
|
|
# server: BunnyCDN-DE1 (or whichever PoP)
|
|
# cache-control: public, max-age=31536000, immutable
|
|
# x-cache: HIT (after the first request warms it)
|
|
|
|
# 2. HTML is NOT cached by Bunny
|
|
curl -sI https://staging.{domain}/ | head -10
|
|
# Expect: cache-control: no-store
|
|
# no `server: BunnyCDN` header
|
|
|
|
# 3. /fonts and /images get the 1-day TTL
|
|
curl -sI https://cdn-staging.{domain}/fonts/inter.woff2 | grep cache-control
|
|
# Expect: cache-control: public, max-age=86400
|
|
|
|
# 4. Purge API is reachable
|
|
curl -sX POST "https://api.bunny.net/pullzone/${BUNNY_PULL_ZONE_ID}/purgeCache" \
|
|
-H "AccessKey: ${BUNNY_API_KEY}" -o /dev/null -w "%{http_code}\n"
|
|
# Expect: 204
|
|
```
|
|
|
|
## Cache-purge script
|
|
|
|
`scripts/cdn/bunny-purge.ts` wraps the Bunny purge API and is wired into
|
|
the `Deploy` workflow as the **Purge asset CDN** step (post-deploy). It
|
|
runs unconditionally — if `BUNNY_API_KEY` or `BUNNY_PULL_ZONE_ID` is unset
|
|
it logs and exits 0, so the step is safe before the operator finishes
|
|
the setup above.
|
|
|
|
```bash
|
|
# Manual whole-zone purge
|
|
BUNNY_PULL_ZONE_ID=<pull-zone-id> BUNNY_API_KEY=<bunny-api-key> \
|
|
pnpm exec tsx scripts/cdn/bunny-purge.ts
|
|
|
|
# Targeted purge (single asset)
|
|
BUNNY_PULL_ZONE_ID=<pull-zone-id> BUNNY_API_KEY=<bunny-api-key> \
|
|
ASSET_CDN_URL=https://cdn.{domain} \
|
|
pnpm exec tsx scripts/cdn/bunny-purge.ts --paths /assets/app-3f4a1b.js
|
|
|
|
# Multiple paths (comma-separated or repeated --paths)
|
|
pnpm exec tsx scripts/cdn/bunny-purge.ts --paths /assets/a.js,/assets/b.js
|
|
|
|
# Dry-run (prints the plan, makes no API call)
|
|
pnpm exec tsx scripts/cdn/bunny-purge.ts --dry-run
|
|
```
|
|
|
|
Exit codes:
|
|
|
|
| Code | Meaning |
|
|
| ---- | -------------------------------------------------------------------- |
|
|
| `0` | Purge succeeded, or skipped because credentials were intentionally absent. |
|
|
| `1` | Bunny returned a non-2xx response, or invalid CLI arguments. |
|
|
|
|
## Cloudflare setup (alternative)
|
|
|
|
Only follow this section if you decided against Bunny.net in the table above.
|
|
|
|
1. Add the apex/subdomain to Cloudflare (free plan suffices for small sites).
|
|
Either migrate DNS fully or proxy only the `cdn.{domain}` CNAME.
|
|
2. Create a *Page Rule* for `cdn.{domain}/assets/*` with:
|
|
*Cache Level → Cache Everything*, *Edge Cache TTL → 1 month*.
|
|
3. Create a *Page Rule* for `cdn.{domain}/api/*` with:
|
|
*Cache Level → Bypass*.
|
|
4. Generate an API token (`Zone → Cache Purge`) — store as `CF_API_TOKEN`.
|
|
You'll need a sibling purge script (out of scope for this runbook;
|
|
the Bunny script is the only one wired into the workflow today).
|
|
5. Set `ASSET_CDN_URL=https://cdn.{domain}` in Coolify.
|
|
|
|
Cloudflare's default cache TTLs and "Smart Routing" can override the
|
|
`Cache-Control` headers the origin sends. Pin explicit *Cache Rules* on
|
|
the zone if behaviour diverges from the table above.
|
|
|
|
## Image optimisation (opt-in, Phase 2)
|
|
|
|
We deliberately do NOT enable image optimisation on day one — it adds cost
|
|
and operational surface area without a measured benefit on a small footprint.
|
|
|
|
**Enable Bunny Optimizer when any of these trigger:**
|
|
|
|
- Outbound image bandwidth from the CDN exceeds ~10 GB/month, OR
|
|
- User uploads (avatars, tenant logos, post images) cross ~1 000/month, OR
|
|
- Lighthouse's *Largest Contentful Paint* sits above 2.5 s and the audit
|
|
attributes the largest portion to images.
|
|
|
|
Steps to enable:
|
|
|
|
1. In the Bunny pull zone, *Optimizer → Enable* (~$9.50/month).
|
|
2. Tag image-heavy URLs with the size hint, e.g.
|
|
`https://cdn.{domain}/uploads/avatar.png?width=200&format=webp`.
|
|
3. Adjust the web app's image components to emit the `?width=&format=`
|
|
query string and a `<picture>` block with WebP/AVIF fallbacks.
|
|
4. Re-run Lighthouse against the staging URL and confirm LCP improvement.
|
|
|
|
## Rollback
|
|
|
|
| Failure mode | Action |
|
|
| ----------------------------------------------------- | ---------------------------------------------------------------------------- |
|
|
| CDN edge serves stale/corrupt assets | Run `pnpm exec tsx scripts/cdn/bunny-purge.ts` (whole-zone purge). |
|
|
| CDN provider outage / TLS failure | Unset `ASSET_CDN_URL` in Coolify and redeploy — Vite emits origin-relative paths next build. |
|
|
| Misconfigured `Cache-Control` shipped (e.g. `immutable` on `/fonts/`) | Push a fix, deploy, then whole-zone purge. |
|
|
| Need to fully cut over to the alternative provider | Provision the new pull zone, switch `ASSET_CDN_URL` + `BUNNY_*` secrets, redeploy. Old `cdn.{domain}` CNAME can stay until the new one stabilises. |
|
|
|
|
No dedicated feature flag is needed — `ASSET_CDN_URL` *is* the flag. Empty
|
|
or unset → no CDN detour, anywhere.
|
|
|
|
## Security notes
|
|
|
|
- **CDN-as-MITM:** Bunny / Cloudflare see every asset request including
|
|
Referrer headers. Static brand assets are fine; never route
|
|
`/uploads/<user-content>` through a public-cache zone — those paths
|
|
inherit the `no-store` default and stay on the origin.
|
|
- **CORS:** When fonts/images load via `fetch` (e.g. preload-on-demand),
|
|
the CDN must forward `Access-Control-Allow-Origin: https://app.{domain}`
|
|
from the origin. Bunny forwards CORS headers transparently; verify in
|
|
the smoke tests if you add cross-origin font loading.
|
|
- **CSP:** The Content-Security-Policy in `server/server.ts` keeps a
|
|
`base-uri 'self'; frame-ancestors 'self'` policy. When you switch on
|
|
the CDN, extend `script-src`, `style-src`, `font-src`, and `img-src`
|
|
to include the CDN hostname (separately tracked).
|
|
- **Origin shield + IP allowlist:** Bunny's Origin Shield gives a single
|
|
outgoing IP range. Optionally whitelist those IPs at the origin
|
|
firewall so attackers cannot bypass the CDN by hitting the origin
|
|
directly.
|
|
- **Cache poisoning:** Only assets that emit `public, max-age=…` go to the
|
|
edge. Loader-rendered HTML defaults to `no-store`; loaders that need
|
|
caching must opt in explicitly and set the right `Vary` headers.
|