Files
2026-07-08 16:15:37 +02:00

255 lines
13 KiB
Markdown

# Asset CDN Operations Runbook
When the asset-CDN rollout is reactivated, front HiveGrid's static assets
(`/assets/*`, `/fonts/*`, `/images/*`) with a CDN so the origin no longer pays
the bandwidth + TLS handshake cost for every visitor and so users far from the
Frankfurt DC see sub-second first paints.
**Default provider:** Bunny.net (EU-based, cheap, simple).
**Documented alternative:** Cloudflare.
**Rollout status:** CDN provisioning and `ASSET_CDN_URL` flips remain parked
until a HiveGrid infra ticket explicitly reactivates them. The deploy workflow
already includes a no-op-safe Bunny purge step for future activation; keep this
runbook as the reactivation guide.
## Architecture
```
Browser
│
├─→ cdn.{domain}/assets/app-3f4a1b.js ──→ Bunny edge (cache hit: ~1 ms)
│ │
│ └─ origin pull on miss:
│ app.{domain}/assets/app-3f4a1b.js
│
└─→ app.{domain}/ (HTML, API, /healthz) ──→ origin server (no CDN detour)
```
Only static directories travel through the CDN. HTML, API responses, and
`/healthz`/`/readyz` go direct to origin so authenticated content and
liveness probes are never edge-cached.
## Cache strategy
| Path pattern | `Cache-Control` | Why |
| ----------------------------- | ------------------------------------------ | --------------------------------------------------------------------- |
| `/assets/*` (Vite-hashed) | `public, max-age=31536000, immutable` | Filenames embed a content hash — safe to cache forever. |
| `/fonts/*`, `/images/*` | `public, max-age=86400` | Static brand assets without hash; 1-day TTL is the longest safe value.|
| HTML documents (loader/SSR) | `no-store` (unless loader overrides) | May contain CSRF tokens, session-derived markup, or feature flags. |
| `/api/*`, `/healthz`, `/readyz` | bypass cache (edge rule) | Authenticated and/or liveness-critical — caching corrupts behaviour. |
The headers above are set by `server/server.ts` (`getStaticHeaders`) for the
static directories and by `apps/web/app/entry.server.tsx` for the SSR HTML
response. Loaders may set their own `Cache-Control` to opt out of `no-store`
(e.g. marketing landing pages choosing `public, max-age=300`).
## Provider comparison
| Concern | Bunny.net (default) | Cloudflare (alternative) |
| --------------------- | --------------------------------------- | ------------------------------------------ |
| HQ | Slovenia (EU) | United States |
| Pricing baseline | ~€1-5/month for small SaaS | Free tier sufficient for small sites |
| DSGVO posture | EU jurisdiction, simple DPA | Requires Schrems-II analysis for EU users |
| DNS migration | Not required — single CNAME suffices | Free tier requires full DNS migration |
| Image optimisation | Bunny Optimizer add-on (~$9.50/mo) | Cloudflare Images ($5/mo + per-image fee) |
| Cache purge API | Simple POST with `AccessKey` header | Token-scoped API; per-zone purge endpoints |
| Origin-shield option | Yes (single optional PoP in front of origin) | Yes (Argo Tunnel / Origin Rules) |
**Choose Bunny.net unless** the operator already runs all DNS through
Cloudflare and wants a single console — then Cloudflare is the lower-friction
choice.
## Bunny.net setup (default, when the rollout is reactivated)
Do not execute these steps while the rollout remains parked. They are the
reactivation checklist for the future CDN rollout. None of this is automated by
the deploy pipeline.
1. **Create the Bunny account.**
Sign up at <https://bunny.net>, add a payment method, generate an
*account-level API key* in *Account → API*. Store it as the `BUNNY_API_KEY`
secret in the Gitea / Coolify environment.
2. **Create a pull zone per environment.**
- Staging pull zone:
- Name: `cdn-staging-{project}`
- Origin URL: `https://staging.{domain}` (the app origin)
- Pricing tier: *Standard Network* (high volume HDD/Volume can be selected later)
- Production pull zone:
- Name: `cdn-{project}`
- Origin URL: `https://app.{domain}`
Note the numeric **pull-zone ID** from the URL — store it as
`BUNNY_PULL_ZONE_ID` (one per environment).
3. **Add a custom hostname.**
In the pull zone, *Hostnames → Add Custom Hostname*:
- Staging: `cdn-staging.{domain}`
- Production: `cdn.{domain}`
Create the matching DNS records at your DNS provider:
```
cdn-staging.{domain} CNAME cdn-staging-{project}.b-cdn.net
cdn.{domain} CNAME cdn-{project}.b-cdn.net
```
Wait until the records resolve (`dig cdn-staging.{domain}`), then enable
*Force SSL* — Bunny provisions a Let's Encrypt certificate automatically
(~5 minutes).
4. **Configure edge rules.**
- **Bypass cache for `/api/*`** (matching: URL path starts with `/api/`).
- **Bypass cache for `/healthz`, `/readyz`** (URL exact match).
- **Force cache for `/assets/*`** (URL path starts with `/assets/`,
`Override Cache Time = 1 year`) — defensive, since the origin already
emits `immutable`, but protects against future regressions.
5. **Enable origin shield (recommended).**
*Configuration → Origin Shield → Enabled* with the closest PoP to the
origin DC (Frankfurt for Hetzner FRA). Reduces origin load when the edge
cold-misses.
6. **Set the runtime env vars in Coolify.**
| Env var | Where to set | Example |
| --------------------- | ------------------------------------- | ------------------------------------ |
| `ASSET_CDN_URL` | Web app (build-time + runtime) | `https://cdn-staging.{domain}` |
| `BUNNY_PULL_ZONE_ID` | Gitea secret + Coolify env (deploy) | `<pull-zone-id>` |
| `BUNNY_API_KEY` | Gitea secret + Coolify env (deploy) | `<bunny-api-key>` |
`ASSET_CDN_URL` must be present at **build time** — Vite bakes the CDN
origin into asset URLs via `apps/web/vite.config.ts` (`resolveAssetBase`).
7. **Trigger a deploy** and run the smoke tests below.
## Smoke tests (when the rollout is reactivated)
Do not run these as active acceptance checks while CDN rollout remains parked.
```bash
# 1. Asset is served via Bunny
curl -sI https://cdn-staging.{domain}/assets/<hash>.js | head -10
# Expect: HTTP/2 200
# server: BunnyCDN-DE1 (or whichever PoP)
# cache-control: public, max-age=31536000, immutable
# x-cache: HIT (after the first request warms it)
# 2. HTML is NOT cached by Bunny
curl -sI https://staging.{domain}/ | head -10
# Expect: cache-control: no-store
# no `server: BunnyCDN` header
# 3. /fonts and /images get the 1-day TTL
curl -sI https://cdn-staging.{domain}/fonts/inter.woff2 | grep cache-control
# Expect: cache-control: public, max-age=86400
# 4. Purge API is reachable
curl -sX POST "https://api.bunny.net/pullzone/${BUNNY_PULL_ZONE_ID}/purgeCache" \
-H "AccessKey: ${BUNNY_API_KEY}" -o /dev/null -w "%{http_code}\n"
# Expect: 204
```
## Cache-purge script
`scripts/cdn/bunny-purge.ts` wraps the Bunny purge API and is wired into
the `Deploy` workflow as the **Purge asset CDN** step (post-deploy). It
runs unconditionally — if `BUNNY_API_KEY` or `BUNNY_PULL_ZONE_ID` is unset
it logs and exits 0, so the step is safe before the operator finishes
the setup above.
```bash
# Manual whole-zone purge
BUNNY_PULL_ZONE_ID=<pull-zone-id> BUNNY_API_KEY=<bunny-api-key> \
pnpm exec tsx scripts/cdn/bunny-purge.ts
# Targeted purge (single asset)
BUNNY_PULL_ZONE_ID=<pull-zone-id> BUNNY_API_KEY=<bunny-api-key> \
ASSET_CDN_URL=https://cdn.{domain} \
pnpm exec tsx scripts/cdn/bunny-purge.ts --paths /assets/app-3f4a1b.js
# Multiple paths (comma-separated or repeated --paths)
pnpm exec tsx scripts/cdn/bunny-purge.ts --paths /assets/a.js,/assets/b.js
# Dry-run (prints the plan, makes no API call)
pnpm exec tsx scripts/cdn/bunny-purge.ts --dry-run
```
Exit codes:
| Code | Meaning |
| ---- | -------------------------------------------------------------------- |
| `0` | Purge succeeded, or skipped because credentials were intentionally absent. |
| `1` | Bunny returned a non-2xx response, or invalid CLI arguments. |
## Cloudflare setup (alternative)
Only follow this section if you decided against Bunny.net in the table above.
1. Add the apex/subdomain to Cloudflare (free plan suffices for small sites).
Either migrate DNS fully or proxy only the `cdn.{domain}` CNAME.
2. Create a *Page Rule* for `cdn.{domain}/assets/*` with:
*Cache Level → Cache Everything*, *Edge Cache TTL → 1 month*.
3. Create a *Page Rule* for `cdn.{domain}/api/*` with:
*Cache Level → Bypass*.
4. Generate an API token (`Zone → Cache Purge`) — store as `CF_API_TOKEN`.
You'll need a sibling purge script (out of scope for this runbook;
the Bunny script is the only one wired into the workflow today).
5. Set `ASSET_CDN_URL=https://cdn.{domain}` in Coolify.
Cloudflare's default cache TTLs and "Smart Routing" can override the
`Cache-Control` headers the origin sends. Pin explicit *Cache Rules* on
the zone if behaviour diverges from the table above.
## Image optimisation (opt-in, Phase 2)
We deliberately do NOT enable image optimisation on day one — it adds cost
and operational surface area without a measured benefit on a small footprint.
**Enable Bunny Optimizer when any of these trigger:**
- Outbound image bandwidth from the CDN exceeds ~10 GB/month, OR
- User uploads (avatars, tenant logos, post images) cross ~1 000/month, OR
- Lighthouse's *Largest Contentful Paint* sits above 2.5 s and the audit
attributes the largest portion to images.
Steps to enable:
1. In the Bunny pull zone, *Optimizer → Enable* (~$9.50/month).
2. Tag image-heavy URLs with the size hint, e.g.
`https://cdn.{domain}/uploads/avatar.png?width=200&format=webp`.
3. Adjust the web app's image components to emit the `?width=&format=`
query string and a `<picture>` block with WebP/AVIF fallbacks.
4. Re-run Lighthouse against the staging URL and confirm LCP improvement.
## Rollback
| Failure mode | Action |
| ----------------------------------------------------- | ---------------------------------------------------------------------------- |
| CDN edge serves stale/corrupt assets | Run `pnpm exec tsx scripts/cdn/bunny-purge.ts` (whole-zone purge). |
| CDN provider outage / TLS failure | Unset `ASSET_CDN_URL` in Coolify and redeploy — Vite emits origin-relative paths next build. |
| Misconfigured `Cache-Control` shipped (e.g. `immutable` on `/fonts/`) | Push a fix, deploy, then whole-zone purge. |
| Need to fully cut over to the alternative provider | Provision the new pull zone, switch `ASSET_CDN_URL` + `BUNNY_*` secrets, redeploy. Old `cdn.{domain}` CNAME can stay until the new one stabilises. |
No dedicated feature flag is needed — `ASSET_CDN_URL` *is* the flag. Empty
or unset → no CDN detour, anywhere.
## Security notes
- **CDN-as-MITM:** Bunny / Cloudflare see every asset request including
Referrer headers. Static brand assets are fine; never route
`/uploads/<user-content>` through a public-cache zone — those paths
inherit the `no-store` default and stay on the origin.
- **CORS:** When fonts/images load via `fetch` (e.g. preload-on-demand),
the CDN must forward `Access-Control-Allow-Origin: https://app.{domain}`
from the origin. Bunny forwards CORS headers transparently; verify in
the smoke tests if you add cross-origin font loading.
- **CSP:** The Content-Security-Policy in `server/server.ts` keeps a
`base-uri 'self'; frame-ancestors 'self'` policy. When you switch on
the CDN, extend `script-src`, `style-src`, `font-src`, and `img-src`
to include the CDN hostname (separately tracked).
- **Origin shield + IP allowlist:** Bunny's Origin Shield gives a single
outgoing IP range. Optionally whitelist those IPs at the origin
firewall so attackers cannot bypass the CDN by hitting the origin
directly.
- **Cache poisoning:** Only assets that emit `public, max-age=…` go to the
edge. Loader-rendered HTML defaults to `no-store`; loaders that need
caching must opt in explicitly and set the right `Vary` headers.