# Authentik Self-Hosted — Ops Runbook This runbook covers the Phase 09 #81 Authentik IdP stack: deploy on staging, post-deploy verification, hardening, backup + restore, and migration to a dedicated prod host. The artifacts live under `infra/authentik/`; this document covers the operator side. > **Status:** artifact-complete (this commit). Staging-deployment is operator > owned — Authentik is **NOT yet running** on staging. --- ## 1. Architecture overview Four containers, all on the `authentik-net` internal Docker network. Only the `server` container is additionally attached to the Coolify-managed `coolify` external network so Traefik can route inbound HTTPS: ``` Traefik (coolify-proxy) | auth.boiler.saas-ctrl.com | +-------------------+--------------------+ | | | authentik-net | coolify-net | | v | +-------+ +-------+ +--------+ +----+-----+ | server|<-->|worker | |postgres| | redis | | 9000 | | | | 5432 | | 6379 | +---+---+ +---+---+ +---+----+ +----+-----+ ^ ^ ^ ^ | | | | +------- authentik-net (internal) -------+ ``` - **server** — Django web + REST API. Mounts the project blueprints from `infra/authentik/bootstrap/blueprints/` at `/blueprints/toolctrl/`. - **worker** — Background jobs, blueprint application, email send. Shares the same image. - **postgresql** — Authentik's own Postgres 16. **Not shared** with the boilerplate app DB. - **redis** — Authentik's own Redis 7. **Not shared** with the boilerplate app Redis. Volumes (named, pinned in `docker-compose.yml`): - `authentik-pg-data` (Postgres data) - `authentik-redis-data` (Redis RDB) - `authentik-media` (uploaded avatars + branding assets) - `authentik-certs` (Authentik-generated signing certs) - `authentik-templates` (custom email + UI templates) --- ## 2. DNS setup Authentik runs on the subdomain `auth.boiler.saas-ctrl.com` pointing at the staging IP `46.225.5.93`. Create the A record via the Cloudflare API: ```bash # Cloudflare creds at ~/.cli-tokens/cloudflare-token # line 1 = Global API Key # line 2 = account email CF_TOKEN="$(sed -n '1p' ~/.cli-tokens/cloudflare-token)" CF_EMAIL="$(sed -n '2p' ~/.cli-tokens/cloudflare-token)" ZONE_ID="$(curl -fsS \ -H "X-Auth-Email: ${CF_EMAIL}" \ -H "X-Auth-Key: ${CF_TOKEN}" \ "https://api.cloudflare.com/client/v4/zones?name=saas-ctrl.com" \ | jq -r '.result[0].id')" curl -fsS -X POST \ -H "X-Auth-Email: ${CF_EMAIL}" \ -H "X-Auth-Key: ${CF_TOKEN}" \ -H "Content-Type: application/json" \ --data '{ "type": "A", "name": "auth.boiler", "content": "46.225.5.93", "ttl": 300, "proxied": false }' \ "https://api.cloudflare.com/client/v4/zones/${ZONE_ID}/dns_records" \ | jq . # Wait ~30s for propagation, then verify: dig +short auth.boiler.saas-ctrl.com # Expected: 46.225.5.93 ``` `proxied: false` keeps it grey-clouded — Traefik handles TLS via Let's Encrypt HTTP-01. If you want Cloudflare-proxied + origin-cert, flip `proxied: true` AND switch Traefik to a TLS-passthrough resolver (out of scope for this task; same pattern as Sentry). --- ## 3. Coolify deploy 1. In the Coolify dashboard, open the `boilerplate` project's `validation` environment and click **+ New Resource** -> **Docker Compose**. 2. Source: this repository, branch `main`, build-context `infra/authentik/`. Compose path: `docker-compose.yml`. 3. **Domain:** add `auth.boiler.saas-ctrl.com` on the `server` service. Coolify auto-attaches the standard Traefik labels (router rule, https entrypoint, `letsencrypt` cert resolver). Verify they match `infra/authentik/traefik.labels.yml`. 4. **Environment variables** — copy from `infra/authentik/.env.example` into Coolify-Vault, filling secrets: ```bash AUTHENTIK_SECRET_KEY=$(openssl rand -base64 60) AUTHENTIK_POSTGRESQL__PASSWORD=$(openssl rand -base64 32) AUTHENTIK_BOOTSTRAP_PASSWORD=$(openssl rand -base64 24) AUTHENTIK_BOOTSTRAP_TOKEN=$(openssl rand -hex 32) TOOLCTRL_ADMIN_OIDC_CLIENT_SECRET=$(openssl rand -hex 32) AUTHENTIK_BOOTSTRAP_EMAIL=ops@toolctrl.de # operator's monitored mailbox AUTHENTIK_SENTRY__DSN=$(grep AUTHENTIK_DSN ~/.cli-tokens/boilerplate-staging-env | cut -d= -f2) ``` Save the four randomly-generated secrets to your offline password vault (1Password / Bitwarden / Yubikey) **as well as** Coolify-Vault. Losing `AUTHENTIK_SECRET_KEY` = full re-onboarding of every admin user. 5. Click **Deploy**. First boot: - postgresql + redis start, server waits for `service_healthy` - server runs migrations (~30-60s) - worker boots, discovers `/blueprints/toolctrl/*.yaml`, applies them - bootstrap admin is created from `AUTHENTIK_BOOTSTRAP_EMAIL` + `AUTHENTIK_BOOTSTRAP_PASSWORD` 6. Once `docker ps` shows the server container `(healthy)`, run the bootstrap wrapper to force-apply blueprints + print operator next steps: ```bash ssh boilerplate-staging cd /var/lib/coolify/services/authentik # or wherever Coolify mounted it AUTHENTIK_BASE_URL=https://auth.boiler.saas-ctrl.com \ AUTHENTIK_BOOTSTRAP_TOKEN= \ bash bootstrap/bootstrap.sh ``` 7. Smoke-test from your laptop: ```bash AUTHENTIK_BASE_URL=https://auth.boiler.saas-ctrl.com \ AUTHENTIK_APP_SLUG=toolctrl-admin-panel \ bash infra/authentik/scripts/smoke.sh ``` All five probes must pass (root 200, health-live 200, health-ready 200, OIDC discovery 200 with the four required fields, self-enrollment 404). --- ## 4. Bootstrap admin first login + password rotation 1. Browse to and log in with `AUTHENTIK_BOOTSTRAP_EMAIL` / `AUTHENTIK_BOOTSTRAP_PASSWORD`. 2. Authentik forces a password change on first login. Set a new strong password (meets the `toolctrl-password-policy`: 12+ chars, mixed case + digit + symbol, no HIBP hit, zxcvbn >= 3). 3. **Enroll TOTP**: User-Settings -> Authenticators -> Set up TOTP. Scan the QR with the operator's hardware-MFA app (Yubikey-Authenticator / 1Password / Authy). Required because the `toolctrl-require-mfa-totp-for-admin` policy denies access to the admin-panel application without it. 4. **Rotate the bootstrap token**: Directory -> Tokens -> find the bootstrap token (look up by intent: `app_password` and the bootstrap-email user), delete it. Create a personal admin token instead and store it in Coolify-Vault under `AUTHENTIK_ADMIN_TOKEN` for future re-bootstrap runs. 5. **Add the bootstrap admin to the `toolctrl-admin` group** (it's NOT in the group by default — Authentik treats it as a built-in superuser; the group binding is what the admin-panel OIDC app uses for authorisation). --- ## 5. Hardening checklist - [ ] `AUTHENTIK_ERROR_REPORTING__ENABLED=false` set in Coolify-Vault - [ ] `AUTHENTIK_DISABLE_UPDATE_CHECK=true` - [ ] `AUTHENTIK_DISABLE_STARTUP_ANALYTICS=true` - [ ] Internal SSL OFF (TLS terminated by Traefik) — verify by `curl -ksI https://auth.boiler.saas-ctrl.com/` shows `server: Traefik` not `server: gunicorn`. - [ ] MFA-TOTP policy applied to `toolctrl-admin` group — verified by creating a test admin without TOTP and seeing the policy deny. - [ ] Public self-registration off (`/if/flow/default-source-enrollment/` returns 404 — covered by `scripts/smoke.sh`). - [ ] Brute-force lockout policy applied to the default-authentication-flow — verified by triggering 6 failed logins in 1 minute and seeing the 7th request hit a reputation-deny. - [ ] Traefik secure-headers middleware attached to the router (HSTS 2yr, frame-deny, no-referrer, X-Robots-Tag noindex). Verify by `curl -ksI https://auth.boiler.saas-ctrl.com/ | grep -i strict-transport`. - [ ] Traefik rate-limit (60/min burst 120) attached on top of Authentik's own brute-force policy. - [ ] Bootstrap token rotated AND removed from Coolify-Vault (replaced by personal admin token). - [ ] `AUTHENTIK_SECRET_KEY` backed up to offline password-vault. - [ ] Backup container deployed + first nightly run verified (`aws s3 ls s3://${AUTHENTIK_S3_BACKUP_BUCKET}/authentik/$(date +%Y-%m-%d)/`). - [ ] Brand whitelisting: Authentik UI -> System -> Brands -> default brand has `default flow background` set to a Toolctrl asset (or default OK). - [ ] OIDC application access bindings reviewed: only `toolctrl-admin` and `toolctrl-staff` allowed; no `everyone` binding. --- ## 6. Backup restore drill The backup container (`infra/authentik/backup/`) writes three encrypted artifacts daily to `s3:///authentik/YYYY-MM-DD/`: - `authentik-postgres-*.dump.gpg` — pg_dump custom format - `authentik-media-*.tar.gz.gpg` — /media volume - `SECRETS-MANIFEST-*.txt.gpg` — list of secrets NOT in the dump To verify restore on a throwaway test host (DO THIS QUARTERLY): ```bash # 1. Fetch latest dump aws s3 cp --endpoint-url "$AUTHENTIK_S3_BACKUP_ENDPOINT" \ "s3://$AUTHENTIK_S3_BACKUP_BUCKET/authentik/$(date -u +%Y-%m-%d)/" \ ./restore/ --recursive # 2. Decrypt (symmetric fallback — use the same AUTHENTIK_SECRET_KEY) for f in ./restore/*.gpg; do gpg --batch --yes --decrypt --passphrase "$AUTHENTIK_SECRET_KEY" \ --output "${f%.gpg}" "$f" done # 3. Spin up a throwaway Postgres + restore docker run -d --name pg-restore -e POSTGRES_PASSWORD=tmp -p 5440:5432 postgres:16-alpine docker exec -i pg-restore psql -U postgres -c "CREATE DATABASE authentik;" docker exec -i pg-restore pg_restore --no-owner --no-privileges --dbname authentik \ < ./restore/authentik-postgres-*.dump # 4. Verify schema + data docker exec -i pg-restore psql -U postgres -d authentik -c "\dt" | head docker exec -i pg-restore psql -U postgres -d authentik -c "SELECT COUNT(*) FROM authentik_core_user;" docker exec -i pg-restore psql -U postgres -d authentik -c "SELECT name FROM authentik_core_group WHERE name LIKE 'toolctrl-%';" # 5. Spin up Authentik server against the restored DB with the SAME secret # key, hit /-/health/ready/ to confirm it boots cleanly. # 6. Teardown docker rm -f pg-restore rm -rf ./restore ``` If the restore boots cleanly and the smoke probes pass, the drill is green. Document the drill date in the next operator-meeting log. --- ## 7. Update procedure To bump the Authentik tag (e.g. 2026.2.3 -> 2026.2.4): 1. Read upstream release notes: 2. Update the tag in two places: - `infra/authentik/UPSTREAM.md` — pinned version table - `infra/authentik/docker-compose.yml` — both `server` and `worker` image refs 3. Take a manual pre-update backup: ```bash ssh boilerplate-staging \ "docker exec authentik-backup-1 /srv/backup/backup.sh" ``` 4. Commit the version bump, push to `main`, let Coolify auto-redeploy. 5. Volumes are preserved; the new `server` runs schema migrations on boot. 6. Smoke-test (`scripts/smoke.sh`). 7. If anything is wrong: rollback via `git revert` of the bump commit. The pre-update backup is the safety net for data corruption. For a major-line jump (e.g. `2026.2.x -> 2026.5.x`): - READ the breaking-changes section of the upstream release notes carefully. - Check whether the blueprint schema changed (fields renamed / removed) — if yes, update `bootstrap/blueprints/*.yaml` IN THE SAME COMMIT as the version bump so the worker doesn't reject blueprints on boot. - Run the restore drill BEFORE the bump on a copy of yesterday's backup using the new image to confirm migrations apply cleanly. --- ## 8. Migration to a dedicated prod-Authentik host Staging shares the host with the boilerplate app. Production should have a dedicated host (smaller blast radius, separate maintenance window): 1. Provision a new Hetzner CCX13 (4 vCPU / 16 GB / 80 GB) in `nbg1` with Coolify pre-installed (or a plain Docker Engine + Traefik if you prefer). 2. DNS: `auth.` -> new host IP (same Cloudflare procedure). 3. Copy `infra/authentik/` to the new host (via Coolify git-pull or scp). 4. Operator-Vault: generate FRESH secrets for prod (`AUTHENTIK_SECRET_KEY` etc) — DO NOT reuse staging keys. 5. Deploy + bootstrap as in Section 3. 6. Smoke + restore drill. 7. Once green, switch the admin-panel `AUTHENTIK_OIDC_ISSUER_URL` (#82) over to the prod host with a maintenance window: - Create the prod-equivalent OIDC application in the prod Authentik - Onboard real admins (create users + assign to `toolctrl-admin`, force TOTP enrollment) - Flip `AUTHENTIK_OIDC_ISSUER_URL` in the app's Coolify env - Verify admin login end-to-end 8. Keep staging Authentik running but stop accepting new prod traffic; archive after 30 days if no rollback was needed. --- ## 9. References - Upstream docs: - Blueprints reference: - This repo: - `infra/authentik/README.md` — overview - `infra/authentik/UPSTREAM.md` — version pin rationale - `infra/authentik/docker-compose.yml` — runtime - `infra/authentik/traefik.labels.yml` — ingress - `infra/authentik/bootstrap/blueprints/` — config-as-code - `infra/authentik/backup/` — daily backup container - `infra/authentik/scripts/smoke.sh` — post-deploy verifier - Phase 09 detail-MD: `docs/improvement-plan/09-authentik-admin-sso/authentik-setup.md` - Lite alternative (LDAP-only): `docs/operations/authentik-alternatives.md`