323 lines
14 KiB
Markdown
323 lines
14 KiB
Markdown
# Authentik Self-Hosted — Ops Runbook
|
|
|
|
This runbook covers the Phase 09 #81 Authentik IdP stack: deploy on staging,
|
|
post-deploy verification, hardening, backup + restore, and migration to a
|
|
dedicated prod host. The artifacts live under `infra/authentik/`; this
|
|
document covers the operator side.
|
|
|
|
> **Status:** artifact-complete (this commit). Staging-deployment is operator
|
|
> owned — Authentik is **NOT yet running** on staging.
|
|
|
|
---
|
|
|
|
## 1. Architecture overview
|
|
|
|
Four containers, all on the `authentik-net` internal Docker network. Only the
|
|
`server` container is additionally attached to the Coolify-managed `coolify`
|
|
external network so Traefik can route inbound HTTPS:
|
|
|
|
```
|
|
Traefik (coolify-proxy)
|
|
|
|
|
auth.boiler.saas-ctrl.com
|
|
|
|
|
+-------------------+--------------------+
|
|
| |
|
|
| authentik-net | coolify-net
|
|
| |
|
|
v |
|
|
+-------+ +-------+ +--------+ +----+-----+
|
|
| server|<-->|worker | |postgres| | redis |
|
|
| 9000 | | | | 5432 | | 6379 |
|
|
+---+---+ +---+---+ +---+----+ +----+-----+
|
|
^ ^ ^ ^
|
|
| | | |
|
|
+------- authentik-net (internal) -------+
|
|
```
|
|
|
|
- **server** — Django web + REST API. Mounts the project blueprints from
|
|
`infra/authentik/bootstrap/blueprints/` at `/blueprints/toolctrl/`.
|
|
- **worker** — Background jobs, blueprint application, email send. Shares
|
|
the same image.
|
|
- **postgresql** — Authentik's own Postgres 16. **Not shared** with the
|
|
boilerplate app DB.
|
|
- **redis** — Authentik's own Redis 7. **Not shared** with the boilerplate
|
|
app Redis.
|
|
|
|
Volumes (named, pinned in `docker-compose.yml`):
|
|
|
|
- `authentik-pg-data` (Postgres data)
|
|
- `authentik-redis-data` (Redis RDB)
|
|
- `authentik-media` (uploaded avatars + branding assets)
|
|
- `authentik-certs` (Authentik-generated signing certs)
|
|
- `authentik-templates` (custom email + UI templates)
|
|
|
|
---
|
|
|
|
## 2. DNS setup
|
|
|
|
Authentik runs on the subdomain `auth.boiler.saas-ctrl.com` pointing at the
|
|
staging IP `46.225.5.93`. Create the A record via the Cloudflare API:
|
|
|
|
```bash
|
|
# Cloudflare creds at ~/.cli-tokens/cloudflare-token
|
|
# line 1 = Global API Key
|
|
# line 2 = account email
|
|
CF_TOKEN="$(sed -n '1p' ~/.cli-tokens/cloudflare-token)"
|
|
CF_EMAIL="$(sed -n '2p' ~/.cli-tokens/cloudflare-token)"
|
|
|
|
ZONE_ID="$(curl -fsS \
|
|
-H "X-Auth-Email: ${CF_EMAIL}" \
|
|
-H "X-Auth-Key: ${CF_TOKEN}" \
|
|
"https://api.cloudflare.com/client/v4/zones?name=saas-ctrl.com" \
|
|
| jq -r '.result[0].id')"
|
|
|
|
curl -fsS -X POST \
|
|
-H "X-Auth-Email: ${CF_EMAIL}" \
|
|
-H "X-Auth-Key: ${CF_TOKEN}" \
|
|
-H "Content-Type: application/json" \
|
|
--data '{
|
|
"type": "A",
|
|
"name": "auth.boiler",
|
|
"content": "46.225.5.93",
|
|
"ttl": 300,
|
|
"proxied": false
|
|
}' \
|
|
"https://api.cloudflare.com/client/v4/zones/${ZONE_ID}/dns_records" \
|
|
| jq .
|
|
|
|
# Wait ~30s for propagation, then verify:
|
|
dig +short auth.boiler.saas-ctrl.com
|
|
# Expected: 46.225.5.93
|
|
```
|
|
|
|
`proxied: false` keeps it grey-clouded — Traefik handles TLS via Let's
|
|
Encrypt HTTP-01. If you want Cloudflare-proxied + origin-cert, flip
|
|
`proxied: true` AND switch Traefik to a TLS-passthrough resolver (out of
|
|
scope for this task; same pattern as Sentry).
|
|
|
|
---
|
|
|
|
## 3. Coolify deploy
|
|
|
|
1. In the Coolify dashboard, open the `boilerplate` project's `validation`
|
|
environment and click **+ New Resource** -> **Docker Compose**.
|
|
2. Source: this repository, branch `main`, build-context `infra/authentik/`.
|
|
Compose path: `docker-compose.yml`.
|
|
3. **Domain:** add `auth.boiler.saas-ctrl.com` on the `server` service.
|
|
Coolify auto-attaches the standard Traefik labels (router rule, https
|
|
entrypoint, `letsencrypt` cert resolver). Verify they match
|
|
`infra/authentik/traefik.labels.yml`.
|
|
4. **Environment variables** — copy from `infra/authentik/.env.example` into
|
|
Coolify-Vault, filling secrets:
|
|
```bash
|
|
AUTHENTIK_SECRET_KEY=$(openssl rand -base64 60)
|
|
AUTHENTIK_POSTGRESQL__PASSWORD=$(openssl rand -base64 32)
|
|
AUTHENTIK_BOOTSTRAP_PASSWORD=$(openssl rand -base64 24)
|
|
AUTHENTIK_BOOTSTRAP_TOKEN=$(openssl rand -hex 32)
|
|
TOOLCTRL_ADMIN_OIDC_CLIENT_SECRET=$(openssl rand -hex 32)
|
|
AUTHENTIK_BOOTSTRAP_EMAIL=ops@toolctrl.de # operator's monitored mailbox
|
|
AUTHENTIK_SENTRY__DSN=$(grep AUTHENTIK_DSN ~/.cli-tokens/boilerplate-staging-env | cut -d= -f2)
|
|
```
|
|
Save the four randomly-generated secrets to your offline password vault
|
|
(1Password / Bitwarden / Yubikey) **as well as** Coolify-Vault. Losing
|
|
`AUTHENTIK_SECRET_KEY` = full re-onboarding of every admin user.
|
|
5. Click **Deploy**. First boot:
|
|
- postgresql + redis start, server waits for `service_healthy`
|
|
- server runs migrations (~30-60s)
|
|
- worker boots, discovers `/blueprints/toolctrl/*.yaml`, applies them
|
|
- bootstrap admin is created from `AUTHENTIK_BOOTSTRAP_EMAIL` +
|
|
`AUTHENTIK_BOOTSTRAP_PASSWORD`
|
|
6. Once `docker ps` shows the server container `(healthy)`, run the
|
|
bootstrap wrapper to force-apply blueprints + print operator next steps:
|
|
```bash
|
|
ssh boilerplate-staging
|
|
cd /var/lib/coolify/services/authentik # or wherever Coolify mounted it
|
|
AUTHENTIK_BASE_URL=https://auth.boiler.saas-ctrl.com \
|
|
AUTHENTIK_BOOTSTRAP_TOKEN=<from-vault> \
|
|
bash bootstrap/bootstrap.sh
|
|
```
|
|
7. Smoke-test from your laptop:
|
|
```bash
|
|
AUTHENTIK_BASE_URL=https://auth.boiler.saas-ctrl.com \
|
|
AUTHENTIK_APP_SLUG=toolctrl-admin-panel \
|
|
bash infra/authentik/scripts/smoke.sh
|
|
```
|
|
All five probes must pass (root 200, health-live 200, health-ready 200,
|
|
OIDC discovery 200 with the four required fields, self-enrollment 404).
|
|
|
|
---
|
|
|
|
## 4. Bootstrap admin first login + password rotation
|
|
|
|
1. Browse to <https://auth.boiler.saas-ctrl.com/> and log in with
|
|
`AUTHENTIK_BOOTSTRAP_EMAIL` / `AUTHENTIK_BOOTSTRAP_PASSWORD`.
|
|
2. Authentik forces a password change on first login. Set a new strong
|
|
password (meets the `toolctrl-password-policy`: 12+ chars, mixed case
|
|
+ digit + symbol, no HIBP hit, zxcvbn >= 3).
|
|
3. **Enroll TOTP**: User-Settings -> Authenticators -> Set up TOTP.
|
|
Scan the QR with the operator's hardware-MFA app (Yubikey-Authenticator
|
|
/ 1Password / Authy). Required because the
|
|
`toolctrl-require-mfa-totp-for-admin` policy denies access to the
|
|
admin-panel application without it.
|
|
4. **Rotate the bootstrap token**: Directory -> Tokens -> find the bootstrap
|
|
token (look up by intent: `app_password` and the bootstrap-email user),
|
|
delete it. Create a personal admin token instead and store it in
|
|
Coolify-Vault under `AUTHENTIK_ADMIN_TOKEN` for future re-bootstrap runs.
|
|
5. **Add the bootstrap admin to the `toolctrl-admin` group** (it's NOT in
|
|
the group by default — Authentik treats it as a built-in superuser; the
|
|
group binding is what the admin-panel OIDC app uses for authorisation).
|
|
|
|
---
|
|
|
|
## 5. Hardening checklist
|
|
|
|
- [ ] `AUTHENTIK_ERROR_REPORTING__ENABLED=false` set in Coolify-Vault
|
|
- [ ] `AUTHENTIK_DISABLE_UPDATE_CHECK=true`
|
|
- [ ] `AUTHENTIK_DISABLE_STARTUP_ANALYTICS=true`
|
|
- [ ] Internal SSL OFF (TLS terminated by Traefik) — verify by
|
|
`curl -ksI https://auth.boiler.saas-ctrl.com/` shows
|
|
`server: Traefik` not `server: gunicorn`.
|
|
- [ ] MFA-TOTP policy applied to `toolctrl-admin` group — verified by
|
|
creating a test admin without TOTP and seeing the policy deny.
|
|
- [ ] Public self-registration off (`/if/flow/default-source-enrollment/`
|
|
returns 404 — covered by `scripts/smoke.sh`).
|
|
- [ ] Brute-force lockout policy applied to the default-authentication-flow
|
|
— verified by triggering 6 failed logins in 1 minute and seeing the
|
|
7th request hit a reputation-deny.
|
|
- [ ] Traefik secure-headers middleware attached to the router
|
|
(HSTS 2yr, frame-deny, no-referrer, X-Robots-Tag noindex). Verify by
|
|
`curl -ksI https://auth.boiler.saas-ctrl.com/ | grep -i strict-transport`.
|
|
- [ ] Traefik rate-limit (60/min burst 120) attached on top of Authentik's
|
|
own brute-force policy.
|
|
- [ ] Bootstrap token rotated AND removed from Coolify-Vault (replaced by
|
|
personal admin token).
|
|
- [ ] `AUTHENTIK_SECRET_KEY` backed up to offline password-vault.
|
|
- [ ] Backup container deployed + first nightly run verified
|
|
(`aws s3 ls s3://${AUTHENTIK_S3_BACKUP_BUCKET}/authentik/$(date +%Y-%m-%d)/`).
|
|
- [ ] Brand whitelisting: Authentik UI -> System -> Brands -> default brand
|
|
has `default flow background` set to a Toolctrl asset (or default OK).
|
|
- [ ] OIDC application access bindings reviewed: only `toolctrl-admin` and
|
|
`toolctrl-staff` allowed; no `everyone` binding.
|
|
|
|
---
|
|
|
|
## 6. Backup restore drill
|
|
|
|
The backup container (`infra/authentik/backup/`) writes three encrypted
|
|
artifacts daily to `s3://<bucket>/authentik/YYYY-MM-DD/`:
|
|
|
|
- `authentik-postgres-*.dump.gpg` — pg_dump custom format
|
|
- `authentik-media-*.tar.gz.gpg` — /media volume
|
|
- `SECRETS-MANIFEST-*.txt.gpg` — list of secrets NOT in the dump
|
|
|
|
To verify restore on a throwaway test host (DO THIS QUARTERLY):
|
|
|
|
```bash
|
|
# 1. Fetch latest dump
|
|
aws s3 cp --endpoint-url "$AUTHENTIK_S3_BACKUP_ENDPOINT" \
|
|
"s3://$AUTHENTIK_S3_BACKUP_BUCKET/authentik/$(date -u +%Y-%m-%d)/" \
|
|
./restore/ --recursive
|
|
|
|
# 2. Decrypt (symmetric fallback — use the same AUTHENTIK_SECRET_KEY)
|
|
for f in ./restore/*.gpg; do
|
|
gpg --batch --yes --decrypt --passphrase "$AUTHENTIK_SECRET_KEY" \
|
|
--output "${f%.gpg}" "$f"
|
|
done
|
|
|
|
# 3. Spin up a throwaway Postgres + restore
|
|
docker run -d --name pg-restore -e POSTGRES_PASSWORD=tmp -p 5440:5432 postgres:16-alpine
|
|
docker exec -i pg-restore psql -U postgres -c "CREATE DATABASE authentik;"
|
|
docker exec -i pg-restore pg_restore --no-owner --no-privileges --dbname authentik \
|
|
< ./restore/authentik-postgres-*.dump
|
|
|
|
# 4. Verify schema + data
|
|
docker exec -i pg-restore psql -U postgres -d authentik -c "\dt" | head
|
|
docker exec -i pg-restore psql -U postgres -d authentik -c "SELECT COUNT(*) FROM authentik_core_user;"
|
|
docker exec -i pg-restore psql -U postgres -d authentik -c "SELECT name FROM authentik_core_group WHERE name LIKE 'toolctrl-%';"
|
|
|
|
# 5. Spin up Authentik server against the restored DB with the SAME secret
|
|
# key, hit /-/health/ready/ to confirm it boots cleanly.
|
|
|
|
# 6. Teardown
|
|
docker rm -f pg-restore
|
|
rm -rf ./restore
|
|
```
|
|
|
|
If the restore boots cleanly and the smoke probes pass, the drill is green.
|
|
Document the drill date in the next operator-meeting log.
|
|
|
|
---
|
|
|
|
## 7. Update procedure
|
|
|
|
To bump the Authentik tag (e.g. 2026.2.3 -> 2026.2.4):
|
|
|
|
1. Read upstream release notes: <https://github.com/goauthentik/authentik/releases>
|
|
2. Update the tag in two places:
|
|
- `infra/authentik/UPSTREAM.md` — pinned version table
|
|
- `infra/authentik/docker-compose.yml` — both `server` and `worker` image refs
|
|
3. Take a manual pre-update backup:
|
|
```bash
|
|
ssh boilerplate-staging \
|
|
"docker exec authentik-backup-1 /srv/backup/backup.sh"
|
|
```
|
|
4. Commit the version bump, push to `main`, let Coolify auto-redeploy.
|
|
5. Volumes are preserved; the new `server` runs schema migrations on boot.
|
|
6. Smoke-test (`scripts/smoke.sh`).
|
|
7. If anything is wrong: rollback via `git revert` of the bump commit. The
|
|
pre-update backup is the safety net for data corruption.
|
|
|
|
For a major-line jump (e.g. `2026.2.x -> 2026.5.x`):
|
|
|
|
- READ the breaking-changes section of the upstream release notes carefully.
|
|
- Check whether the blueprint schema changed (fields renamed / removed) — if
|
|
yes, update `bootstrap/blueprints/*.yaml` IN THE SAME COMMIT as the version
|
|
bump so the worker doesn't reject blueprints on boot.
|
|
- Run the restore drill BEFORE the bump on a copy of yesterday's backup
|
|
using the new image to confirm migrations apply cleanly.
|
|
|
|
---
|
|
|
|
## 8. Migration to a dedicated prod-Authentik host
|
|
|
|
Staging shares the host with the boilerplate app. Production should have a
|
|
dedicated host (smaller blast radius, separate maintenance window):
|
|
|
|
1. Provision a new Hetzner CCX13 (4 vCPU / 16 GB / 80 GB) in `nbg1` with
|
|
Coolify pre-installed (or a plain Docker Engine + Traefik if you prefer).
|
|
2. DNS: `auth.<prod-domain>` -> new host IP (same Cloudflare procedure).
|
|
3. Copy `infra/authentik/` to the new host (via Coolify git-pull or scp).
|
|
4. Operator-Vault: generate FRESH secrets for prod
|
|
(`AUTHENTIK_SECRET_KEY` etc) — DO NOT reuse staging keys.
|
|
5. Deploy + bootstrap as in Section 3.
|
|
6. Smoke + restore drill.
|
|
7. Once green, switch the admin-panel `AUTHENTIK_OIDC_ISSUER_URL` (#82) over
|
|
to the prod host with a maintenance window:
|
|
- Create the prod-equivalent OIDC application in the prod Authentik
|
|
- Onboard real admins (create users + assign to `toolctrl-admin`,
|
|
force TOTP enrollment)
|
|
- Flip `AUTHENTIK_OIDC_ISSUER_URL` in the app's Coolify env
|
|
- Verify admin login end-to-end
|
|
8. Keep staging Authentik running but stop accepting new prod traffic;
|
|
archive after 30 days if no rollback was needed.
|
|
|
|
---
|
|
|
|
## 9. References
|
|
|
|
- Upstream docs: <https://goauthentik.io/docs/>
|
|
- Blueprints reference: <https://goauthentik.io/docs/customize/blueprints>
|
|
- This repo:
|
|
- `infra/authentik/README.md` — overview
|
|
- `infra/authentik/UPSTREAM.md` — version pin rationale
|
|
- `infra/authentik/docker-compose.yml` — runtime
|
|
- `infra/authentik/traefik.labels.yml` — ingress
|
|
- `infra/authentik/bootstrap/blueprints/` — config-as-code
|
|
- `infra/authentik/backup/` — daily backup container
|
|
- `infra/authentik/scripts/smoke.sh` — post-deploy verifier
|
|
- Phase 09 detail-MD:
|
|
`docs/improvement-plan/09-authentik-admin-sso/authentik-setup.md`
|
|
- Lite alternative (LDAP-only):
|
|
`docs/operations/authentik-alternatives.md`
|