toolctrl-labs/HiveGrid-Wireframes
9
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
e341c5938ac25ff069732732835837de3d39d5e1
HiveGrid-Wireframes/hivegrid-cloud/docs/operations/authentik-setup.md
T

14 KiB
Raw Blame History

Authentik Self-Hosted — Ops Runbook

This runbook covers the Phase 09 #81 Authentik IdP stack: deploy on staging, post-deploy verification, hardening, backup + restore, and migration to a dedicated prod host. The artifacts live under infra/authentik/; this document covers the operator side.

Status: artifact-complete (this commit). Staging-deployment is operator owned — Authentik is NOT yet running on staging.

1. Architecture overview

Four containers, all on the authentik-net internal Docker network. Only the server container is additionally attached to the Coolify-managed coolify external network so Traefik can route inbound HTTPS:

                    Traefik (coolify-proxy)
                            |
                   auth.boiler.saas-ctrl.com
                            |
        +-------------------+--------------------+
        |                                        |
        | authentik-net                          | coolify-net
        |                                        |
        v                                        |
    +-------+    +-------+    +--------+    +----+-----+
    | server|<-->|worker |    |postgres|    |   redis  |
    |  9000 |    |       |    |  5432  |    |   6379   |
    +---+---+    +---+---+    +---+----+    +----+-----+
        ^            ^            ^              ^
        |            |            |              |
        +------- authentik-net (internal) -------+
  • server — Django web + REST API. Mounts the project blueprints from infra/authentik/bootstrap/blueprints/ at /blueprints/toolctrl/.
  • worker — Background jobs, blueprint application, email send. Shares the same image.
  • postgresql — Authentik's own Postgres 16. Not shared with the boilerplate app DB.
  • redis — Authentik's own Redis 7. Not shared with the boilerplate app Redis.

Volumes (named, pinned in docker-compose.yml):

  • authentik-pg-data (Postgres data)
  • authentik-redis-data (Redis RDB)
  • authentik-media (uploaded avatars + branding assets)
  • authentik-certs (Authentik-generated signing certs)
  • authentik-templates (custom email + UI templates)

2. DNS setup

Authentik runs on the subdomain auth.boiler.saas-ctrl.com pointing at the staging IP 46.225.5.93. Create the A record via the Cloudflare API:

# Cloudflare creds at ~/.cli-tokens/cloudflare-token
#   line 1 = Global API Key
#   line 2 = account email
CF_TOKEN="$(sed -n '1p' ~/.cli-tokens/cloudflare-token)"
CF_EMAIL="$(sed -n '2p' ~/.cli-tokens/cloudflare-token)"

ZONE_ID="$(curl -fsS \
  -H "X-Auth-Email: ${CF_EMAIL}" \
  -H "X-Auth-Key: ${CF_TOKEN}" \
  "https://api.cloudflare.com/client/v4/zones?name=saas-ctrl.com" \
  | jq -r '.result[0].id')"

curl -fsS -X POST \
  -H "X-Auth-Email: ${CF_EMAIL}" \
  -H "X-Auth-Key: ${CF_TOKEN}" \
  -H "Content-Type: application/json" \
  --data '{
    "type": "A",
    "name": "auth.boiler",
    "content": "46.225.5.93",
    "ttl": 300,
    "proxied": false
  }' \
  "https://api.cloudflare.com/client/v4/zones/${ZONE_ID}/dns_records" \
  | jq .

# Wait ~30s for propagation, then verify:
dig +short auth.boiler.saas-ctrl.com
# Expected: 46.225.5.93

proxied: false keeps it grey-clouded — Traefik handles TLS via Let's Encrypt HTTP-01. If you want Cloudflare-proxied + origin-cert, flip proxied: true AND switch Traefik to a TLS-passthrough resolver (out of scope for this task; same pattern as Sentry).

3. Coolify deploy

  1. In the Coolify dashboard, open the boilerplate project's validation environment and click + New Resource -> Docker Compose.
  2. Source: this repository, branch main, build-context infra/authentik/. Compose path: docker-compose.yml.
  3. Domain: add auth.boiler.saas-ctrl.com on the server service. Coolify auto-attaches the standard Traefik labels (router rule, https entrypoint, letsencrypt cert resolver). Verify they match infra/authentik/traefik.labels.yml.
  4. Environment variables — copy from infra/authentik/.env.example into Coolify-Vault, filling secrets: 
    AUTHENTIK_SECRET_KEY=$(openssl rand -base64 60)
AUTHENTIK_POSTGRESQL__PASSWORD=$(openssl rand -base64 32)
AUTHENTIK_BOOTSTRAP_PASSWORD=$(openssl rand -base64 24)
AUTHENTIK_BOOTSTRAP_TOKEN=$(openssl rand -hex 32)
TOOLCTRL_ADMIN_OIDC_CLIENT_SECRET=$(openssl rand -hex 32)
AUTHENTIK_BOOTSTRAP_EMAIL=ops@toolctrl.de  # operator's monitored mailbox
AUTHENTIK_SENTRY__DSN=$(grep AUTHENTIK_DSN ~/.cli-tokens/boilerplate-staging-env | cut -d= -f2)
    Save the four randomly-generated secrets to your offline password vault (1Password / Bitwarden / Yubikey) as well as Coolify-Vault. Losing AUTHENTIK_SECRET_KEY = full re-onboarding of every admin user.
  5. Click Deploy. First boot:
    • postgresql + redis start, server waits for service_healthy
    • server runs migrations (~30-60s)
    • worker boots, discovers /blueprints/toolctrl/*.yaml, applies them
    • bootstrap admin is created from AUTHENTIK_BOOTSTRAP_EMAIL + AUTHENTIK_BOOTSTRAP_PASSWORD
  6. Once docker ps shows the server container (healthy), run the bootstrap wrapper to force-apply blueprints + print operator next steps: 
    ssh boilerplate-staging
cd /var/lib/coolify/services/authentik    # or wherever Coolify mounted it
AUTHENTIK_BASE_URL=https://auth.boiler.saas-ctrl.com \
AUTHENTIK_BOOTSTRAP_TOKEN=<from-vault> \
bash bootstrap/bootstrap.sh
  7. Smoke-test from your laptop: 
    AUTHENTIK_BASE_URL=https://auth.boiler.saas-ctrl.com \
AUTHENTIK_APP_SLUG=toolctrl-admin-panel \
bash infra/authentik/scripts/smoke.sh
    All five probes must pass (root 200, health-live 200, health-ready 200, OIDC discovery 200 with the four required fields, self-enrollment 404).

4. Bootstrap admin first login + password rotation

  1. Browse to https://auth.boiler.saas-ctrl.com/ and log in with AUTHENTIK_BOOTSTRAP_EMAIL / AUTHENTIK_BOOTSTRAP_PASSWORD.
  2. Authentik forces a password change on first login. Set a new strong password (meets the toolctrl-password-policy: 12+ chars, mixed case
    • digit + symbol, no HIBP hit, zxcvbn >= 3).
  3. Enroll TOTP: User-Settings -> Authenticators -> Set up TOTP. Scan the QR with the operator's hardware-MFA app (Yubikey-Authenticator / 1Password / Authy). Required because the toolctrl-require-mfa-totp-for-admin policy denies access to the admin-panel application without it.
  4. Rotate the bootstrap token: Directory -> Tokens -> find the bootstrap token (look up by intent: app_password and the bootstrap-email user), delete it. Create a personal admin token instead and store it in Coolify-Vault under AUTHENTIK_ADMIN_TOKEN for future re-bootstrap runs.
  5. Add the bootstrap admin to the toolctrl-admin group (it's NOT in the group by default — Authentik treats it as a built-in superuser; the group binding is what the admin-panel OIDC app uses for authorisation).

5. Hardening checklist

  • AUTHENTIK_ERROR_REPORTING__ENABLED=false set in Coolify-Vault
  • AUTHENTIK_DISABLE_UPDATE_CHECK=true
  • AUTHENTIK_DISABLE_STARTUP_ANALYTICS=true
  • Internal SSL OFF (TLS terminated by Traefik) — verify by curl -ksI https://auth.boiler.saas-ctrl.com/ shows server: Traefik not server: gunicorn.
  • MFA-TOTP policy applied to toolctrl-admin group — verified by creating a test admin without TOTP and seeing the policy deny.
  • Public self-registration off (/if/flow/default-source-enrollment/ returns 404 — covered by scripts/smoke.sh).
  • Brute-force lockout policy applied to the default-authentication-flow — verified by triggering 6 failed logins in 1 minute and seeing the 7th request hit a reputation-deny.
  • Traefik secure-headers middleware attached to the router (HSTS 2yr, frame-deny, no-referrer, X-Robots-Tag noindex). Verify by curl -ksI https://auth.boiler.saas-ctrl.com/ | grep -i strict-transport.
  • Traefik rate-limit (60/min burst 120) attached on top of Authentik's own brute-force policy.
  • Bootstrap token rotated AND removed from Coolify-Vault (replaced by personal admin token).
  • AUTHENTIK_SECRET_KEY backed up to offline password-vault.
  • Backup container deployed + first nightly run verified (aws s3 ls s3://${AUTHENTIK_S3_BACKUP_BUCKET}/authentik/$(date +%Y-%m-%d)/).
  • Brand whitelisting: Authentik UI -> System -> Brands -> default brand has default flow background set to a Toolctrl asset (or default OK).
  • OIDC application access bindings reviewed: only toolctrl-admin and toolctrl-staff allowed; no everyone binding.

6. Backup restore drill

The backup container (infra/authentik/backup/) writes three encrypted artifacts daily to s3://<bucket>/authentik/YYYY-MM-DD/:

  • authentik-postgres-*.dump.gpg — pg_dump custom format
  • authentik-media-*.tar.gz.gpg — /media volume
  • SECRETS-MANIFEST-*.txt.gpg — list of secrets NOT in the dump

To verify restore on a throwaway test host (DO THIS QUARTERLY):

# 1. Fetch latest dump
aws s3 cp --endpoint-url "$AUTHENTIK_S3_BACKUP_ENDPOINT" \
  "s3://$AUTHENTIK_S3_BACKUP_BUCKET/authentik/$(date -u +%Y-%m-%d)/" \
  ./restore/ --recursive

# 2. Decrypt (symmetric fallback — use the same AUTHENTIK_SECRET_KEY)
for f in ./restore/*.gpg; do
  gpg --batch --yes --decrypt --passphrase "$AUTHENTIK_SECRET_KEY" \
      --output "${f%.gpg}" "$f"
done

# 3. Spin up a throwaway Postgres + restore
docker run -d --name pg-restore -e POSTGRES_PASSWORD=tmp -p 5440:5432 postgres:16-alpine
docker exec -i pg-restore psql -U postgres -c "CREATE DATABASE authentik;"
docker exec -i pg-restore pg_restore --no-owner --no-privileges --dbname authentik \
  < ./restore/authentik-postgres-*.dump

# 4. Verify schema + data
docker exec -i pg-restore psql -U postgres -d authentik -c "\dt" | head
docker exec -i pg-restore psql -U postgres -d authentik -c "SELECT COUNT(*) FROM authentik_core_user;"
docker exec -i pg-restore psql -U postgres -d authentik -c "SELECT name FROM authentik_core_group WHERE name LIKE 'toolctrl-%';"

# 5. Spin up Authentik server against the restored DB with the SAME secret
#    key, hit /-/health/ready/ to confirm it boots cleanly.

# 6. Teardown
docker rm -f pg-restore
rm -rf ./restore

If the restore boots cleanly and the smoke probes pass, the drill is green. Document the drill date in the next operator-meeting log.

7. Update procedure

To bump the Authentik tag (e.g. 2026.2.3 -> 2026.2.4):

  1. Read upstream release notes: https://github.com/goauthentik/authentik/releases
  2. Update the tag in two places:
    • infra/authentik/UPSTREAM.md — pinned version table
    • infra/authentik/docker-compose.yml — both server and worker image refs
  3. Take a manual pre-update backup: 
    ssh boilerplate-staging \
  "docker exec authentik-backup-1 /srv/backup/backup.sh"
  4. Commit the version bump, push to main, let Coolify auto-redeploy.
  5. Volumes are preserved; the new server runs schema migrations on boot.
  6. Smoke-test (scripts/smoke.sh).
  7. If anything is wrong: rollback via git revert of the bump commit. The pre-update backup is the safety net for data corruption.

For a major-line jump (e.g. 2026.2.x -> 2026.5.x):

  • READ the breaking-changes section of the upstream release notes carefully.
  • Check whether the blueprint schema changed (fields renamed / removed) — if yes, update bootstrap/blueprints/*.yaml IN THE SAME COMMIT as the version bump so the worker doesn't reject blueprints on boot.
  • Run the restore drill BEFORE the bump on a copy of yesterday's backup using the new image to confirm migrations apply cleanly.

8. Migration to a dedicated prod-Authentik host

Staging shares the host with the boilerplate app. Production should have a dedicated host (smaller blast radius, separate maintenance window):

  1. Provision a new Hetzner CCX13 (4 vCPU / 16 GB / 80 GB) in nbg1 with Coolify pre-installed (or a plain Docker Engine + Traefik if you prefer).
  2. DNS: auth.<prod-domain> -> new host IP (same Cloudflare procedure).
  3. Copy infra/authentik/ to the new host (via Coolify git-pull or scp).
  4. Operator-Vault: generate FRESH secrets for prod (AUTHENTIK_SECRET_KEY etc) — DO NOT reuse staging keys.
  5. Deploy + bootstrap as in Section 3.
  6. Smoke + restore drill.
  7. Once green, switch the admin-panel AUTHENTIK_OIDC_ISSUER_URL (#82) over to the prod host with a maintenance window:
    • Create the prod-equivalent OIDC application in the prod Authentik
    • Onboard real admins (create users + assign to toolctrl-admin, force TOTP enrollment)
    • Flip AUTHENTIK_OIDC_ISSUER_URL in the app's Coolify env
    • Verify admin login end-to-end
  8. Keep staging Authentik running but stop accepting new prod traffic; archive after 30 days if no rollback was needed.

9. References

  • Upstream docs: https://goauthentik.io/docs/
  • Blueprints reference: https://goauthentik.io/docs/customize/blueprints
  • This repo:
    • infra/authentik/README.md — overview
    • infra/authentik/UPSTREAM.md — version pin rationale
    • infra/authentik/docker-compose.yml — runtime
    • infra/authentik/traefik.labels.yml — ingress
    • infra/authentik/bootstrap/blueprints/ — config-as-code
    • infra/authentik/backup/ — daily backup container
    • infra/authentik/scripts/smoke.sh — post-deploy verifier
  • Phase 09 detail-MD: docs/improvement-plan/09-authentik-admin-sso/authentik-setup.md
  • Lite alternative (LDAP-only): docs/operations/authentik-alternatives.md
Reference in new issue View Git Blame Copy Permalink