toolctrl-labs/HiveGrid-Wireframes
9
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Track HiveGrid Cloud planning reference

Browse Source
This commit is contained in:
david committed 2026-06-26 20:48:01 +02:00
1 parent b3d5ea6764
commit e341c5938a
5784 files changed
+1104929 -3
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

hivegrid-cloud/docs/operations/admin-sso.md
+182
View File
@@ -0,0 +1,182 @@
# Admin-Panel SSO via Authentik
Phase 09 Task #82. Status: **code-complete, awaits Authentik deploy**.
This document covers operating the admin-panel SSO flow once Phase 09 #81
has shipped the Authentik instance.
## Overview
```mermaid
sequenceDiagram
participant Browser
participant Web as Web (admin host)
participant Authentik
participant DB
Browser->>Web: GET /auth/admin/login
Web->>Web: createAdminSsoChallenge() (PKCE+state+nonce)
Web-->>Browser: 302 -> Authentik authorize, Set-Cookie: admin_sso_challenge
Browser->>Authentik: GET /application/o/authorize/...
Authentik-->>Browser: Login + MFA, then 302 /auth/admin/callback?code=...&state=...
Browser->>Web: GET /auth/admin/callback (with cookie)
Web->>Web: consumeAdminSsoChallenge() -> verify state+nonce+sig
Web->>Authentik: POST /token (PKCE verifier + client_secret_post)
Authentik-->>Web: id_token + access_token
Web->>Web: extractAdminSsoClaims(idToken) + email_verified check
Web->>DB: jitProvisionAdmin() -> upsert User + AdminUser
Web->>DB: createAuditEvent('admin.login.sso.success')
Web-->>Browser: 302 /admin/dashboard + Set-Cookie session + id_token cookie
Note over Browser,Web: Logout (RP-initiated)
Browser->>Web: POST /auth/admin/logout
Web->>Web: readIdTokenFromRequest() + destroyAppSession()
Web-->>Browser: 302 Authentik end_session_endpoint?id_token_hint=...
Browser->>Authentik: end_session
Authentik-->>Browser: 302 post_logout_redirect_uri -> /admin/login?logged_out=1
```
## Required environment
See `.env.example` for the canonical commented block. Coolify deployment:
| Key | Source |
| ---------------------------------------------- | ------------------------------------------------- |
| `AUTHENTIK_ISSUER_URL` | `https://auth.<base-domain>/application/o/toolctrl-admin-panel/` |
| `AUTHENTIK_ADMIN_CLIENT_ID` | Blueprint client_id (default `toolctrl-admin-panel`) |
| `AUTHENTIK_ADMIN_CLIENT_SECRET` | Authentik admin UI (Coolify vault) |
| `AUTHENTIK_ADMIN_REDIRECT_URI` | `https://admin.<base-domain>/auth/admin/callback` |
| `AUTHENTIK_ADMIN_POST_LOGOUT_REDIRECT_URI` | `https://admin.<base-domain>/admin/login?logged_out=1` |
| `ADMIN_FALLBACK_LOGIN_ENABLED` | `false` (default), flip to `true` during recovery |
| `ADMIN_FALLBACK_LOGIN_EMAILS` | Comma-separated bootstrap emails |
## Authentik group → role mapping
The OIDC `groups` claim drives the AdminUser role. Highest priority wins.
| Authentik group | Internal AdminUser role | Notes |
| ---------------- | ----------------------- | -------------------------------------- |
| `toolctrl-admin` | `super_admin` | Full admin permissions; if user is in both `toolctrl-admin` and `toolctrl-staff`, this wins. |
| `toolctrl-staff` | `staff` | Limited admin permissions. |
| _other_ | _login rejected (403)_ | `admin.login.sso.failure` audit event with `reason=forbidden-group`. |
Granular permission checks remain in `apps/web/app/utils/permissions/`;
the role is the coarse switch.
## Onboarding a new admin
1. Create the user in Authentik (manual or via SCIM/LDAP if integrated).
2. Add the user to the `toolctrl-admin` or `toolctrl-staff` group.
3. Enforce MFA enrollment in Authentik (provider-side).
4. User clicks "Sign in with SSO" on `https://admin.<domain>/admin/login`.
5. First login triggers JIT-provisioning — AdminUser row appears in DB
with `ssoProvider='authentik'`, `ssoSubjectId=<sub>`, `ssoGroups=[...]`,
`lastSsoLoginAt=<now>`.
## Offboarding an admin
1. **Authentik** — remove the user from `toolctrl-admin`/`toolctrl-staff`
or deactivate the user.
2. Next SSO login will be rejected (`admin.login.sso.failure` with
`reason=forbidden-group`).
3. **Force-Logout** existing session: until the app session cookie expires,
the admin can still load admin pages. Force-revoke via:
- Operator CLI (planned, Phase 10): `pnpm cli admin:force-logout --userId=<id>`.
- Or: clear the user's session row directly in DB (search by `userId`).
- Or: rotate `SESSION_SECRET` (invalidates every session — last resort).
## Fallback (emergency) login
The email/password fallback bypasses Authentik. It is locked down by two
env-gates plus the existing rate-limit on `/login`:
- `ADMIN_FALLBACK_LOGIN_ENABLED=true`
- `ADMIN_FALLBACK_LOGIN_EMAILS` contains the requesting email (case-insensitive)
Every fallback attempt — success or failure — writes an
`admin.login.fallback.*` audit event with `severity=warn` in the metadata
(also visible as a Sentry breadcrumb when the reporter is configured).
### Disaster-recovery procedure
1. Verify Authentik is genuinely down. Sentry banner + `/auth/admin/login`
redirect failing both confirm.
2. In Coolify, set:
```
ADMIN_FALLBACK_LOGIN_ENABLED=true
ADMIN_FALLBACK_LOGIN_EMAILS=bootstrap@<base-domain>
```
Redeploy the web service.
3. Visit `https://admin.<base-domain>/admin/login` — expand the disclosure
("Use emergency password login") and sign in with the bootstrap admin.
4. Resolve the Authentik outage (see `docs/operations/authentik-setup.md`).
5. **Immediately** revert the env flag back to `ADMIN_FALLBACK_LOGIN_ENABLED=false`
and redeploy. Verify the fallback toggle is no longer visible.
6. Resolve the Sentry alert / Slack-notification (when wired).
The drill MUST be exercised end-to-end at least once on staging before
production rollout.
## App-MFA gate (Phase 01 #26 TOTP)
`requireAdminMfaForAction(request, action, userId)` enforces a fresh TOTP
step-up challenge for the closed taxonomy:
```ts
const SENSITIVE_ADMIN_ACTIONS = [
"impersonation.start",
"tenant.delete",
"permission.grant",
"audit.export",
"data.export",
] as const;
```
The cache window matches the existing step-up state (10 minutes — see
`MFA_STEP_UP_MAX_AGE_MS`). Successful challenges write
`admin.mfa.app.challenge.success`; redirects to enrollment/challenge write
`admin.mfa.app.challenge.failure`. Enrollment + challenge UI is the
existing `/admin/security/mfa-*` route from Phase 01 #26 — no new TOTP
implementation here.
## Audit-event reference
| Action key | Severity | Emitted by |
| ---------------------------------------- | -------- | ---------------------------------------------- |
| `admin.login.sso.success` | info | `/auth/admin/callback` after JIT success |
| `admin.login.sso.failure` | warn | Any `/auth/admin/*` early-return path |
| `admin.login.fallback.success` | warn | `/login` action (admin host, fallback path) |
| `admin.login.fallback.failure` | warn | `/login` action when gate rejects |
| `admin.logout.sso` | info | `/auth/admin/logout` |
| `admin.mfa.app.challenge.success` | info | `requireAdminMfaForAction` happy path |
| `admin.mfa.app.challenge.failure` | warn | `requireAdminMfaForAction` redirect/deny path |
All events carry `ip`, `userAgent`, `actorAdminUserId` (when resolvable)
and an action-specific `meta` JSON blob. The audit wrapper redacts any
suspicious key (authorization/cookie/password/secret/token) before
persisting to `AuditEvent`.
## Troubleshooting
| Symptom | Cause / Fix |
| ------------------------------------------------------- | ------------------------------------------------------ |
| `/auth/admin/login` redirects to `/admin/login?error=sso_unavailable` | env vars missing or Authentik unreachable. Check Sentry for `AdminSsoConfigError` (`discovery-failed`). |
| `error=sso_state_invalid` after Authentik | Challenge cookie expired, signed under stale secret, or tampered. Have user retry; check `CRYPTO_SECRET` rotation. |
| `error=sso_forbidden_group` | User not in `toolctrl-admin`/`toolctrl-staff`. Add them in Authentik then retry. |
| `error=sso_email_not_verified` | Authentik user has not verified their email; enforce verification in Authentik provider settings. |
| `error=sso_token_exchange_failed` | Client secret mismatch, clock-skew, or redirect-URI drift. Compare `.env` vs Authentik UI. |
| JWKS-stale after key rotation | openid-client refreshes JWKS lazily; first failure forces a refetch. If repeated, restart the web container. |
| Group claim empty in ID-token | Authentik provider must include `groups` scope mapping — see blueprint `03-application-toolctrl-admin.yaml`. |
## Rollback / kill-switch
- **Disable SSO entry-point only**: unset the Authentik env vars and
enable the fallback. `/auth/admin/login` will redirect with
`?error=sso_unavailable`; the UI shows only the password form.
- **Disable the whole flow**: revert the commits tagged
`09/admin-oidc-login: ...`. The Prisma migration is additive
(`AdminUser` SSO columns are nullable) so no data-loss.
- **Compromise (client_secret leak)**: rotate via Authentik admin UI →
update `AUTHENTIK_ADMIN_CLIENT_SECRET` in Coolify vault → redeploy.
Existing app sessions remain valid (they are independent of the
Authentik secret).
hivegrid-cloud/docs/operations/authentik-alternatives.md
+63
View File
@@ -0,0 +1,63 @@
# Authentik — Lite Alternative: GLAuth
This is a short note about a deliberately-smaller IdP we considered for the
Phase 09 #81 setup but did **not** ship. It's documented here so the choice
is reversible later if the operational profile changes.
## Recommendation
**Use Authentik for staging + prod** (the chosen approach — see
`docs/operations/authentik-setup.md`). It supports MFA-TOTP enforcement,
OIDC + SAML + SCIM, declarative blueprints, brand-flows, a real web UI,
and is actively maintained.
**Consider GLAuth ONLY if** all of these apply:
- You only need **LDAP** (no OIDC, no SAML, no MFA enforcement at the IdP
layer).
- You are running on a VM with **less than 1 GB RAM free** for the whole IdP
stack (Authentik server+worker+postgres+redis needs ~1.5-2 GB).
- You are comfortable managing groups + users via YAML files + restart
rather than a UI.
- You don't need a self-service password reset, email-stage flows, brand
customisation, audit log UI, or invitation flows.
For our boilerplate (DSGVO, admin SSO, MFA pflicht, OIDC for the admin
panel, eventual customer-tenant-SSO) — **none** of those conditions apply.
## GLAuth profile (for reference)
- **Project:** <https://github.com/glauth/glauth>
- **Footprint:** single Go binary, ~25 MB image, ~15-50 MB RAM resident
- **Storage:** file-backed config (`config.toml` with users + groups inline)
OR a SQL backend for larger directories
- **Protocols:** LDAP (LDAPv3) only; some optional plugins for plain HTTP
password verification
- **Auth:** username + password (with bcrypt / SHA / OTP via plugin); NO
built-in OIDC, NO SAML, NO SCIM
- **MFA:** TOTP via plugin only; not enforceable by group/policy in the
same way Authentik does it
- **Audit:** stdout logs only; no event-bus
- **License:** MIT
- **Maintenance status (2026-05):** active, but small community; releases
every 2-3 months
## If we ever switch
The migration path from Authentik to GLAuth would be **manual**:
1. Export users + groups from Authentik via the API
(`/api/v3/core/users/`, `/api/v3/core/groups/`).
2. Translate into GLAuth's `config.toml` schema (users + groups + LDAP
attributes). Passwords are NOT exportable — every user must reset their
password on the new IdP.
3. Adapt every consuming app: the admin panel (#82) would need to swap its
OIDC client for an LDAP-bind code path. There is **no** OIDC bridge in
GLAuth.
4. Lose: MFA enforcement, audit-log UI, brand-flow customisation, the
declarative-blueprint workflow.
Realistically: **don't switch**. If Authentik's footprint genuinely becomes
the bottleneck (e.g. Hetzner-CX21 with 4 GB RAM is too small), the better
move is to give Authentik its own VM (see Section 8 of
`docs/operations/authentik-setup.md`) rather than swap the IdP.
hivegrid-cloud/docs/operations/authentik-setup.md
+322
View File
@@ -0,0 +1,322 @@
# Authentik Self-Hosted — Ops Runbook
This runbook covers the Phase 09 #81 Authentik IdP stack: deploy on staging,
post-deploy verification, hardening, backup + restore, and migration to a
dedicated prod host. The artifacts live under `infra/authentik/`; this
document covers the operator side.
> **Status:** artifact-complete (this commit). Staging-deployment is operator
> owned — Authentik is **NOT yet running** on staging.
---
## 1. Architecture overview
Four containers, all on the `authentik-net` internal Docker network. Only the
`server` container is additionally attached to the Coolify-managed `coolify`
external network so Traefik can route inbound HTTPS:
```
Traefik (coolify-proxy)
|
auth.boiler.saas-ctrl.com
|
+-------------------+--------------------+
| |
| authentik-net | coolify-net
| |
v |
+-------+ +-------+ +--------+ +----+-----+
| server|<-->|worker | |postgres| | redis |
| 9000 | | | | 5432 | | 6379 |
+---+---+ +---+---+ +---+----+ +----+-----+
^ ^ ^ ^
| | | |
+------- authentik-net (internal) -------+
```
- **server** — Django web + REST API. Mounts the project blueprints from
`infra/authentik/bootstrap/blueprints/` at `/blueprints/toolctrl/`.
- **worker** — Background jobs, blueprint application, email send. Shares
the same image.
- **postgresql** — Authentik's own Postgres 16. **Not shared** with the
boilerplate app DB.
- **redis** — Authentik's own Redis 7. **Not shared** with the boilerplate
app Redis.
Volumes (named, pinned in `docker-compose.yml`):
- `authentik-pg-data` (Postgres data)
- `authentik-redis-data` (Redis RDB)
- `authentik-media` (uploaded avatars + branding assets)
- `authentik-certs` (Authentik-generated signing certs)
- `authentik-templates` (custom email + UI templates)
---
## 2. DNS setup
Authentik runs on the subdomain `auth.boiler.saas-ctrl.com` pointing at the
staging IP `46.225.5.93`. Create the A record via the Cloudflare API:
```bash
# Cloudflare creds at ~/.cli-tokens/cloudflare-token
# line 1 = Global API Key
# line 2 = account email
CF_TOKEN="$(sed -n '1p' ~/.cli-tokens/cloudflare-token)"
CF_EMAIL="$(sed -n '2p' ~/.cli-tokens/cloudflare-token)"
ZONE_ID="$(curl -fsS \
-H "X-Auth-Email: ${CF_EMAIL}" \
-H "X-Auth-Key: ${CF_TOKEN}" \
"https://api.cloudflare.com/client/v4/zones?name=saas-ctrl.com" \
| jq -r '.result[0].id')"
curl -fsS -X POST \
-H "X-Auth-Email: ${CF_EMAIL}" \
-H "X-Auth-Key: ${CF_TOKEN}" \
-H "Content-Type: application/json" \
--data '{
"type": "A",
"name": "auth.boiler",
"content": "46.225.5.93",
"ttl": 300,
"proxied": false
}' \
"https://api.cloudflare.com/client/v4/zones/${ZONE_ID}/dns_records" \
| jq .
# Wait ~30s for propagation, then verify:
dig +short auth.boiler.saas-ctrl.com
# Expected: 46.225.5.93
```
`proxied: false` keeps it grey-clouded — Traefik handles TLS via Let's
Encrypt HTTP-01. If you want Cloudflare-proxied + origin-cert, flip
`proxied: true` AND switch Traefik to a TLS-passthrough resolver (out of
scope for this task; same pattern as Sentry).
---
## 3. Coolify deploy
1. In the Coolify dashboard, open the `boilerplate` project's `validation`
environment and click **+ New Resource** -> **Docker Compose**.
2. Source: this repository, branch `main`, build-context `infra/authentik/`.
Compose path: `docker-compose.yml`.
3. **Domain:** add `auth.boiler.saas-ctrl.com` on the `server` service.
Coolify auto-attaches the standard Traefik labels (router rule, https
entrypoint, `letsencrypt` cert resolver). Verify they match
`infra/authentik/traefik.labels.yml`.
4. **Environment variables** — copy from `infra/authentik/.env.example` into
Coolify-Vault, filling secrets:
```bash
AUTHENTIK_SECRET_KEY=$(openssl rand -base64 60)
AUTHENTIK_POSTGRESQL__PASSWORD=$(openssl rand -base64 32)
AUTHENTIK_BOOTSTRAP_PASSWORD=$(openssl rand -base64 24)
AUTHENTIK_BOOTSTRAP_TOKEN=$(openssl rand -hex 32)
TOOLCTRL_ADMIN_OIDC_CLIENT_SECRET=$(openssl rand -hex 32)
AUTHENTIK_BOOTSTRAP_EMAIL=ops@toolctrl.de # operator's monitored mailbox
AUTHENTIK_SENTRY__DSN=$(grep AUTHENTIK_DSN ~/.cli-tokens/boilerplate-staging-env | cut -d= -f2)
```
Save the four randomly-generated secrets to your offline password vault
(1Password / Bitwarden / Yubikey) **as well as** Coolify-Vault. Losing
`AUTHENTIK_SECRET_KEY` = full re-onboarding of every admin user.
5. Click **Deploy**. First boot:
- postgresql + redis start, server waits for `service_healthy`
- server runs migrations (~30-60s)
- worker boots, discovers `/blueprints/toolctrl/*.yaml`, applies them
- bootstrap admin is created from `AUTHENTIK_BOOTSTRAP_EMAIL` +
`AUTHENTIK_BOOTSTRAP_PASSWORD`
6. Once `docker ps` shows the server container `(healthy)`, run the
bootstrap wrapper to force-apply blueprints + print operator next steps:
```bash
ssh boilerplate-staging
cd /var/lib/coolify/services/authentik # or wherever Coolify mounted it
AUTHENTIK_BASE_URL=https://auth.boiler.saas-ctrl.com \
AUTHENTIK_BOOTSTRAP_TOKEN=<from-vault> \
bash bootstrap/bootstrap.sh
```
7. Smoke-test from your laptop:
```bash
AUTHENTIK_BASE_URL=https://auth.boiler.saas-ctrl.com \
AUTHENTIK_APP_SLUG=toolctrl-admin-panel \
bash infra/authentik/scripts/smoke.sh
```
All five probes must pass (root 200, health-live 200, health-ready 200,
OIDC discovery 200 with the four required fields, self-enrollment 404).
---
## 4. Bootstrap admin first login + password rotation
1. Browse to <https://auth.boiler.saas-ctrl.com/> and log in with
`AUTHENTIK_BOOTSTRAP_EMAIL` / `AUTHENTIK_BOOTSTRAP_PASSWORD`.
2. Authentik forces a password change on first login. Set a new strong
password (meets the `toolctrl-password-policy`: 12+ chars, mixed case
+ digit + symbol, no HIBP hit, zxcvbn >= 3).
3. **Enroll TOTP**: User-Settings -> Authenticators -> Set up TOTP.
Scan the QR with the operator's hardware-MFA app (Yubikey-Authenticator
/ 1Password / Authy). Required because the
`toolctrl-require-mfa-totp-for-admin` policy denies access to the
admin-panel application without it.
4. **Rotate the bootstrap token**: Directory -> Tokens -> find the bootstrap
token (look up by intent: `app_password` and the bootstrap-email user),
delete it. Create a personal admin token instead and store it in
Coolify-Vault under `AUTHENTIK_ADMIN_TOKEN` for future re-bootstrap runs.
5. **Add the bootstrap admin to the `toolctrl-admin` group** (it's NOT in
the group by default — Authentik treats it as a built-in superuser; the
group binding is what the admin-panel OIDC app uses for authorisation).
---
## 5. Hardening checklist
- [ ] `AUTHENTIK_ERROR_REPORTING__ENABLED=false` set in Coolify-Vault
- [ ] `AUTHENTIK_DISABLE_UPDATE_CHECK=true`
- [ ] `AUTHENTIK_DISABLE_STARTUP_ANALYTICS=true`
- [ ] Internal SSL OFF (TLS terminated by Traefik) — verify by
`curl -ksI https://auth.boiler.saas-ctrl.com/` shows
`server: Traefik` not `server: gunicorn`.
- [ ] MFA-TOTP policy applied to `toolctrl-admin` group — verified by
creating a test admin without TOTP and seeing the policy deny.
- [ ] Public self-registration off (`/if/flow/default-source-enrollment/`
returns 404 — covered by `scripts/smoke.sh`).
- [ ] Brute-force lockout policy applied to the default-authentication-flow
— verified by triggering 6 failed logins in 1 minute and seeing the
7th request hit a reputation-deny.
- [ ] Traefik secure-headers middleware attached to the router
(HSTS 2yr, frame-deny, no-referrer, X-Robots-Tag noindex). Verify by
`curl -ksI https://auth.boiler.saas-ctrl.com/ | grep -i strict-transport`.
- [ ] Traefik rate-limit (60/min burst 120) attached on top of Authentik's
own brute-force policy.
- [ ] Bootstrap token rotated AND removed from Coolify-Vault (replaced by
personal admin token).
- [ ] `AUTHENTIK_SECRET_KEY` backed up to offline password-vault.
- [ ] Backup container deployed + first nightly run verified
(`aws s3 ls s3://${AUTHENTIK_S3_BACKUP_BUCKET}/authentik/$(date +%Y-%m-%d)/`).
- [ ] Brand whitelisting: Authentik UI -> System -> Brands -> default brand
has `default flow background` set to a Toolctrl asset (or default OK).
- [ ] OIDC application access bindings reviewed: only `toolctrl-admin` and
`toolctrl-staff` allowed; no `everyone` binding.
---
## 6. Backup restore drill
The backup container (`infra/authentik/backup/`) writes three encrypted
artifacts daily to `s3://<bucket>/authentik/YYYY-MM-DD/`:
- `authentik-postgres-*.dump.gpg` — pg_dump custom format
- `authentik-media-*.tar.gz.gpg` — /media volume
- `SECRETS-MANIFEST-*.txt.gpg` — list of secrets NOT in the dump
To verify restore on a throwaway test host (DO THIS QUARTERLY):
```bash
# 1. Fetch latest dump
aws s3 cp --endpoint-url "$AUTHENTIK_S3_BACKUP_ENDPOINT" \
"s3://$AUTHENTIK_S3_BACKUP_BUCKET/authentik/$(date -u +%Y-%m-%d)/" \
./restore/ --recursive
# 2. Decrypt (symmetric fallback — use the same AUTHENTIK_SECRET_KEY)
for f in ./restore/*.gpg; do
gpg --batch --yes --decrypt --passphrase "$AUTHENTIK_SECRET_KEY" \
--output "${f%.gpg}" "$f"
done
# 3. Spin up a throwaway Postgres + restore
docker run -d --name pg-restore -e POSTGRES_PASSWORD=tmp -p 5440:5432 postgres:16-alpine
docker exec -i pg-restore psql -U postgres -c "CREATE DATABASE authentik;"
docker exec -i pg-restore pg_restore --no-owner --no-privileges --dbname authentik \
< ./restore/authentik-postgres-*.dump
# 4. Verify schema + data
docker exec -i pg-restore psql -U postgres -d authentik -c "\dt" | head
docker exec -i pg-restore psql -U postgres -d authentik -c "SELECT COUNT(*) FROM authentik_core_user;"
docker exec -i pg-restore psql -U postgres -d authentik -c "SELECT name FROM authentik_core_group WHERE name LIKE 'toolctrl-%';"
# 5. Spin up Authentik server against the restored DB with the SAME secret
# key, hit /-/health/ready/ to confirm it boots cleanly.
# 6. Teardown
docker rm -f pg-restore
rm -rf ./restore
```
If the restore boots cleanly and the smoke probes pass, the drill is green.
Document the drill date in the next operator-meeting log.
---
## 7. Update procedure
To bump the Authentik tag (e.g. 2026.2.3 -> 2026.2.4):
1. Read upstream release notes: <https://github.com/goauthentik/authentik/releases>
2. Update the tag in two places:
- `infra/authentik/UPSTREAM.md` — pinned version table
- `infra/authentik/docker-compose.yml` — both `server` and `worker` image refs
3. Take a manual pre-update backup:
```bash
ssh boilerplate-staging \
"docker exec authentik-backup-1 /srv/backup/backup.sh"
```
4. Commit the version bump, push to `main`, let Coolify auto-redeploy.
5. Volumes are preserved; the new `server` runs schema migrations on boot.
6. Smoke-test (`scripts/smoke.sh`).
7. If anything is wrong: rollback via `git revert` of the bump commit. The
pre-update backup is the safety net for data corruption.
For a major-line jump (e.g. `2026.2.x -> 2026.5.x`):
- READ the breaking-changes section of the upstream release notes carefully.
- Check whether the blueprint schema changed (fields renamed / removed) — if
yes, update `bootstrap/blueprints/*.yaml` IN THE SAME COMMIT as the version
bump so the worker doesn't reject blueprints on boot.
- Run the restore drill BEFORE the bump on a copy of yesterday's backup
using the new image to confirm migrations apply cleanly.
---
## 8. Migration to a dedicated prod-Authentik host
Staging shares the host with the boilerplate app. Production should have a
dedicated host (smaller blast radius, separate maintenance window):
1. Provision a new Hetzner CCX13 (4 vCPU / 16 GB / 80 GB) in `nbg1` with
Coolify pre-installed (or a plain Docker Engine + Traefik if you prefer).
2. DNS: `auth.<prod-domain>` -> new host IP (same Cloudflare procedure).
3. Copy `infra/authentik/` to the new host (via Coolify git-pull or scp).
4. Operator-Vault: generate FRESH secrets for prod
(`AUTHENTIK_SECRET_KEY` etc) — DO NOT reuse staging keys.
5. Deploy + bootstrap as in Section 3.
6. Smoke + restore drill.
7. Once green, switch the admin-panel `AUTHENTIK_OIDC_ISSUER_URL` (#82) over
to the prod host with a maintenance window:
- Create the prod-equivalent OIDC application in the prod Authentik
- Onboard real admins (create users + assign to `toolctrl-admin`,
force TOTP enrollment)
- Flip `AUTHENTIK_OIDC_ISSUER_URL` in the app's Coolify env
- Verify admin login end-to-end
8. Keep staging Authentik running but stop accepting new prod traffic;
archive after 30 days if no rollback was needed.
---
## 9. References
- Upstream docs: <https://goauthentik.io/docs/>
- Blueprints reference: <https://goauthentik.io/docs/customize/blueprints>
- This repo:
- `infra/authentik/README.md` — overview
- `infra/authentik/UPSTREAM.md` — version pin rationale
- `infra/authentik/docker-compose.yml` — runtime
- `infra/authentik/traefik.labels.yml` — ingress
- `infra/authentik/bootstrap/blueprints/` — config-as-code
- `infra/authentik/backup/` — daily backup container
- `infra/authentik/scripts/smoke.sh` — post-deploy verifier
- Phase 09 detail-MD:
`docs/improvement-plan/09-authentik-admin-sso/authentik-setup.md`
- Lite alternative (LDAP-only):
`docs/operations/authentik-alternatives.md`
hivegrid-cloud/docs/operations/capacitor.md
+249
View File
@@ -0,0 +1,249 @@
# Capacitor mobile wrapper — operations
Last updated: 2026-05-25 (mobile optimization sweep)
The toolctrl mobile app is a Capacitor 7 native wrapper around the
React Router v7 web app at `apps/web/`. There is no separate UI
codebase. This document covers local development, building for iOS /
Android, signing, and store submission.
## Quick reference
| Concern | Where |
|---|---|
| App workspace | `apps/mobile/` |
| Bundle id | `de.toolctrl.app` |
| Capacitor config | `apps/mobile/capacitor.config.ts` |
| Push backend worker | `apps/worker/src/jobs/push-notification.ts` |
| Device-token API | `apps/api/src/routes/devices-routes.ts` (`POST /v1/devices/register`) |
| Prisma model | `prisma/schema/notifications.prisma` → `model DeviceToken` |
| Sentry init | `apps/mobile/src/sentry.ts` |
## Status: code-complete, native builds and store submission deferred
The Phase 08 #65 scope ships:
- The `apps/mobile/` workspace skeleton + Capacitor 7 config.
- Hand-scaffolded iOS + Android native projects under
`apps/mobile/{ios,android}/`. An operator with Xcode / Android Studio
installed should re-run `npx cap add ios && npx cap add android` once
to overwrite the skeletons with the canonical CLI output.
- DeviceToken Prisma model + `POST /v1/devices/register` upsert endpoint.
- `dispatchPushNotification` worker orchestrator with APNs + FCM seams
(real SDK adapters wire in operator-side at deployment).
- `@sentry/capacitor` init mirroring the web client config.
**Explicitly deferred** (operator-owned):
- App-Store and Play-Store submission flows (screenshots, review notes,
release marketing).
- iOS signing certificate + provisioning profile management.
- Android keystore generation + native encrypted storage for long-lived tokens.
- `google-services.json` + APNs `.p8` provisioning.
- CI pipelines (Codemagic / GitHub Actions macOS-runner / Play upload).
## Local development
### Prerequisites
- macOS for iOS (Xcode 15+, CocoaPods, an Apple Developer team)
- Any OS for Android (Android Studio Hedgehog+, Android SDK 34, JDK 17)
- pnpm 10+, Node 22+
### Initial setup
```bash
# 1. Install workspace deps (from repo root)
pnpm install
# 2. Build the web bundle that the wrapper loads
pnpm --filter @toolctrl/mobile build:web
# 3. Sync web assets + plugins into the native projects
pnpm --filter @toolctrl/mobile sync
# 4. Open in the platform IDE
pnpm --filter @toolctrl/mobile open:ios # Xcode
pnpm --filter @toolctrl/mobile open:android # Android Studio
```
### Live-reload dev loop
For tight iteration cycles, point the wrapper at a running web dev
origin instead of the bundled web assets:
```bash
# iOS simulator can reach the host dev server through localhost.
export MOBILE_LIVE_RELOAD_URL=http://localhost:3000
# Android emulator maps 10.0.2.2 to the host loopback instead.
# export MOBILE_LIVE_RELOAD_URL=http://10.0.2.2:3000
pnpm --filter @toolctrl/mobile sync
pnpm --filter @toolctrl/mobile open:ios # or open:android
```
The Capacitor config emits `server.url` only when `MOBILE_LIVE_RELOAD_URL`
is set. It must be an absolute `http://` or `https://` URL. Production
builds run with the variable unset so the bundle loads the embedded
`webDir`.
Use `http://localhost:3000` for the iOS simulator, `http://10.0.2.2:3000`
for the Android emulator, and HTTPS staging/tunnel URLs for physical
devices. The committed Android skeleton only permits cleartext live reload
for `localhost` and `10.0.2.2`; do not broaden cleartext allowances in
release builds.
### Resetting the native projects
When the canonical Capacitor CLI is needed (signing config, new plugin,
platform upgrade):
```bash
cd apps/mobile
rm -rf ios android
npx cap add ios
npx cap add android
npx cap sync
```
Bundle id, app name and webDir come from `capacitor.config.ts`, so
re-creation is deterministic.
## Building iOS
```bash
cd apps/mobile/ios/App
pod install
xcodebuild \
-workspace App.xcworkspace \
-scheme App \
-configuration Release \
-archivePath build/App.xcarchive \
archive
xcodebuild \
-exportArchive \
-archivePath build/App.xcarchive \
-exportOptionsPlist ExportOptions.plist \
-exportPath build/ipa
```
`ExportOptions.plist` is operator-provided (not committed). Upload to
TestFlight via Transporter.app or `xcrun altool --upload-app`.
## Building Android
```bash
cd apps/mobile/android
./gradlew bundleRelease
# AAB lands at app/build/outputs/bundle/release/app-release.aab
```
Signing requires `signingConfigs.release` in `app/build.gradle` and a
keystore. The committed skeleton does not declare a release signing
config — operator must add one before the first release build.
## Push notifications
### Backend
The push job is implemented as a pure orchestrator
(`apps/worker/src/jobs/push-notification.ts`) with three injected
provider adapters:
- `sendApns(token, payload)` → `node-apn` / `@parse/node-apn` against the
PEM key configured in `APNS_KEY` / `APNS_KEY_ID` / `APNS_TEAM_ID`.
- `sendFcm(token, payload)` → `firebase-admin` using credentials from
`FCM_SERVICE_ACCOUNT_JSON`.
- `sendWebPush(token, payload)` → optional `web-push` adapter for the
PWA fallback path.
The adapters live in `apps/worker/src/index.ts` (operator-side
implementation, kept out of the orchestrator so unit tests stay
SDK-free). When a provider returns one of the canonical revocation
codes (`Unregistered`, `InvalidRegistration`, `BadDeviceToken`,
`messaging/registration-token-not-registered`,
`messaging/invalid-registration-token`), the corresponding DeviceToken
row is soft-deleted via `revokedAt`.
### Provider credentials
| Env var | Source | Notes |
|---|---|---|
| `FCM_SERVICE_ACCOUNT_JSON` | Firebase console → Project settings → Service accounts → Generate new private key | Either raw JSON or base64-encoded JSON |
| `APNS_KEY` | Apple Developer → Certificates, Identifiers & Profiles → Keys → Create `.p8` | PKCS#8 PEM block from the downloaded `.p8` file (the standard `BEGIN PRIVATE KEY` framing). |
| `APNS_KEY_ID` | Apple Developer (10 chars) | Shown alongside the `.p8` |
| `APNS_TEAM_ID` | Apple Developer → Account → Membership (10 chars) | |
| `APNS_BUNDLE_ID` | Defaults to `de.toolctrl.app` | Must match `capacitor.config.ts` |
**Rotate APNs keys every 12 months.** Apple does not auto-expire `.p8`
files but rotation contains compromise blast-radius.
### Device-token API
The Capacitor shell calls `POST /v1/devices/register` after the native
push plugin yields a token:
```json
POST /v1/devices/register
Authorization: Bearer <service-token>
Content-Type: application/json
{
"userId": "u-1",
"tenantId": "t-1",
"platform": "ios",
"token": "<APNs / FCM / WebPush token>",
"pushProvider": "apns",
"appVersion": "1.0.0"
}
```
The endpoint upserts by `token` (globally unique) so cold-start
re-registration produces zero duplicates. Logout calls
`POST /v1/devices/revoke` with `{ "token": "..." }`.
## App-Store / Play-Store submission
Both flows are **operator-owned** and deferred from this scaffold.
High-level checklists:
### iOS
1. Apple Developer account active (USD 99/year).
2. App Store Connect record created with `de.toolctrl.app` bundle id.
3. Push certificate / Auth Key (`.p8`) generated.
4. Privacy Policy URL published at `/legal/privacy`.
5. App Privacy questionnaire answered (we collect: contact info,
identifiers, usage data; we do NOT collect: precise location,
financial info, health data).
6. Screenshots (6.7" iPhone 15 Pro Max, 5.5" iPhone 8 Plus, optionally
12.9" iPad Pro).
7. Review notes with demo credentials.
8. Submit via TestFlight → "Submit for Review".
### Android
1. Google Play Developer account active (USD 25 one-time).
2. Play Console app created with `de.toolctrl.app` application id.
3. Service account JSON for Play API uploads.
4. Privacy Policy URL.
5. Data safety form (same answers as iOS).
6. Screenshots (phone 9:16 + tablet 16:10).
7. Internal testing → Production rollout (staged 5% → 25% → 100%).
## Rollback
- Revert the `08/capacitor:` commit range to drop the workspace.
- Disable push fan-out by flipping `mobile.push.enabled` feature flag
off (the dispatcher short-circuits without touching DeviceToken rows).
- Roll back the Prisma migration `20260517200000_08_capacitor_device_tokens`
with `prisma migrate resolve --rolled-back` + a manual `DROP TABLE`
if the schema needs to revert.
## See also
- [`docs/improvement-plan/08-mobile-cli-storybook-sdk/capacitor.md`](../improvement-plan/08-mobile-cli-storybook-sdk/capacitor.md)
- [`apps/mobile/README.md`](../../apps/mobile/README.md)
- [`apps/mobile/ios/README.md`](../../apps/mobile/ios/README.md)
- [`apps/mobile/android/README.md`](../../apps/mobile/android/README.md)
hivegrid-cloud/docs/operations/database-rollback.md
+126
View File
@@ -0,0 +1,126 @@
# Database Rollback Runbook
Practical runbook for rolling back a Prisma migration or seed in the boilerplate's PostgreSQL deployment. Pair every rollback with a forensic note in the relevant incident channel — rollbacks are not silent operations.
## Decision tree
Use this order before reaching for any destructive command:
1. **Migration applied but app fails at boot** — Most likely cause: code does not yet handle a new column, OR new column lacks a backfill. Prefer **forward-fix**: deploy a hotfix patch that handles the schema, then continue.
2. **Migration breaks data integrity** — Restore from the most recent verified backup. Do not run `prisma migrate resolve --rolled-back` against production data without a backup snapshot.
3. **Seed produced wrong rows** — Idempotent seeds (the boilerplate's permissions/entitlements seeds are idempotent by design) tolerate re-runs after manual cleanup. Non-idempotent seed sections should never run in production.
## Reverting a migration locally
The boilerplate's `prisma/schema` ships forward-only Prisma migrations. There is no Prisma equivalent of `down` migrations. To reverse a change locally:
```sh
# 1. Inspect the migration to revert
ls prisma/schema/migrations
cat prisma/schema/migrations/<TIMESTAMP>_<name>/migration.sql
# 2. Hand-roll the inverse SQL into a NEW migration
pnpm exec prisma migrate dev --name revert-<original-name>
# Edit the generated migration.sql to contain the inverse DDL.
# 3. Apply
pnpm exec prisma migrate dev
```
Never delete an applied migration directory from `prisma/schema/migrations/` on a branch other than your own work branch — the migration record on staging/production will go inconsistent.
## Reverting a migration on staging or production
Prerequisites:
- Working PostgreSQL `pg_dump`/`pg_restore` or snapshot on the deployment host.
- Confirmed shell access to the database (Coolify staging SSH defined in `verify:db-migrate-status:staging`).
- Approval recorded in the change/incident channel.
Procedure:
```sh
# 1. Snapshot first.
pg_dump --format=custom --no-owner --file=pre-rollback-$(date -u +%Y%m%dT%H%M%SZ).dump "$DATABASE_URL"
# 2. Take the application offline (Coolify -> scale to 0 / put the service in maintenance).
# TOOLCTRL_API_MAINTENANCE_MODE=1 is available for the API surface; the web surface relies on the deployment platform's maintenance toggle.
# 3. Apply the inverse migration that was prepared and committed beforehand.
pnpm exec prisma migrate deploy
# 4. Verify schema with the same gate CI uses:
pnpm run verify:db-migrate-status
# 5. Bring the application back online; tail logs for at least 5 minutes.
```
If step 3 fails halfway through, `prisma migrate resolve --rolled-back <migration-name>` marks the record as rolled-back so the runtime can re-apply. Never use `--applied` to silently mark a partial migration as done.
## Undoing a seed run
The boilerplate's production seed (`packages/db/dist/production-seed.server.js`) is gated on `NODE_ENV=production` and explicit `SEED_ADMIN_EMAIL` / `SEED_ADMIN_PASSWORD`. It is idempotent for the canonical permission/entitlement rows but may insert demo data when `SEED_DEMO_DATA=true` accidentally leaks into a non-local environment.
Undo procedure:
```sh
# 1. Snapshot first (same command as above).
# 2. Identify rows that should not exist:
psql "$DATABASE_URL" -c "SELECT id, email, tenant_id FROM \"User\" WHERE email LIKE '%@company.com';"
psql "$DATABASE_URL" -c "SELECT id, name FROM \"Tenant\" WHERE slug IN ('acme-corp-1','acme-corp-2');"
# 3. Delete in dependency order (children first). NEVER use DELETE without a WHERE clause on a production database.
```
After cleanup, re-run the canonical (non-demo) seed to confirm idempotency:
```sh
SEED_DEMO_DATA=false NODE_ENV=production node packages/db/dist/production-seed.server.js
```
## Permission/role sync rollback
`pnpm run permissions:sync` is idempotent. To undo an unwanted permission rename:
1. Revert the offending commit in `packages/db/src/permissions-seed.server.ts`.
2. Re-run `pnpm run permissions:sync` against staging.
3. Run `verify:permission-matrix` to assert documentation parity.
Permission rows themselves are immutable identifiers; the seed only adds/updates. Manual `DELETE FROM "Permission" WHERE name = '…'` is risky because foreign-key references in `RolePermission` will cascade.
## Practice drill
`scripts/drill-migration-rollback.sh` boots an ephemeral PostgreSQL
container, replays the full migration chain, runs the production seed
twice (idempotency probe), and asserts the second `prisma migrate
deploy` is a no-op. Requires Docker and pnpm.
```sh
./scripts/drill-migration-rollback.sh # full forward + seed cycle
./scripts/drill-migration-rollback.sh --no-seed # forward chain only
./scripts/drill-migration-rollback.sh --keep-container # leave PG running for inspection
```
Run it before any release that introduces migrations. A failing drill
means the migration chain or the seed is not safe to apply against a
fresh production database.
## Verification gates that detect drift
```sh
pnpm run verify:db-migrate-status # local Prisma diff against schema
pnpm run verify:db-migrate-status:staging # SSH'd staging diff (CI gate)
pnpm run verify:permission-matrix # role/permission documentation parity
pnpm run verify:production-seed-artifact # seed artifact compile + idempotency contract
```
A green run of all four after rollback is the signal that the database is back on a known-good baseline.
## Incident handoff template
When handing off after a rollback, include:
- Snapshot file location and SHA.
- Migration name and direction (forward/reverse) at the start and end of the incident.
- The four gate outputs above.
- A link to the post-incident commit that locks in the fix or the deferred follow-up.
hivegrid-cloud/docs/operations/plugin-lifecycle.md
+208
View File
@@ -0,0 +1,208 @@
# Plugin Lifecycle Operations
> **Runtime gate:** Use this runbook for staging/foundation operations until
> the external WASM sandbox security review is signed off in
> [Plugin Runtime Signoff](../security/plugin-runtime-signoff.md). Production
> plugin enablement remains blocked before that sign-off.
## Purpose
This runbook describes how operators install, grant, enable, disable, uninstall, and investigate plugins. It complements the product overview in [Plugins](../features/plugins.md) and the architecture docs in [Plugin system](../architecture/plugin-system.md) and [Plugin Hooks & Lifecycle](../architecture/plugin-hooks.md). Plugin upgrades are not a first-class lifecycle yet; treat them as reviewed replace operations until an explicit update command and data migration contract exist.
## Runtime Topology
Provision the plugin worker service now so staging and future production
topology stay identical. In production, a `worker-plugin` service with
`PLUGIN_RUNTIME_PRODUCTION_ENABLED` unset or `false` starts idle and does not
consume `toolctrl.plugin-invoke`; readiness treats the production plugin runtime
as skipped in that gated state. After the external WASM sandbox signoff,
`PLUGIN_RUNTIME_PRODUCTION_ENABLED=true` makes plugin readiness active on both
`worker-general` and `worker-plugin`, requires `PLUGIN_TRUSTED_PUBLIC_KEYS`, and
allows the plugin worker to consume invocation jobs.
- General worker group: consumes outbox fanout, including `outbox.plugin-hook`.
- Plugin worker group: runs sandboxed plugin invocations with `WORKER_QUEUE_GROUP=plugin`.
- Redis/BullMQ queue: `toolctrl.plugin-invoke`.
- Postgres: stores installed plugin rows, per-user plugin config, capability use audit, settings-change audit, and lifecycle audit events.
Do not run plugin invocation in the web container. Keep plugin execution isolated from tenant web traffic so sandbox stalls, plugin traps, and crash tracking do not block core app requests.
The canonical worker app, queue-group, port, and scaling matrix lives in
[Worker queue groups](../ops/worker-queues.md); keep this runbook focused on
plugin-specific gates and operator actions.
## Required Configuration
| Variable | Purpose | Notes |
| ---------------------------------- | ----------------------------------------- | -------------------------------------------------------------------------------- |
| `PLUGIN_RUNTIME_PRODUCTION_ENABLED` | Production rollout gate | Must be `true` in production before install, grant, enable, hook fanout, or plugin invoke workers run. Keep unset/false until external sign-off. |
| `PLUGIN_HOOK_FANOUT_REQUIRED` | General-worker fanout fail-closed switch | Optional for staging/canaries. In production with `PLUGIN_RUNTIME_PRODUCTION_ENABLED=true` and Redis configured, hook fanout is required even when this is unset. |
| `PLUGIN_TRUSTED_PUBLIC_KEYS` | CSV of trusted Ed25519 public keys | Empty means plugin signature verification fails closed. |
| `WORKER_QUEUE_GROUP=plugin` | Starts the isolated plugin worker service | Required for `toolctrl.plugin-invoke` jobs. In production before runtime signoff it starts idle without queue consumers; after `PLUGIN_RUNTIME_PRODUCTION_ENABLED=true`, startup fails closed if invocation runtime dependencies cannot bootstrap. |
| `WORKER_CONCURRENCY_PLUGIN` | Plugin worker concurrency | Defaults to the worker runtime value. Tune separately from mail/general workers. |
| `PLUGIN_HOOK_FANOUT_ENABLED` | General-worker plugin hook fanout toggle | Leave unset/true when production runtime is enabled; explicit `false` makes readiness fail. |
| `PLUGIN_CAPABILITY_USE_ENABLED` | General-worker capability-use persistence toggle | Leave unset/true when production runtime is enabled; explicit `false` makes readiness fail. |
| `PLUGIN_CAPABILITY_USE_CONCURRENCY` | Capability-use audit sink concurrency | Defaults to `1` so per-tenant capability-use hash chains stay linear. Raise only after adding store-level serialization or advisory locking. |
| `PLUGIN_BINARY_BASE_PATH` | Plugin binary lookup root | Defaults to `/app/var/plugins`; required to contain verified `<pluginId>/<version>/plugin.wasm` files before install/invoke checks can pass. |
| `PLUGIN_MAX_FUEL` | Sandbox fuel limit | Defaults to 1000000. Charged for the hook call plus capability gateway calls. |
| `PLUGIN_MAX_MEMORY_MB` | Sandbox memory cap | Defaults to 64 MiB. |
| `PLUGIN_MAX_WALL_CLOCK_MS` | Sandbox wall-clock kill budget | Defaults to 5000 ms; the plugin sandbox worker thread is terminated on timeout. |
| `PLUGIN_RATE_LIMIT_PER_MINUTE` | Per-tenant/plugin invocation rate limit | Defaults to 100. |
| `PLUGIN_RATE_LIMIT_WINDOW_SECONDS` | Rate-limit window | Defaults to 60. |
| `PLUGIN_AUTO_DISABLE_THRESHOLD` | Tenant plugin crash threshold | Default is 10. |
| `PLUGIN_AUTO_DISABLE_WINDOW_MS` | Sliding crash window | Default is 600000. |
Higher-risk host-binding resource controls remain runtime follow-up work. Do not document additional knobs as production-enforced without checking the worker bootstrap first.
## Production Readiness Matrix
| Capability slice | Operational status | Required before production use |
| ---------------- | ------------------ | ------------------------------ |
| Install / grant / revoke / enable / disable lifecycle | Implemented with tenant scope, permissions, audit events, JSON request validation, and capability review digest checks. | External WASM sandbox signoff plus `PLUGIN_RUNTIME_PRODUCTION_ENABLED=true` in the target environment. |
| Uninstall lifecycle | CLI/operator-only escape hatch that deletes the tenant `Plugin` row and writes an audit event. It is not a tenant API/UI action. | Operator backup/delete decision for plugin-owned data plus the external runtime signoff gate. |
| Settings iframe | Implemented with sandboxed iframe, restrictive CSP including `navigate-to 'none'`, denied iframe `allow` permissions, parent-side unexpected-navigation remount, channel-bound strict-schema versioned `plugin.settings.*` postMessage contract, and non-secret per-user config persistence. | Keep secret-like settings blocked until encrypted secret storage and host binding review land. |
| Hook fanout | Implemented for the foundation hook set through `outbox.plugin-hook` on `worker-general` and `toolctrl.plugin-invoke` on `worker-plugin`. | Add payload/test coverage before enabling new hook families such as billing, auth, or richer workflow hooks. |
| Safe host bindings | `log` and `audit` are gateway-controlled and audited. | Confirm trusted public keys and worker-plugin health in the deployment runbook. |
| Deferred host bindings | `http.outbound`, `secret.read`, `db.read`, `db.write`, `storage.read`, `storage.write`, and `tenant.info` intentionally fail closed. Future `http.outbound`, `secret.read`, and `db.read`/`db.write` calls already have runtime per-call scope guards for exact HTTPS host, key-pattern, and table-prefix contracts, but the bindings are not exposed yet. | Dedicated security review, SSRF/tenant-boundary tests, tenant-boundary/storage tests, secret redaction, audit contracts, and operator copy for each binding. |
## Lifecycle
Lifecycle API mutations require JSON requests. Install, grant, revoke, enable,
and disable calls to the tenant plugin API must send
`Content-Type: application/json`; malformed JSON returns `400 invalid_json`,
and missing or unsupported media types return `415 unsupported_media_type`.
Enable and disable have no fields, but API clients still send `{}` as the JSON
body so the lifecycle route keeps one validation contract.
### Install
1. Verify manifest schema and `schemaVersion`.
2. Verify `.wasm` SHA-256 against the manifest.
3. Verify signature against `PLUGIN_TRUSTED_PUBLIC_KEYS`.
4. Persist the tenant-scoped plugin row.
5. Record a lifecycle audit event.
Use `app.plugins.manage` for routes or CLI actions that install plugins.
### Grant
Capability and hook-consent grants are tenant-scoped. Manifest hook
subscriptions are granted with the synthetic `hook:<hook-name>` capability name,
so hook-only plugins cannot be enabled without explicit tenant-admin review.
The backend permission split exists and the tenant-admin route now requires
per-consent review checkboxes with a server-generated manifest digest, so stale
or unreviewed capability and hook sets are rejected before grants are written.
`app.plugins.use` intentionally covers using installed plugins and writing the
current user's non-secret plugin settings. `app.plugins.manage` remains the
tenant API/UI boundary for install, capability grant/revoke, enable/disable,
and future update operations. Uninstall is currently CLI/operator-only.
### Revoke
Tenant admins can revoke individual capability grants with `app.plugins.manage`.
The lifecycle service disables the plugin first when it is currently enabled,
then removes the tenant-scoped grant rows and records
`plugin.capabilities.revoked` with the revoked capability names and auto-disable
status. Re-enable only after the requested capabilities are reviewed again.
### Enable
Enable only after install, signature validation, capability review, and required worker services are live. Enabling records a lifecycle audit event.
### Disable
Disable when a plugin is no longer needed, fails review, exceeds crash thresholds, or needs emergency containment. Disabling should prevent new invocations from being enqueued or executed.
### Update
There is no production `plugins update` command and the database stores one
installed row per `(tenantId, pluginId)`, so same-plugin side-by-side versions
are not supported. Until a reviewed update lifecycle exists, upgrades are
operator-owned replace operations: disable the installed plugin, back up and
decide retention for plugin-owned data, uninstall through the CLI escape hatch,
then reinstall and re-grant the reviewed version. Version compatibility rules
live in [Plugin Versioning](../plugins/versioning.md).
### Uninstall
Uninstall is currently an operator CLI command, not a tenant API/UI action. It
removes the tenant plugin row, writes a lifecycle audit event, and cascades
owned plugin config where schema FKs define that behavior. Plugin-owned tables,
migrations, binaries, and external artifacts require an explicit backup/delete
decision before destructive cleanup.
## Hook Delivery
The web/API side emits `plugin.hook.*` outbox events. The general worker consumes those events and fans out matching subscribers into `toolctrl.plugin-invoke`. The plugin worker then executes the WASM hook.
Current foundation hooks:
- `tenant.created`
- `user.created`
- `entity.created`
- `entity.updated`
- `entity.deleted`
- `workflow.created`
- `workflow.updated`
- `workflow.deleted`
- `workflow.executed`
- `workflow.failed`
`tenant.created`, `user.created`, row-backed `entity.created`/`entity.updated`/`entity.deleted`, tenant-scoped service-path `workflow.created`/`workflow.updated`/`workflow.deleted`, tenant-scoped template-import `workflow.created`, and the terminal `workflow.executed` hook are currently producer-wired. Entity hooks are registered for the generic contract, but non-row entity producers still need explicit fire points before they are treated as covered. Workflow definition hooks intentionally emit metadata only; platform-scoped workflow definitions and definition writes that bypass the reviewed service/import paths are not covered.
Failed tenant-scoped web/API/manual and worker-native workflow executions
additionally emit `workflow.failed` with metadata-only failure details so
plugins can subscribe to failure handling without filtering the full
`workflow.executed` stream.
## Auto-Disable
The crash tracker counts failures per plugin and tenant in a sliding window. When the configured threshold is reached, the worker disables the tenant plugin and enqueues a `plugin.auto_disabled` audit event through `audit.event.create`, ideally in the same transaction.
Operator checks after auto-disable:
1. Inspect `AuditEvent` rows for `plugin.auto_disabled` (`sourceModule=worker.plugin-runtime`, `outcome=blocked`) and then lifecycle/capability-use rows.
2. Run `toolctrl-cli plugins verify-audit-chain <tenantId> [--pluginId <pluginId>]` to verify the tenant-wide `PluginCapabilityUse` hash chain. `--pluginId` reports matching rows but does not slice the tenant chain.
3. Inspect capability-use audit rows.
4. Inspect worker logs with `pluginId`, `tenantId`, and queue job id.
5. Confirm whether the failure is plugin code, manifest/capability mismatch, missing host binding, or host runtime error.
6. Re-enable only after the root cause is fixed or the plugin version is rolled back.
## Current Known Gaps
- Host capability bindings are not complete. The worker routes `log` and `audit` through the capability gateway; higher-risk capabilities still need dedicated security review and tests before they are exposed. In production, install, grant, and enable reject manifest capabilities without reviewed host bindings.
- Invocation audit persists one `PluginInvocation` row per worker job outcome and correlates capability-use rows by `pluginInvocationId`; raw plugin input/output payloads are intentionally not stored there.
- Tenant-admin capability consent has a guided review panel with explicit confirmation, capability scope review, stale-manifest digest checks, and enable gating.
- The reusable plugin settings iframe shell, CSP, and versioned postMessage
contract exist in `PluginSettingsFrame`; manifest `ui.settings.entry`
metadata is routed through the server-side bundle loader, and non-secret
per-user settings are stored in `PluginUserConfig.configJson` with
`plugin.configuration.updated` audit events. The frame CSP denies navigation,
the iframe `allow` policy denies browser capabilities, the host binds
postMessage traffic to a per-render `plugin-settings-channel` nonce, rejects
unexpected protocol fields before handlers run, and the host remounts the
sandbox if plugin content triggers an unexpected iframe load. Rejected
postMessage attempts and unexpected reloads create rate-limited,
client-reported `plugin.settings_frame.violation` audit events with redacted
metadata only; treat them as telemetry, not cryptographic proof of plugin
behavior. Secret-like keys remain blocked until a dedicated encrypted secret
binding is reviewed and shipped.
- Additional workflow plugin hooks beyond the current definition-metadata plus terminal-execution set are not complete.
- External WASM sandbox security review remains an external blocker for production enablement.
## Verification
After docs-only changes:
```sh
pnpm run verify:docs-links
pnpm run verify:production-docs
```
After runtime changes:
```sh
pnpm exec vitest run packages/plugin-runtime
pnpm --filter @toolctrl/worker build
pnpm run verify:deploy-workflow
pnpm run verify:docker-baseline
```