Update hivegrid-cloud reference to latest main

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
davidandClaude Fable 5 committed 2026-07-08 16:15:37 +02:00
1 parent e98d38e716
commit 7b21524c86
3781 files changed
+373484 -225607

No files matched your search

@@ -16,7 +16,7 @@ sequenceDiagram
Browser->>Web: GET /auth/admin/login
Web->>Web: createAdminSsoChallenge() (PKCE+state+nonce)
Web-->>Browser: 302 -> Authentik authorize, Set-Cookie: admin_sso_challenge
Web-->>Browser: 302 -> Authentik authorize, Set-Cookie: hivegrid_admin_sso_challenge
Browser->>Authentik: GET /application/o/authorize/...
Authentik-->>Browser: Login + MFA, then 302 /auth/admin/callback?code=...&state=...
Browser->>Web: GET /auth/admin/callback (with cookie)
@@ -33,7 +33,7 @@ sequenceDiagram
Web->>Web: readIdTokenFromRequest() + destroyAppSession()
Web-->>Browser: 302 Authentik end_session_endpoint?id_token_hint=...
Browser->>Authentik: end_session
Authentik-->>Browser: 302 post_logout_redirect_uri -> /admin/login?logged_out=1
Authentik-->>Browser: 302 post_logout_redirect_uri -> /login?logged_out=1
```
## Required environment
@@ -42,11 +42,11 @@ See `.env.example` for the canonical commented block. Coolify deployment:
| Key | Source |
| ---------------------------------------------- | ------------------------------------------------- |
| `AUTHENTIK_ISSUER_URL` | `https://auth.<base-domain>/application/o/toolctrl-admin-panel/` |
| `AUTHENTIK_ADMIN_CLIENT_ID` | Blueprint client_id (default `toolctrl-admin-panel`) |
| `AUTHENTIK_ISSUER_URL` | `https://auth.<base-domain>/application/o/hivegrid-admin-panel/` |
| `AUTHENTIK_ADMIN_CLIENT_ID` | Blueprint client_id (default `hivegrid-admin-panel`) |
| `AUTHENTIK_ADMIN_CLIENT_SECRET` | Authentik admin UI (Coolify vault) |
| `AUTHENTIK_ADMIN_REDIRECT_URI` | `https://admin.<base-domain>/auth/admin/callback` |
| `AUTHENTIK_ADMIN_POST_LOGOUT_REDIRECT_URI` | `https://admin.<base-domain>/admin/login?logged_out=1` |
| `AUTHENTIK_ADMIN_POST_LOGOUT_REDIRECT_URI` | `https://admin.<base-domain>/login?logged_out=1` |
| `ADMIN_FALLBACK_LOGIN_ENABLED` | `false` (default), flip to `true` during recovery |
| `ADMIN_FALLBACK_LOGIN_EMAILS` | Comma-separated bootstrap emails |
@@ -56,8 +56,8 @@ The OIDC `groups` claim drives the AdminUser role. Highest priority wins.
| Authentik group | Internal AdminUser role | Notes |
| ---------------- | ----------------------- | -------------------------------------- |
| `toolctrl-admin` | `super_admin` | Full admin permissions; if user is in both `toolctrl-admin` and `toolctrl-staff`, this wins. |
| `toolctrl-staff` | `staff` | Limited admin permissions. |
| `hivegrid-admin` | `super_admin` | Full admin permissions; if user is in both `hivegrid-admin` and `hivegrid-staff`, this wins. |
| `hivegrid-staff` | `staff` | Limited admin permissions. |
| _other_ | _login rejected (403)_ | `admin.login.sso.failure` audit event with `reason=forbidden-group`. |
Granular permission checks remain in `apps/web/app/utils/permissions/`;
@@ -66,16 +66,16 @@ the role is the coarse switch.
## Onboarding a new admin
1. Create the user in Authentik (manual or via SCIM/LDAP if integrated).
2. Add the user to the `toolctrl-admin` or `toolctrl-staff` group.
2. Add the user to the `hivegrid-admin` or `hivegrid-staff` group.
3. Enforce MFA enrollment in Authentik (provider-side).
4. User clicks "Sign in with SSO" on `https://admin.<domain>/admin/login`.
4. User clicks "Sign in with SSO" on `https://admin.<domain>/login`.
5. First login triggers JIT-provisioning — AdminUser row appears in DB
with `ssoProvider='authentik'`, `ssoSubjectId=<sub>`, `ssoGroups=[...]`,
`lastSsoLoginAt=<now>`.
## Offboarding an admin
1. **Authentik** — remove the user from `toolctrl-admin`/`toolctrl-staff`
1. **Authentik** — remove the user from `hivegrid-admin`/`hivegrid-staff`
or deactivate the user.
2. Next SSO login will be rejected (`admin.login.sso.failure` with
`reason=forbidden-group`).
@@ -107,7 +107,7 @@ Every fallback attempt — success or failure — writes an
ADMIN_FALLBACK_LOGIN_EMAILS=bootstrap@<base-domain>
```
Redeploy the web service.
3. Visit `https://admin.<base-domain>/admin/login` — expand the disclosure
3. Visit `https://admin.<base-domain>/login` — expand the disclosure
("Use emergency password login") and sign in with the bootstrap admin.
4. Resolve the Authentik outage (see `docs/operations/authentik-setup.md`).
5. **Immediately** revert the env flag back to `ADMIN_FALLBACK_LOGIN_ENABLED=false`
@@ -160,13 +160,13 @@ persisting to `AuditEvent`.
| Symptom | Cause / Fix |
| ------------------------------------------------------- | ------------------------------------------------------ |
| `/auth/admin/login` redirects to `/admin/login?error=sso_unavailable` | env vars missing or Authentik unreachable. Check Sentry for `AdminSsoConfigError` (`discovery-failed`). |
| `/auth/admin/login` redirects to `/login?error=sso_unavailable` | env vars missing or Authentik unreachable. Check Sentry for `AdminSsoConfigError` (`discovery-failed`). |
| `error=sso_state_invalid` after Authentik | Challenge cookie expired, signed under stale secret, or tampered. Have user retry; check `CRYPTO_SECRET` rotation. |
| `error=sso_forbidden_group` | User not in `toolctrl-admin`/`toolctrl-staff`. Add them in Authentik then retry. |
| `error=sso_forbidden_group` | User not in `hivegrid-admin`/`hivegrid-staff`. Add them in Authentik then retry. |
| `error=sso_email_not_verified` | Authentik user has not verified their email; enforce verification in Authentik provider settings. |
| `error=sso_token_exchange_failed` | Client secret mismatch, clock-skew, or redirect-URI drift. Compare `.env` vs Authentik UI. |
| JWKS-stale after key rotation | openid-client refreshes JWKS lazily; first failure forces a refetch. If repeated, restart the web container. |
| Group claim empty in ID-token | Authentik provider must include `groups` scope mapping — see blueprint `03-application-toolctrl-admin.yaml`. |
| Group claim empty in ID-token | Authentik provider must include `groups` scope mapping — see blueprint `03-application-hivegrid-admin.yaml`. |
## Rollback / kill-switch
@@ -36,7 +36,7 @@ external network so Traefik can route inbound HTTPS:
```
- **server** — Django web + REST API. Mounts the project blueprints from
`infra/authentik/bootstrap/blueprints/` at `/blueprints/toolctrl/`.
`infra/authentik/bootstrap/blueprints/` at `/blueprints/hivegrid/`.
- **worker** — Background jobs, blueprint application, email send. Shares
the same image.
- **postgresql** — Authentik's own Postgres 16. **Not shared** with the
@@ -115,8 +115,8 @@ scope for this task; same pattern as Sentry).
AUTHENTIK_POSTGRESQL__PASSWORD=$(openssl rand -base64 32)
AUTHENTIK_BOOTSTRAP_PASSWORD=$(openssl rand -base64 24)
AUTHENTIK_BOOTSTRAP_TOKEN=$(openssl rand -hex 32)
TOOLCTRL_ADMIN_OIDC_CLIENT_SECRET=$(openssl rand -hex 32)
AUTHENTIK_BOOTSTRAP_EMAIL=ops@toolctrl.de # operator's monitored mailbox
HIVEGRID_ADMIN_OIDC_CLIENT_SECRET=$(openssl rand -hex 32)
AUTHENTIK_BOOTSTRAP_EMAIL=ops@hivegrid.de # operator's monitored mailbox
AUTHENTIK_SENTRY__DSN=$(grep AUTHENTIK_DSN ~/.cli-tokens/boilerplate-staging-env | cut -d= -f2)
```
Save the four randomly-generated secrets to your offline password vault
@@ -125,7 +125,7 @@ scope for this task; same pattern as Sentry).
5. Click **Deploy**. First boot:
- postgresql + redis start, server waits for `service_healthy`
- server runs migrations (~30-60s)
- worker boots, discovers `/blueprints/toolctrl/*.yaml`, applies them
- worker boots, discovers `/blueprints/hivegrid/*.yaml`, applies them
- bootstrap admin is created from `AUTHENTIK_BOOTSTRAP_EMAIL` +
`AUTHENTIK_BOOTSTRAP_PASSWORD`
6. Once `docker ps` shows the server container `(healthy)`, run the
@@ -140,7 +140,7 @@ scope for this task; same pattern as Sentry).
7. Smoke-test from your laptop:
```bash
AUTHENTIK_BASE_URL=https://auth.boiler.saas-ctrl.com \
AUTHENTIK_APP_SLUG=toolctrl-admin-panel \
AUTHENTIK_APP_SLUG=hivegrid-admin-panel \
bash infra/authentik/scripts/smoke.sh
```
All five probes must pass (root 200, health-live 200, health-ready 200,
@@ -153,18 +153,18 @@ scope for this task; same pattern as Sentry).
1. Browse to <https://auth.boiler.saas-ctrl.com/> and log in with
`AUTHENTIK_BOOTSTRAP_EMAIL` / `AUTHENTIK_BOOTSTRAP_PASSWORD`.
2. Authentik forces a password change on first login. Set a new strong
password (meets the `toolctrl-password-policy`: 12+ chars, mixed case
password (meets the `hivegrid-password-policy`: 12+ chars, mixed case
+ digit + symbol, no HIBP hit, zxcvbn >= 3).
3. **Enroll TOTP**: User-Settings -> Authenticators -> Set up TOTP.
Scan the QR with the operator's hardware-MFA app (Yubikey-Authenticator
/ 1Password / Authy). Required because the
`toolctrl-require-mfa-totp-for-admin` policy denies access to the
`hivegrid-require-mfa-totp-for-admin` policy denies access to the
admin-panel application without it.
4. **Rotate the bootstrap token**: Directory -> Tokens -> find the bootstrap
token (look up by intent: `app_password` and the bootstrap-email user),
delete it. Create a personal admin token instead and store it in
Coolify-Vault under `AUTHENTIK_ADMIN_TOKEN` for future re-bootstrap runs.
5. **Add the bootstrap admin to the `toolctrl-admin` group** (it's NOT in
5. **Add the bootstrap admin to the `hivegrid-admin` group** (it's NOT in
the group by default — Authentik treats it as a built-in superuser; the
group binding is what the admin-panel OIDC app uses for authorisation).
@@ -178,7 +178,7 @@ scope for this task; same pattern as Sentry).
- [ ] Internal SSL OFF (TLS terminated by Traefik) — verify by
`curl -ksI https://auth.boiler.saas-ctrl.com/` shows
`server: Traefik` not `server: gunicorn`.
- [ ] MFA-TOTP policy applied to `toolctrl-admin` group — verified by
- [ ] MFA-TOTP policy applied to `hivegrid-admin` group — verified by
creating a test admin without TOTP and seeing the policy deny.
- [ ] Public self-registration off (`/if/flow/default-source-enrollment/`
returns 404 — covered by `scripts/smoke.sh`).
@@ -196,9 +196,9 @@ scope for this task; same pattern as Sentry).
- [ ] Backup container deployed + first nightly run verified
(`aws s3 ls s3://${AUTHENTIK_S3_BACKUP_BUCKET}/authentik/$(date +%Y-%m-%d)/`).
- [ ] Brand whitelisting: Authentik UI -> System -> Brands -> default brand
has `default flow background` set to a Toolctrl asset (or default OK).
- [ ] OIDC application access bindings reviewed: only `toolctrl-admin` and
`toolctrl-staff` allowed; no `everyone` binding.
has `default flow background` set to a HiveGrid asset (or default OK).
- [ ] OIDC application access bindings reviewed: only `hivegrid-admin` and
`hivegrid-staff` allowed; no `everyone` binding.
---
@@ -234,7 +234,7 @@ docker exec -i pg-restore pg_restore --no-owner --no-privileges --dbname authent
# 4. Verify schema + data
docker exec -i pg-restore psql -U postgres -d authentik -c "\dt" | head
docker exec -i pg-restore psql -U postgres -d authentik -c "SELECT COUNT(*) FROM authentik_core_user;"
docker exec -i pg-restore psql -U postgres -d authentik -c "SELECT name FROM authentik_core_group WHERE name LIKE 'toolctrl-%';"
docker exec -i pg-restore psql -U postgres -d authentik -c "SELECT name FROM authentik_core_group WHERE name LIKE 'hivegrid-%';"
# 5. Spin up Authentik server against the restored DB with the SAME secret
# key, hit /-/health/ready/ to confirm it boots cleanly.
@@ -295,7 +295,7 @@ dedicated host (smaller blast radius, separate maintenance window):
7. Once green, switch the admin-panel `AUTHENTIK_OIDC_ISSUER_URL` (#82) over
to the prod host with a maintenance window:
- Create the prod-equivalent OIDC application in the prod Authentik
- Onboard real admins (create users + assign to `toolctrl-admin`,
- Onboard real admins (create users + assign to `hivegrid-admin`,
force TOTP enrollment)
- Flip `AUTHENTIK_OIDC_ISSUER_URL` in the app's Coolify env
- Verify admin login end-to-end
@@ -316,7 +316,5 @@ dedicated host (smaller blast radius, separate maintenance window):
- `infra/authentik/bootstrap/blueprints/` — config-as-code
- `infra/authentik/backup/` — daily backup container
- `infra/authentik/scripts/smoke.sh` — post-deploy verifier
- Phase 09 detail-MD:
`docs/improvement-plan/09-authentik-admin-sso/authentik-setup.md`
- Lite alternative (LDAP-only):
`docs/operations/authentik-alternatives.md`
@@ -1,249 +0,0 @@
# Capacitor mobile wrapper — operations
Last updated: 2026-05-25 (mobile optimization sweep)
The toolctrl mobile app is a Capacitor 7 native wrapper around the
React Router v7 web app at `apps/web/`. There is no separate UI
codebase. This document covers local development, building for iOS /
Android, signing, and store submission.
## Quick reference
| Concern | Where |
|---|---|
| App workspace | `apps/mobile/` |
| Bundle id | `de.toolctrl.app` |
| Capacitor config | `apps/mobile/capacitor.config.ts` |
| Push backend worker | `apps/worker/src/jobs/push-notification.ts` |
| Device-token API | `apps/api/src/routes/devices-routes.ts` (`POST /v1/devices/register`) |
| Prisma model | `prisma/schema/notifications.prisma` → `model DeviceToken` |
| Sentry init | `apps/mobile/src/sentry.ts` |
## Status: code-complete, native builds and store submission deferred
The Phase 08 #65 scope ships:
- The `apps/mobile/` workspace skeleton + Capacitor 7 config.
- Hand-scaffolded iOS + Android native projects under
`apps/mobile/{ios,android}/`. An operator with Xcode / Android Studio
installed should re-run `npx cap add ios && npx cap add android` once
to overwrite the skeletons with the canonical CLI output.
- DeviceToken Prisma model + `POST /v1/devices/register` upsert endpoint.
- `dispatchPushNotification` worker orchestrator with APNs + FCM seams
(real SDK adapters wire in operator-side at deployment).
- `@sentry/capacitor` init mirroring the web client config.
**Explicitly deferred** (operator-owned):
- App-Store and Play-Store submission flows (screenshots, review notes,
release marketing).
- iOS signing certificate + provisioning profile management.
- Android keystore generation + native encrypted storage for long-lived tokens.
- `google-services.json` + APNs `.p8` provisioning.
- CI pipelines (Codemagic / GitHub Actions macOS-runner / Play upload).
## Local development
### Prerequisites
- macOS for iOS (Xcode 15+, CocoaPods, an Apple Developer team)
- Any OS for Android (Android Studio Hedgehog+, Android SDK 34, JDK 17)
- pnpm 10+, Node 22+
### Initial setup
```bash
# 1. Install workspace deps (from repo root)
pnpm install
# 2. Build the web bundle that the wrapper loads
pnpm --filter @toolctrl/mobile build:web
# 3. Sync web assets + plugins into the native projects
pnpm --filter @toolctrl/mobile sync
# 4. Open in the platform IDE
pnpm --filter @toolctrl/mobile open:ios # Xcode
pnpm --filter @toolctrl/mobile open:android # Android Studio
```
### Live-reload dev loop
For tight iteration cycles, point the wrapper at a running web dev
origin instead of the bundled web assets:
```bash
# iOS simulator can reach the host dev server through localhost.
export MOBILE_LIVE_RELOAD_URL=http://localhost:3000
# Android emulator maps 10.0.2.2 to the host loopback instead.
# export MOBILE_LIVE_RELOAD_URL=http://10.0.2.2:3000
pnpm --filter @toolctrl/mobile sync
pnpm --filter @toolctrl/mobile open:ios # or open:android
```
The Capacitor config emits `server.url` only when `MOBILE_LIVE_RELOAD_URL`
is set. It must be an absolute `http://` or `https://` URL. Production
builds run with the variable unset so the bundle loads the embedded
`webDir`.
Use `http://localhost:3000` for the iOS simulator, `http://10.0.2.2:3000`
for the Android emulator, and HTTPS staging/tunnel URLs for physical
devices. The committed Android skeleton only permits cleartext live reload
for `localhost` and `10.0.2.2`; do not broaden cleartext allowances in
release builds.
### Resetting the native projects
When the canonical Capacitor CLI is needed (signing config, new plugin,
platform upgrade):
```bash
cd apps/mobile
rm -rf ios android
npx cap add ios
npx cap add android
npx cap sync
```
Bundle id, app name and webDir come from `capacitor.config.ts`, so
re-creation is deterministic.
## Building iOS
```bash
cd apps/mobile/ios/App
pod install
xcodebuild \
-workspace App.xcworkspace \
-scheme App \
-configuration Release \
-archivePath build/App.xcarchive \
archive
xcodebuild \
-exportArchive \
-archivePath build/App.xcarchive \
-exportOptionsPlist ExportOptions.plist \
-exportPath build/ipa
```
`ExportOptions.plist` is operator-provided (not committed). Upload to
TestFlight via Transporter.app or `xcrun altool --upload-app`.
## Building Android
```bash
cd apps/mobile/android
./gradlew bundleRelease
# AAB lands at app/build/outputs/bundle/release/app-release.aab
```
Signing requires `signingConfigs.release` in `app/build.gradle` and a
keystore. The committed skeleton does not declare a release signing
config — operator must add one before the first release build.
## Push notifications
### Backend
The push job is implemented as a pure orchestrator
(`apps/worker/src/jobs/push-notification.ts`) with three injected
provider adapters:
- `sendApns(token, payload)` → `node-apn` / `@parse/node-apn` against the
PEM key configured in `APNS_KEY` / `APNS_KEY_ID` / `APNS_TEAM_ID`.
- `sendFcm(token, payload)` → `firebase-admin` using credentials from
`FCM_SERVICE_ACCOUNT_JSON`.
- `sendWebPush(token, payload)` → optional `web-push` adapter for the
PWA fallback path.
The adapters live in `apps/worker/src/index.ts` (operator-side
implementation, kept out of the orchestrator so unit tests stay
SDK-free). When a provider returns one of the canonical revocation
codes (`Unregistered`, `InvalidRegistration`, `BadDeviceToken`,
`messaging/registration-token-not-registered`,
`messaging/invalid-registration-token`), the corresponding DeviceToken
row is soft-deleted via `revokedAt`.
### Provider credentials
| Env var | Source | Notes |
|---|---|---|
| `FCM_SERVICE_ACCOUNT_JSON` | Firebase console → Project settings → Service accounts → Generate new private key | Either raw JSON or base64-encoded JSON |
| `APNS_KEY` | Apple Developer → Certificates, Identifiers & Profiles → Keys → Create `.p8` | PKCS#8 PEM block from the downloaded `.p8` file (the standard `BEGIN PRIVATE KEY` framing). |
| `APNS_KEY_ID` | Apple Developer (10 chars) | Shown alongside the `.p8` |
| `APNS_TEAM_ID` | Apple Developer → Account → Membership (10 chars) | |
| `APNS_BUNDLE_ID` | Defaults to `de.toolctrl.app` | Must match `capacitor.config.ts` |
**Rotate APNs keys every 12 months.** Apple does not auto-expire `.p8`
files but rotation contains compromise blast-radius.
### Device-token API
The Capacitor shell calls `POST /v1/devices/register` after the native
push plugin yields a token:
```json
POST /v1/devices/register
Authorization: Bearer <service-token>
Content-Type: application/json
{
"userId": "u-1",
"tenantId": "t-1",
"platform": "ios",
"token": "<APNs / FCM / WebPush token>",
"pushProvider": "apns",
"appVersion": "1.0.0"
}
```
The endpoint upserts by `token` (globally unique) so cold-start
re-registration produces zero duplicates. Logout calls
`POST /v1/devices/revoke` with `{ "token": "..." }`.
## App-Store / Play-Store submission
Both flows are **operator-owned** and deferred from this scaffold.
High-level checklists:
### iOS
1. Apple Developer account active (USD 99/year).
2. App Store Connect record created with `de.toolctrl.app` bundle id.
3. Push certificate / Auth Key (`.p8`) generated.
4. Privacy Policy URL published at `/legal/privacy`.
5. App Privacy questionnaire answered (we collect: contact info,
identifiers, usage data; we do NOT collect: precise location,
financial info, health data).
6. Screenshots (6.7" iPhone 15 Pro Max, 5.5" iPhone 8 Plus, optionally
12.9" iPad Pro).
7. Review notes with demo credentials.
8. Submit via TestFlight → "Submit for Review".
### Android
1. Google Play Developer account active (USD 25 one-time).
2. Play Console app created with `de.toolctrl.app` application id.
3. Service account JSON for Play API uploads.
4. Privacy Policy URL.
5. Data safety form (same answers as iOS).
6. Screenshots (phone 9:16 + tablet 16:10).
7. Internal testing → Production rollout (staged 5% → 25% → 100%).
## Rollback
- Revert the `08/capacitor:` commit range to drop the workspace.
- Disable push fan-out by flipping `mobile.push.enabled` feature flag
off (the dispatcher short-circuits without touching DeviceToken rows).
- Roll back the Prisma migration `20260517200000_08_capacitor_device_tokens`
with `prisma migrate resolve --rolled-back` + a manual `DROP TABLE`
if the schema needs to revert.
## See also
- [`docs/improvement-plan/08-mobile-cli-storybook-sdk/capacitor.md`](../improvement-plan/08-mobile-cli-storybook-sdk/capacitor.md)
- [`apps/mobile/README.md`](../../apps/mobile/README.md)
- [`apps/mobile/ios/README.md`](../../apps/mobile/ios/README.md)
- [`apps/mobile/android/README.md`](../../apps/mobile/android/README.md)
@@ -44,7 +44,7 @@ Procedure:
pg_dump --format=custom --no-owner --file=pre-rollback-$(date -u +%Y%m%dT%H%M%SZ).dump "$DATABASE_URL"
# 2. Take the application offline (Coolify -> scale to 0 / put the service in maintenance).
# TOOLCTRL_API_MAINTENANCE_MODE=1 is available for the API surface; the web surface relies on the deployment platform's maintenance toggle.
# HIVEGRID_API_MAINTENANCE_MODE=1 is available for the API surface; the web surface relies on the deployment platform's maintenance toggle.
# 3. Apply the inverse migration that was prepared and committed beforehand.
pnpm exec prisma migrate deploy
@@ -14,7 +14,7 @@ This runbook describes how operators install, grant, enable, disable, uninstall,
Provision the plugin worker service now so staging and future production
topology stay identical. In production, a `worker-plugin` service with
`PLUGIN_RUNTIME_PRODUCTION_ENABLED` unset or `false` starts idle and does not
consume `toolctrl.plugin-invoke`; readiness treats the production plugin runtime
consume `hivegrid.plugin-invoke`; readiness treats the production plugin runtime
as skipped in that gated state. After the external WASM sandbox signoff,
`PLUGIN_RUNTIME_PRODUCTION_ENABLED=true` makes plugin readiness active on both
`worker-general` and `worker-plugin`, requires `PLUGIN_TRUSTED_PUBLIC_KEYS`, and
@@ -22,7 +22,7 @@ allows the plugin worker to consume invocation jobs.
- General worker group: consumes outbox fanout, including `outbox.plugin-hook`.
- Plugin worker group: runs sandboxed plugin invocations with `WORKER_QUEUE_GROUP=plugin`.
- Redis/BullMQ queue: `toolctrl.plugin-invoke`.
- Redis/BullMQ queue: `hivegrid.plugin-invoke`.
- Postgres: stores installed plugin rows, per-user plugin config, capability use audit, settings-change audit, and lifecycle audit events.
Do not run plugin invocation in the web container. Keep plugin execution isolated from tenant web traffic so sandbox stalls, plugin traps, and crash tracking do not block core app requests.
@@ -37,7 +37,7 @@ plugin-specific gates and operator actions.
| `PLUGIN_RUNTIME_PRODUCTION_ENABLED` | Production rollout gate | Must be `true` in production before install, grant, enable, hook fanout, or plugin invoke workers run. Keep unset/false until external sign-off. |
| `PLUGIN_HOOK_FANOUT_REQUIRED` | General-worker fanout fail-closed switch | Optional for staging/canaries. In production with `PLUGIN_RUNTIME_PRODUCTION_ENABLED=true` and Redis configured, hook fanout is required even when this is unset. |
| `PLUGIN_TRUSTED_PUBLIC_KEYS` | CSV of trusted Ed25519 public keys | Empty means plugin signature verification fails closed. |
| `WORKER_QUEUE_GROUP=plugin` | Starts the isolated plugin worker service | Required for `toolctrl.plugin-invoke` jobs. In production before runtime signoff it starts idle without queue consumers; after `PLUGIN_RUNTIME_PRODUCTION_ENABLED=true`, startup fails closed if invocation runtime dependencies cannot bootstrap. |
| `WORKER_QUEUE_GROUP=plugin` | Starts the isolated plugin worker service | Required for `hivegrid.plugin-invoke` jobs. In production before runtime signoff it starts idle without queue consumers; after `PLUGIN_RUNTIME_PRODUCTION_ENABLED=true`, startup fails closed if invocation runtime dependencies cannot bootstrap. |
| `WORKER_CONCURRENCY_PLUGIN` | Plugin worker concurrency | Defaults to the worker runtime value. Tune separately from mail/general workers. |
| `PLUGIN_HOOK_FANOUT_ENABLED` | General-worker plugin hook fanout toggle | Leave unset/true when production runtime is enabled; explicit `false` makes readiness fail. |
| `PLUGIN_CAPABILITY_USE_ENABLED` | General-worker capability-use persistence toggle | Leave unset/true when production runtime is enabled; explicit `false` makes readiness fail. |
@@ -60,7 +60,7 @@ Higher-risk host-binding resource controls remain runtime follow-up work. Do not
| Install / grant / revoke / enable / disable lifecycle | Implemented with tenant scope, permissions, audit events, JSON request validation, and capability review digest checks. | External WASM sandbox signoff plus `PLUGIN_RUNTIME_PRODUCTION_ENABLED=true` in the target environment. |
| Uninstall lifecycle | CLI/operator-only escape hatch that deletes the tenant `Plugin` row and writes an audit event. It is not a tenant API/UI action. | Operator backup/delete decision for plugin-owned data plus the external runtime signoff gate. |
| Settings iframe | Implemented with sandboxed iframe, restrictive CSP including `navigate-to 'none'`, denied iframe `allow` permissions, parent-side unexpected-navigation remount, channel-bound strict-schema versioned `plugin.settings.*` postMessage contract, and non-secret per-user config persistence. | Keep secret-like settings blocked until encrypted secret storage and host binding review land. |
| Hook fanout | Implemented for the foundation hook set through `outbox.plugin-hook` on `worker-general` and `toolctrl.plugin-invoke` on `worker-plugin`. | Add payload/test coverage before enabling new hook families such as billing, auth, or richer workflow hooks. |
| Hook fanout | Implemented for the foundation hook set through `outbox.plugin-hook` on `worker-general` and `hivegrid.plugin-invoke` on `worker-plugin`. | Add payload/test coverage before enabling new hook families such as billing, auth, or richer workflow hooks. |
| Safe host bindings | `log` and `audit` are gateway-controlled and audited. | Confirm trusted public keys and worker-plugin health in the deployment runbook. |
| Deferred host bindings | `http.outbound`, `secret.read`, `db.read`, `db.write`, `storage.read`, `storage.write`, and `tenant.info` intentionally fail closed. Future `http.outbound`, `secret.read`, and `db.read`/`db.write` calls already have runtime per-call scope guards for exact HTTPS host, key-pattern, and table-prefix contracts, but the bindings are not exposed yet. | Dedicated security review, SSRF/tenant-boundary tests, tenant-boundary/storage tests, secret redaction, audit contracts, and operator copy for each binding. |
@@ -133,7 +133,7 @@ decision before destructive cleanup.
## Hook Delivery
The web/API side emits `plugin.hook.*` outbox events. The general worker consumes those events and fans out matching subscribers into `toolctrl.plugin-invoke`. The plugin worker then executes the WASM hook.
The web/API side emits `plugin.hook.*` outbox events. The general worker consumes those events and fans out matching subscribers into `hivegrid.plugin-invoke`. The plugin worker then executes the WASM hook.
Current foundation hooks:
@@ -161,7 +161,7 @@ The crash tracker counts failures per plugin and tenant in a sliding window. Whe
Operator checks after auto-disable:
1. Inspect `AuditEvent` rows for `plugin.auto_disabled` (`sourceModule=worker.plugin-runtime`, `outcome=blocked`) and then lifecycle/capability-use rows.
2. Run `toolctrl-cli plugins verify-audit-chain <tenantId> [--pluginId <pluginId>]` to verify the tenant-wide `PluginCapabilityUse` hash chain. `--pluginId` reports matching rows but does not slice the tenant chain.
2. Run `hivegrid-cli plugins verify-audit-chain <tenantId> [--pluginId <pluginId>]` to verify the tenant-wide `PluginCapabilityUse` hash chain. `--pluginId` reports matching rows but does not slice the tenant chain.
3. Inspect capability-use audit rows.
4. Inspect worker logs with `pluginId`, `tenantId`, and queue job id.
5. Confirm whether the failure is plugin code, manifest/capability mismatch, missing host binding, or host runtime error.
@@ -202,7 +202,7 @@ After runtime changes:
```sh
pnpm exec vitest run packages/plugin-runtime
pnpm --filter @toolctrl/worker build
pnpm --filter @hivegrid/worker build
pnpm run verify:deploy-workflow
pnpm run verify:docker-baseline
```