209 lines
15 KiB
Markdown
209 lines
15 KiB
Markdown
# Plugin Lifecycle Operations
|
|
|
|
> **Runtime gate:** Use this runbook for staging/foundation operations until
|
|
> the external WASM sandbox security review is signed off in
|
|
> [Plugin Runtime Signoff](../security/plugin-runtime-signoff.md). Production
|
|
> plugin enablement remains blocked before that sign-off.
|
|
|
|
## Purpose
|
|
|
|
This runbook describes how operators install, grant, enable, disable, uninstall, and investigate plugins. It complements the product overview in [Plugins](../features/plugins.md) and the architecture docs in [Plugin system](../architecture/plugin-system.md) and [Plugin Hooks & Lifecycle](../architecture/plugin-hooks.md). Plugin upgrades are not a first-class lifecycle yet; treat them as reviewed replace operations until an explicit update command and data migration contract exist.
|
|
|
|
## Runtime Topology
|
|
|
|
Provision the plugin worker service now so staging and future production
|
|
topology stay identical. In production, a `worker-plugin` service with
|
|
`PLUGIN_RUNTIME_PRODUCTION_ENABLED` unset or `false` starts idle and does not
|
|
consume `toolctrl.plugin-invoke`; readiness treats the production plugin runtime
|
|
as skipped in that gated state. After the external WASM sandbox signoff,
|
|
`PLUGIN_RUNTIME_PRODUCTION_ENABLED=true` makes plugin readiness active on both
|
|
`worker-general` and `worker-plugin`, requires `PLUGIN_TRUSTED_PUBLIC_KEYS`, and
|
|
allows the plugin worker to consume invocation jobs.
|
|
|
|
- General worker group: consumes outbox fanout, including `outbox.plugin-hook`.
|
|
- Plugin worker group: runs sandboxed plugin invocations with `WORKER_QUEUE_GROUP=plugin`.
|
|
- Redis/BullMQ queue: `toolctrl.plugin-invoke`.
|
|
- Postgres: stores installed plugin rows, per-user plugin config, capability use audit, settings-change audit, and lifecycle audit events.
|
|
|
|
Do not run plugin invocation in the web container. Keep plugin execution isolated from tenant web traffic so sandbox stalls, plugin traps, and crash tracking do not block core app requests.
|
|
The canonical worker app, queue-group, port, and scaling matrix lives in
|
|
[Worker queue groups](../ops/worker-queues.md); keep this runbook focused on
|
|
plugin-specific gates and operator actions.
|
|
|
|
## Required Configuration
|
|
|
|
| Variable | Purpose | Notes |
|
|
| ---------------------------------- | ----------------------------------------- | -------------------------------------------------------------------------------- |
|
|
| `PLUGIN_RUNTIME_PRODUCTION_ENABLED` | Production rollout gate | Must be `true` in production before install, grant, enable, hook fanout, or plugin invoke workers run. Keep unset/false until external sign-off. |
|
|
| `PLUGIN_HOOK_FANOUT_REQUIRED` | General-worker fanout fail-closed switch | Optional for staging/canaries. In production with `PLUGIN_RUNTIME_PRODUCTION_ENABLED=true` and Redis configured, hook fanout is required even when this is unset. |
|
|
| `PLUGIN_TRUSTED_PUBLIC_KEYS` | CSV of trusted Ed25519 public keys | Empty means plugin signature verification fails closed. |
|
|
| `WORKER_QUEUE_GROUP=plugin` | Starts the isolated plugin worker service | Required for `toolctrl.plugin-invoke` jobs. In production before runtime signoff it starts idle without queue consumers; after `PLUGIN_RUNTIME_PRODUCTION_ENABLED=true`, startup fails closed if invocation runtime dependencies cannot bootstrap. |
|
|
| `WORKER_CONCURRENCY_PLUGIN` | Plugin worker concurrency | Defaults to the worker runtime value. Tune separately from mail/general workers. |
|
|
| `PLUGIN_HOOK_FANOUT_ENABLED` | General-worker plugin hook fanout toggle | Leave unset/true when production runtime is enabled; explicit `false` makes readiness fail. |
|
|
| `PLUGIN_CAPABILITY_USE_ENABLED` | General-worker capability-use persistence toggle | Leave unset/true when production runtime is enabled; explicit `false` makes readiness fail. |
|
|
| `PLUGIN_CAPABILITY_USE_CONCURRENCY` | Capability-use audit sink concurrency | Defaults to `1` so per-tenant capability-use hash chains stay linear. Raise only after adding store-level serialization or advisory locking. |
|
|
| `PLUGIN_BINARY_BASE_PATH` | Plugin binary lookup root | Defaults to `/app/var/plugins`; required to contain verified `<pluginId>/<version>/plugin.wasm` files before install/invoke checks can pass. |
|
|
| `PLUGIN_MAX_FUEL` | Sandbox fuel limit | Defaults to 1000000. Charged for the hook call plus capability gateway calls. |
|
|
| `PLUGIN_MAX_MEMORY_MB` | Sandbox memory cap | Defaults to 64 MiB. |
|
|
| `PLUGIN_MAX_WALL_CLOCK_MS` | Sandbox wall-clock kill budget | Defaults to 5000 ms; the plugin sandbox worker thread is terminated on timeout. |
|
|
| `PLUGIN_RATE_LIMIT_PER_MINUTE` | Per-tenant/plugin invocation rate limit | Defaults to 100. |
|
|
| `PLUGIN_RATE_LIMIT_WINDOW_SECONDS` | Rate-limit window | Defaults to 60. |
|
|
| `PLUGIN_AUTO_DISABLE_THRESHOLD` | Tenant plugin crash threshold | Default is 10. |
|
|
| `PLUGIN_AUTO_DISABLE_WINDOW_MS` | Sliding crash window | Default is 600000. |
|
|
|
|
Higher-risk host-binding resource controls remain runtime follow-up work. Do not document additional knobs as production-enforced without checking the worker bootstrap first.
|
|
|
|
## Production Readiness Matrix
|
|
|
|
| Capability slice | Operational status | Required before production use |
|
|
| ---------------- | ------------------ | ------------------------------ |
|
|
| Install / grant / revoke / enable / disable lifecycle | Implemented with tenant scope, permissions, audit events, JSON request validation, and capability review digest checks. | External WASM sandbox signoff plus `PLUGIN_RUNTIME_PRODUCTION_ENABLED=true` in the target environment. |
|
|
| Uninstall lifecycle | CLI/operator-only escape hatch that deletes the tenant `Plugin` row and writes an audit event. It is not a tenant API/UI action. | Operator backup/delete decision for plugin-owned data plus the external runtime signoff gate. |
|
|
| Settings iframe | Implemented with sandboxed iframe, restrictive CSP including `navigate-to 'none'`, denied iframe `allow` permissions, parent-side unexpected-navigation remount, channel-bound strict-schema versioned `plugin.settings.*` postMessage contract, and non-secret per-user config persistence. | Keep secret-like settings blocked until encrypted secret storage and host binding review land. |
|
|
| Hook fanout | Implemented for the foundation hook set through `outbox.plugin-hook` on `worker-general` and `toolctrl.plugin-invoke` on `worker-plugin`. | Add payload/test coverage before enabling new hook families such as billing, auth, or richer workflow hooks. |
|
|
| Safe host bindings | `log` and `audit` are gateway-controlled and audited. | Confirm trusted public keys and worker-plugin health in the deployment runbook. |
|
|
| Deferred host bindings | `http.outbound`, `secret.read`, `db.read`, `db.write`, `storage.read`, `storage.write`, and `tenant.info` intentionally fail closed. Future `http.outbound`, `secret.read`, and `db.read`/`db.write` calls already have runtime per-call scope guards for exact HTTPS host, key-pattern, and table-prefix contracts, but the bindings are not exposed yet. | Dedicated security review, SSRF/tenant-boundary tests, tenant-boundary/storage tests, secret redaction, audit contracts, and operator copy for each binding. |
|
|
|
|
## Lifecycle
|
|
|
|
Lifecycle API mutations require JSON requests. Install, grant, revoke, enable,
|
|
and disable calls to the tenant plugin API must send
|
|
`Content-Type: application/json`; malformed JSON returns `400 invalid_json`,
|
|
and missing or unsupported media types return `415 unsupported_media_type`.
|
|
Enable and disable have no fields, but API clients still send `{}` as the JSON
|
|
body so the lifecycle route keeps one validation contract.
|
|
|
|
### Install
|
|
|
|
1. Verify manifest schema and `schemaVersion`.
|
|
2. Verify `.wasm` SHA-256 against the manifest.
|
|
3. Verify signature against `PLUGIN_TRUSTED_PUBLIC_KEYS`.
|
|
4. Persist the tenant-scoped plugin row.
|
|
5. Record a lifecycle audit event.
|
|
|
|
Use `app.plugins.manage` for routes or CLI actions that install plugins.
|
|
|
|
### Grant
|
|
|
|
Capability and hook-consent grants are tenant-scoped. Manifest hook
|
|
subscriptions are granted with the synthetic `hook:<hook-name>` capability name,
|
|
so hook-only plugins cannot be enabled without explicit tenant-admin review.
|
|
The backend permission split exists and the tenant-admin route now requires
|
|
per-consent review checkboxes with a server-generated manifest digest, so stale
|
|
or unreviewed capability and hook sets are rejected before grants are written.
|
|
|
|
`app.plugins.use` intentionally covers using installed plugins and writing the
|
|
current user's non-secret plugin settings. `app.plugins.manage` remains the
|
|
tenant API/UI boundary for install, capability grant/revoke, enable/disable,
|
|
and future update operations. Uninstall is currently CLI/operator-only.
|
|
|
|
### Revoke
|
|
|
|
Tenant admins can revoke individual capability grants with `app.plugins.manage`.
|
|
The lifecycle service disables the plugin first when it is currently enabled,
|
|
then removes the tenant-scoped grant rows and records
|
|
`plugin.capabilities.revoked` with the revoked capability names and auto-disable
|
|
status. Re-enable only after the requested capabilities are reviewed again.
|
|
|
|
### Enable
|
|
|
|
Enable only after install, signature validation, capability review, and required worker services are live. Enabling records a lifecycle audit event.
|
|
|
|
### Disable
|
|
|
|
Disable when a plugin is no longer needed, fails review, exceeds crash thresholds, or needs emergency containment. Disabling should prevent new invocations from being enqueued or executed.
|
|
|
|
### Update
|
|
|
|
There is no production `plugins update` command and the database stores one
|
|
installed row per `(tenantId, pluginId)`, so same-plugin side-by-side versions
|
|
are not supported. Until a reviewed update lifecycle exists, upgrades are
|
|
operator-owned replace operations: disable the installed plugin, back up and
|
|
decide retention for plugin-owned data, uninstall through the CLI escape hatch,
|
|
then reinstall and re-grant the reviewed version. Version compatibility rules
|
|
live in [Plugin Versioning](../plugins/versioning.md).
|
|
|
|
### Uninstall
|
|
|
|
Uninstall is currently an operator CLI command, not a tenant API/UI action. It
|
|
removes the tenant plugin row, writes a lifecycle audit event, and cascades
|
|
owned plugin config where schema FKs define that behavior. Plugin-owned tables,
|
|
migrations, binaries, and external artifacts require an explicit backup/delete
|
|
decision before destructive cleanup.
|
|
|
|
## Hook Delivery
|
|
|
|
The web/API side emits `plugin.hook.*` outbox events. The general worker consumes those events and fans out matching subscribers into `toolctrl.plugin-invoke`. The plugin worker then executes the WASM hook.
|
|
|
|
Current foundation hooks:
|
|
|
|
- `tenant.created`
|
|
- `user.created`
|
|
- `entity.created`
|
|
- `entity.updated`
|
|
- `entity.deleted`
|
|
- `workflow.created`
|
|
- `workflow.updated`
|
|
- `workflow.deleted`
|
|
- `workflow.executed`
|
|
- `workflow.failed`
|
|
|
|
`tenant.created`, `user.created`, row-backed `entity.created`/`entity.updated`/`entity.deleted`, tenant-scoped service-path `workflow.created`/`workflow.updated`/`workflow.deleted`, tenant-scoped template-import `workflow.created`, and the terminal `workflow.executed` hook are currently producer-wired. Entity hooks are registered for the generic contract, but non-row entity producers still need explicit fire points before they are treated as covered. Workflow definition hooks intentionally emit metadata only; platform-scoped workflow definitions and definition writes that bypass the reviewed service/import paths are not covered.
|
|
Failed tenant-scoped web/API/manual and worker-native workflow executions
|
|
additionally emit `workflow.failed` with metadata-only failure details so
|
|
plugins can subscribe to failure handling without filtering the full
|
|
`workflow.executed` stream.
|
|
|
|
## Auto-Disable
|
|
|
|
The crash tracker counts failures per plugin and tenant in a sliding window. When the configured threshold is reached, the worker disables the tenant plugin and enqueues a `plugin.auto_disabled` audit event through `audit.event.create`, ideally in the same transaction.
|
|
|
|
Operator checks after auto-disable:
|
|
|
|
1. Inspect `AuditEvent` rows for `plugin.auto_disabled` (`sourceModule=worker.plugin-runtime`, `outcome=blocked`) and then lifecycle/capability-use rows.
|
|
2. Run `toolctrl-cli plugins verify-audit-chain <tenantId> [--pluginId <pluginId>]` to verify the tenant-wide `PluginCapabilityUse` hash chain. `--pluginId` reports matching rows but does not slice the tenant chain.
|
|
3. Inspect capability-use audit rows.
|
|
4. Inspect worker logs with `pluginId`, `tenantId`, and queue job id.
|
|
5. Confirm whether the failure is plugin code, manifest/capability mismatch, missing host binding, or host runtime error.
|
|
6. Re-enable only after the root cause is fixed or the plugin version is rolled back.
|
|
|
|
## Current Known Gaps
|
|
|
|
- Host capability bindings are not complete. The worker routes `log` and `audit` through the capability gateway; higher-risk capabilities still need dedicated security review and tests before they are exposed. In production, install, grant, and enable reject manifest capabilities without reviewed host bindings.
|
|
- Invocation audit persists one `PluginInvocation` row per worker job outcome and correlates capability-use rows by `pluginInvocationId`; raw plugin input/output payloads are intentionally not stored there.
|
|
- Tenant-admin capability consent has a guided review panel with explicit confirmation, capability scope review, stale-manifest digest checks, and enable gating.
|
|
- The reusable plugin settings iframe shell, CSP, and versioned postMessage
|
|
contract exist in `PluginSettingsFrame`; manifest `ui.settings.entry`
|
|
metadata is routed through the server-side bundle loader, and non-secret
|
|
per-user settings are stored in `PluginUserConfig.configJson` with
|
|
`plugin.configuration.updated` audit events. The frame CSP denies navigation,
|
|
the iframe `allow` policy denies browser capabilities, the host binds
|
|
postMessage traffic to a per-render `plugin-settings-channel` nonce, rejects
|
|
unexpected protocol fields before handlers run, and the host remounts the
|
|
sandbox if plugin content triggers an unexpected iframe load. Rejected
|
|
postMessage attempts and unexpected reloads create rate-limited,
|
|
client-reported `plugin.settings_frame.violation` audit events with redacted
|
|
metadata only; treat them as telemetry, not cryptographic proof of plugin
|
|
behavior. Secret-like keys remain blocked until a dedicated encrypted secret
|
|
binding is reviewed and shipped.
|
|
- Additional workflow plugin hooks beyond the current definition-metadata plus terminal-execution set are not complete.
|
|
- External WASM sandbox security review remains an external blocker for production enablement.
|
|
|
|
## Verification
|
|
|
|
After docs-only changes:
|
|
|
|
```sh
|
|
pnpm run verify:docs-links
|
|
pnpm run verify:production-docs
|
|
```
|
|
|
|
After runtime changes:
|
|
|
|
```sh
|
|
pnpm exec vitest run packages/plugin-runtime
|
|
pnpm --filter @toolctrl/worker build
|
|
pnpm run verify:deploy-workflow
|
|
pnpm run verify:docker-baseline
|
|
```
|