HiveGrid-Wireframes/03 Quellen und Referenzen/hivegrid-cloud/docs/operations/plugin-lifecycle.md

Plugin Lifecycle Operations

Runtime gate: Use this runbook for staging/foundation operations until the external WASM sandbox security review is signed off in Plugin Runtime Signoff. Production plugin enablement remains blocked before that sign-off.

Purpose

This runbook describes how operators install, grant, enable, disable, uninstall, and investigate plugins. It complements the product overview in Plugins and the architecture docs in Plugin system and Plugin Hooks & Lifecycle. Plugin upgrades are not a first-class lifecycle yet; treat them as reviewed replace operations until an explicit update command and data migration contract exist.

Runtime Topology

Provision the plugin worker service now so staging and future production topology stay identical. In production, a worker-plugin service with PLUGIN_RUNTIME_PRODUCTION_ENABLED unset or false starts idle and does not consume hivegrid.plugin-invoke; readiness treats the production plugin runtime as skipped in that gated state. After the external WASM sandbox signoff, PLUGIN_RUNTIME_PRODUCTION_ENABLED=true makes plugin readiness active on both worker-general and worker-plugin, requires PLUGIN_TRUSTED_PUBLIC_KEYS, and allows the plugin worker to consume invocation jobs.

• General worker group: consumes outbox fanout, including outbox.plugin-hook.
• Plugin worker group: runs sandboxed plugin invocations with WORKER_QUEUE_GROUP=plugin.
• Redis/BullMQ queue: hivegrid.plugin-invoke.
• Postgres: stores installed plugin rows, per-user plugin config, capability use audit, settings-change audit, and lifecycle audit events.

Do not run plugin invocation in the web container. Keep plugin execution isolated from tenant web traffic so sandbox stalls, plugin traps, and crash tracking do not block core app requests. The canonical worker app, queue-group, port, and scaling matrix lives in Worker queue groups; keep this runbook focused on plugin-specific gates and operator actions.

Required Configuration

Variable	Purpose	Notes
PLUGIN_RUNTIME_PRODUCTION_ENABLED	Production rollout gate	Must be true in production before install, grant, enable, hook fanout, or plugin invoke workers run. Keep unset/false until external sign-off.
PLUGIN_HOOK_FANOUT_REQUIRED	General-worker fanout fail-closed switch	Optional for staging/canaries. In production with PLUGIN_RUNTIME_PRODUCTION_ENABLED=true and Redis configured, hook fanout is required even when this is unset.
PLUGIN_TRUSTED_PUBLIC_KEYS	CSV of trusted Ed25519 public keys	Empty means plugin signature verification fails closed.
WORKER_QUEUE_GROUP=plugin	Starts the isolated plugin worker service	Required for hivegrid.plugin-invoke jobs. In production before runtime signoff it starts idle without queue consumers; after PLUGIN_RUNTIME_PRODUCTION_ENABLED=true, startup fails closed if invocation runtime dependencies cannot bootstrap.
WORKER_CONCURRENCY_PLUGIN	Plugin worker concurrency	Defaults to the worker runtime value. Tune separately from mail/general workers.
PLUGIN_HOOK_FANOUT_ENABLED	General-worker plugin hook fanout toggle	Leave unset/true when production runtime is enabled; explicit false makes readiness fail.
PLUGIN_CAPABILITY_USE_ENABLED	General-worker capability-use persistence toggle	Leave unset/true when production runtime is enabled; explicit false makes readiness fail.
PLUGIN_CAPABILITY_USE_CONCURRENCY	Capability-use audit sink concurrency	Defaults to 1 so per-tenant capability-use hash chains stay linear. Raise only after adding store-level serialization or advisory locking.
PLUGIN_BINARY_BASE_PATH	Plugin binary lookup root	Defaults to /app/var/plugins; required to contain verified <pluginId>/<version>/plugin.wasm files before install/invoke checks can pass.
PLUGIN_MAX_FUEL	Sandbox fuel limit	Defaults to 1000000. Charged for the hook call plus capability gateway calls.
PLUGIN_MAX_MEMORY_MB	Sandbox memory cap	Defaults to 64 MiB.
PLUGIN_MAX_WALL_CLOCK_MS	Sandbox wall-clock kill budget	Defaults to 5000 ms; the plugin sandbox worker thread is terminated on timeout.
PLUGIN_RATE_LIMIT_PER_MINUTE	Per-tenant/plugin invocation rate limit	Defaults to 100.
PLUGIN_RATE_LIMIT_WINDOW_SECONDS	Rate-limit window	Defaults to 60.
PLUGIN_AUTO_DISABLE_THRESHOLD	Tenant plugin crash threshold	Default is 10.
PLUGIN_AUTO_DISABLE_WINDOW_MS	Sliding crash window	Default is 600000.

Higher-risk host-binding resource controls remain runtime follow-up work. Do not document additional knobs as production-enforced without checking the worker bootstrap first.

Production Readiness Matrix

Capability slice	Operational status	Required before production use
Install / grant / revoke / enable / disable lifecycle	Implemented with tenant scope, permissions, audit events, JSON request validation, and capability review digest checks.	External WASM sandbox signoff plus PLUGIN_RUNTIME_PRODUCTION_ENABLED=true in the target environment.
Uninstall lifecycle	CLI/operator-only escape hatch that deletes the tenant Plugin row and writes an audit event. It is not a tenant API/UI action.	Operator backup/delete decision for plugin-owned data plus the external runtime signoff gate.
Settings iframe	Implemented with sandboxed iframe, restrictive CSP including navigate-to 'none', denied iframe allow permissions, parent-side unexpected-navigation remount, channel-bound strict-schema versioned plugin.settings.* postMessage contract, and non-secret per-user config persistence.	Keep secret-like settings blocked until encrypted secret storage and host binding review land.
Hook fanout	Implemented for the foundation hook set through outbox.plugin-hook on worker-general and hivegrid.plugin-invoke on worker-plugin.	Add payload/test coverage before enabling new hook families such as billing, auth, or richer workflow hooks.
Safe host bindings	log and audit are gateway-controlled and audited.	Confirm trusted public keys and worker-plugin health in the deployment runbook.
Deferred host bindings	http.outbound, secret.read, db.read, db.write, storage.read, storage.write, and tenant.info intentionally fail closed. Future http.outbound, secret.read, and db.read/db.write calls already have runtime per-call scope guards for exact HTTPS host, key-pattern, and table-prefix contracts, but the bindings are not exposed yet.	Dedicated security review, SSRF/tenant-boundary tests, tenant-boundary/storage tests, secret redaction, audit contracts, and operator copy for each binding.

Lifecycle

Lifecycle API mutations require JSON requests. Install, grant, revoke, enable, and disable calls to the tenant plugin API must send Content-Type: application/json; malformed JSON returns 400 invalid_json, and missing or unsupported media types return 415 unsupported_media_type. Enable and disable have no fields, but API clients still send {} as the JSON body so the lifecycle route keeps one validation contract.

Install

1. Verify manifest schema and schemaVersion.
2. Verify .wasm SHA-256 against the manifest.
3. Verify signature against PLUGIN_TRUSTED_PUBLIC_KEYS.
4. Persist the tenant-scoped plugin row.
5. Record a lifecycle audit event.

Use app.plugins.manage for routes or CLI actions that install plugins.

Grant

Capability and hook-consent grants are tenant-scoped. Manifest hook subscriptions are granted with the synthetic hook:<hook-name> capability name, so hook-only plugins cannot be enabled without explicit tenant-admin review. The backend permission split exists and the tenant-admin route now requires per-consent review checkboxes with a server-generated manifest digest, so stale or unreviewed capability and hook sets are rejected before grants are written.

app.plugins.use intentionally covers using installed plugins and writing the current user's non-secret plugin settings. app.plugins.manage remains the tenant API/UI boundary for install, capability grant/revoke, enable/disable, and future update operations. Uninstall is currently CLI/operator-only.

Revoke

Tenant admins can revoke individual capability grants with app.plugins.manage. The lifecycle service disables the plugin first when it is currently enabled, then removes the tenant-scoped grant rows and records plugin.capabilities.revoked with the revoked capability names and auto-disable status. Re-enable only after the requested capabilities are reviewed again.

Enable

Enable only after install, signature validation, capability review, and required worker services are live. Enabling records a lifecycle audit event.

Disable

Disable when a plugin is no longer needed, fails review, exceeds crash thresholds, or needs emergency containment. Disabling should prevent new invocations from being enqueued or executed.

Update

There is no production plugins update command and the database stores one installed row per (tenantId, pluginId), so same-plugin side-by-side versions are not supported. Until a reviewed update lifecycle exists, upgrades are operator-owned replace operations: disable the installed plugin, back up and decide retention for plugin-owned data, uninstall through the CLI escape hatch, then reinstall and re-grant the reviewed version. Version compatibility rules live in Plugin Versioning.

Uninstall

Uninstall is currently an operator CLI command, not a tenant API/UI action. It removes the tenant plugin row, writes a lifecycle audit event, and cascades owned plugin config where schema FKs define that behavior. Plugin-owned tables, migrations, binaries, and external artifacts require an explicit backup/delete decision before destructive cleanup.

Hook Delivery

The web/API side emits plugin.hook.* outbox events. The general worker consumes those events and fans out matching subscribers into hivegrid.plugin-invoke. The plugin worker then executes the WASM hook.

Current foundation hooks:

• tenant.created
• user.created
• entity.created
• entity.updated
• entity.deleted
• workflow.created
• workflow.updated
• workflow.deleted
• workflow.executed
• workflow.failed

tenant.created, user.created, row-backed entity.created/entity.updated/entity.deleted, tenant-scoped service-path workflow.created/workflow.updated/workflow.deleted, tenant-scoped template-import workflow.created, and the terminal workflow.executed hook are currently producer-wired. Entity hooks are registered for the generic contract, but non-row entity producers still need explicit fire points before they are treated as covered. Workflow definition hooks intentionally emit metadata only; platform-scoped workflow definitions and definition writes that bypass the reviewed service/import paths are not covered. Failed tenant-scoped web/API/manual and worker-native workflow executions additionally emit workflow.failed with metadata-only failure details so plugins can subscribe to failure handling without filtering the full workflow.executed stream.

Auto-Disable

The crash tracker counts failures per plugin and tenant in a sliding window. When the configured threshold is reached, the worker disables the tenant plugin and enqueues a plugin.auto_disabled audit event through audit.event.create, ideally in the same transaction.

Operator checks after auto-disable:

1. Inspect AuditEvent rows for plugin.auto_disabled (sourceModule=worker.plugin-runtime, outcome=blocked) and then lifecycle/capability-use rows.
2. Run hivegrid-cli plugins verify-audit-chain <tenantId> [--pluginId <pluginId>] to verify the tenant-wide PluginCapabilityUse hash chain. --pluginId reports matching rows but does not slice the tenant chain.
3. Inspect capability-use audit rows.
4. Inspect worker logs with pluginId, tenantId, and queue job id.
5. Confirm whether the failure is plugin code, manifest/capability mismatch, missing host binding, or host runtime error.
6. Re-enable only after the root cause is fixed or the plugin version is rolled back.

Current Known Gaps

• Host capability bindings are not complete. The worker routes log and audit through the capability gateway; higher-risk capabilities still need dedicated security review and tests before they are exposed. In production, install, grant, and enable reject manifest capabilities without reviewed host bindings.
• Invocation audit persists one PluginInvocation row per worker job outcome and correlates capability-use rows by pluginInvocationId; raw plugin input/output payloads are intentionally not stored there.
• Tenant-admin capability consent has a guided review panel with explicit confirmation, capability scope review, stale-manifest digest checks, and enable gating.
• The reusable plugin settings iframe shell, CSP, and versioned postMessage contract exist in PluginSettingsFrame; manifest ui.settings.entry metadata is routed through the server-side bundle loader, and non-secret per-user settings are stored in PluginUserConfig.configJson with plugin.configuration.updated audit events. The frame CSP denies navigation, the iframe allow policy denies browser capabilities, the host binds postMessage traffic to a per-render plugin-settings-channel nonce, rejects unexpected protocol fields before handlers run, and the host remounts the sandbox if plugin content triggers an unexpected iframe load. Rejected postMessage attempts and unexpected reloads create rate-limited, client-reported plugin.settings_frame.violation audit events with redacted metadata only; treat them as telemetry, not cryptographic proof of plugin behavior. Secret-like keys remain blocked until a dedicated encrypted secret binding is reviewed and shipped.
• Additional workflow plugin hooks beyond the current definition-metadata plus terminal-execution set are not complete.
• External WASM sandbox security review remains an external blocker for production enablement.

Verification

After docs-only changes:

pnpm run verify:docs-links

After runtime changes:

pnpm exec vitest run packages/plugin-runtime
pnpm --filter @hivegrid/worker build
pnpm run verify:deploy-workflow
pnpm run verify:docker-baseline