HiveGrid-Wireframes/hivegrid-cloud/docs/observability.md

Production Observability Runbook

This document is the production observability entrypoint for Production Docs diagnostics. It defines the safe surfaces operators and future agents may use to inspect health, readiness, request correlation, queues, worker failures, deployment verifier output, and redacted state without exposing raw secrets, customer data, provider payloads, tenant identifiers, or secret-shaped examples.

Use this runbook with docs/api.md, docs/deployment.md, docs/security.md, docs/configuration.md, and docs/worker.md. Those documents own the detailed API, deployment, security, configuration, and worker contracts; this document explains how to observe those surfaces safely during production triage.

Observability principles

Production diagnostics must answer three questions without broadening data exposure:

1. Which surface is unhealthy: web, API, worker, database, Redis, Mailjet, browser runtime, deployment, or verifier contract?
2. Which safe correlation value links the symptom to redacted logs or queue state: x-request-id, JSON requestId, job ID, queue name, timestamp, or deployment target?
3. Which non-secret status or diagnostic code explains the likely owner: configuration, dependency readiness, provider availability, payload validation, retry exhaustion, release mismatch, or manual runtime boundary?

Record small structured facts rather than raw logs. A good incident note contains service label, endpoint or queue name, status, request ID or job ID, timestamp, diagnostic code, retry count, and next owner. A bad incident note contains full headers, request bodies, job payloads, provider responses, stack traces, environment dumps, database rows, Redis URLs, database URLs, bearer tokens, API keys, cookies, customer domains, tenant identifiers, or screenshots with credentials.

Health and readiness surfaces

Use liveness checks to prove a process can respond and readiness checks to decide whether dependencies are usable. Do not turn health checks into customer-data probes or provider-console dumps.

Surface	Safe health/readiness evidence	Unsafe evidence
Web runtime	GET /healthz status, GET /readyz status, service label, dependency diagnostic code, timestamp, release target label.	Cookies, rendered customer pages, tenant hostnames, browser screenshots, full response bodies with user data.
API runtime	GET /healthz, GET /readyz, /v1/health, response status, x-request-id, JSON requestId, stable error code, timestamp.	Authorization headers, API keys, bearer tokens, request bodies, customer identifiers, raw CORS headers containing customer domains.
Worker runtime	Worker enabled/disabled state, health command status, service label, revision, queue names, Redis parse/readiness state, database readiness state, dependency diagnostic codes.	Redis connection strings, database URLs, raw job payloads, provider payloads, browser runtime credentials, stack traces with connection strings.
Database readiness	Ready/unready status, migration/status code, dependency diagnostic code, timestamp, owning service.	DATABASE_URL, hostnames tied to tenants, usernames, passwords, query output, customer rows, connection strings.
Redis and queue readiness	Selected key name such as BULLMQ_REDIS_URL or fallback REDIS_URL, parse status, queue readiness status, queue names, job counts by status.	Redis URL values, usernames, passwords, hosts, database numbers, TLS material, payload contents, tenant data.
Mailjet readiness	Provider category, configured/missing state for required key names, delivery status class, webhook signature pass/fail, request ID or job ID.	MAILJET_API_KEY, sender secrets, webhook signing secrets, provider payloads, email addresses, message bodies, provider dashboard screenshots.
Browser runtime readiness	Enabled/disabled status, endpoint configured/disabled code, timeout parse status, dependency code, queue/job correlation.	Runtime endpoint values, API keys, session credentials, screenshots with secrets, browser artifacts, captured page contents.
Deployment readiness	Static verifier PASS/FAIL lines, image target names, immutable tag IDs, Coolify deployment status label, health/readiness status after approved runtime inspection.	Registry credentials, Coolify webhook URLs, UUID-like deploy endpoints, bearer tokens, passwords, environment dumps, raw deploy logs containing secrets.

Request IDs, timestamps, and correlation IDs

x-request-id is the primary API correlation value. The API may also return a JSON requestId; when both are present, compare them and use the matching value to search redacted logs. Clients may supply a request ID, and generated IDs are also acceptable when they do not encode customer data.

For worker paths, use job ID, queue name, job type, attempt number, last attempted timestamp, final failure timestamp, and any safe request ID that was already attached by the enqueueing API call. For deployment paths, use target name, immutable tag ID, static verifier status, workflow run label, and timestamp. For provider paths, use provider category, status class, redacted diagnostic code, request ID or job ID, and timestamp.

Correlation values are not permission to collect sensitive data. Keep payloads inside approved production logging or provider-console boundaries and summarize only safe facts in tickets, pull requests, docs, and verification summaries.

Structured and redacted logs

Production logs should be structured enough to filter by service, surface, status, request ID, job ID, queue name, timestamp, dependency code, and error class. They must be redacted before they become shared artifacts.

Safe log fields:

• service: web, api, worker, database, redis, mailjet, browser-runtime, or deployment.
• surface: endpoint path class, queue name, job type, health/readiness check, verifier name, or deployment target.
• status: HTTP status, ready/unready, pass/fail, enabled/disabled, retrying, failed, completed, delayed, paused, or dead-lettered.
• requestId, jobId, correlationId, and timestamps when they are non-secret.
• Stable error code, redacted dependency diagnostic code, retry count, maximum attempts, and remediation owner.

Unsafe log fields:

• Raw request or response bodies, full headers, cookies, authorization headers, bearer tokens, API keys, service-token values, and secret-shaped substrings.
• DATABASE_URL, BULLMQ_REDIS_URL, REDIS_URL, Supabase keys, Mailjet keys, browser runtime credentials, registry credentials, Coolify webhook values, and passwords.
• Tenant identifiers, customer identifiers, customer domains, account names, email bodies, uploaded file contents, raw database records, queue payloads, provider payloads, browser screenshots, and artifact contents.
• Stack traces or provider responses that contain connection strings, tokens, customer data, or endpoint secrets.

If an unsafe field appears in a shared artifact, remove the artifact from the shared surface, rotate the affected credential when applicable, and replace the note with a redacted summary that names only keys, codes, statuses, and containment steps.

Queue and job inspection

Worker triage may inspect queue state when a feature is queue-backed, retryable, external-provider-backed, scheduled, browser-backed, or too slow for request/response handling. Use the queue contract from docs/worker.md and the feature manifest before replaying or draining work.

Safe queue inspection includes:

• Queue name and job type.
• Job ID, status, attempt number, maximum attempts, retry count, and next retry timestamp.
• Counts for waiting, active, completed, failed, delayed, paused, stalled, retrying, or dead-lettered jobs.
• Last error class or redacted diagnostic code.
• Cleanup status for temporary artifacts.
• Correlation ID or request ID from the enqueueing request when already safe.
• Remediation owner such as configuration, provider, payload validation, dependency availability, or replay decision.

Unsafe queue inspection includes raw payload JSON, message bodies, uploaded file contents, tenant IDs, customer IDs, customer email addresses, Redis URLs, database URLs, Mailjet API keys, browser runtime credentials, provider responses, screenshots, and stack traces with connection strings.

Failed-job views and dead-letter queues are sensitive operational data. They may be used to identify failed job classes, retry decisions, and ownership, but they must not become shared dumps of payloads or customer records. Replays, drains, queue pauses, and provider-console inspection are manual runtime actions through approved tooling.

Worker failed-job diagnostics

When a worker job fails, capture enough redacted information to decide whether to retry, pause, fix configuration, or escalate:

1. Record queue name, job type, job ID, request ID or correlation ID, attempt number, retry limit, status, timestamp, and redacted error class.
2. Check worker health for enabled/disabled state, Redis parse/readiness status, database readiness state, queue readiness, and dependency codes.
3. Classify the failure owner as configuration, provider availability, payload validation, code defect, timeout, idempotency, or manual dependency outage.
4. If Mailjet is involved, record provider category, delivery status class, webhook signature pass/fail, and request/job correlation only.
5. If browser runtime work is involved, record runtime configured/disabled status, timeout parse status, queue state, and diagnostic code only.
6. If the retry limit is exhausted, keep the payload in approved production tooling and share only the redacted classification, timestamps, and replay decision.

Do not paste job payloads, provider payloads, email bodies, browser artifacts, customer records, credentials, Redis/database URLs, or stack traces with secrets into issue comments, chat, documentation, or verification output.

Deployment verifier output and release diagnostics

Deployment diagnostics are split between deterministic static proof and manual runtime proof.

Auto-Mode-safe static evidence:

• pnpm run verify:production-docs for the production documentation contract.
• pnpm run verify:deploy-workflow for the deploy workflow contract.
• node scripts/verify-gitea-deploy-workflow.mjs and node scripts/verify-gitea-ci-workflow.mjs for static workflow PASS/FAIL output.
• Focused verifier tests that do not launch browser automation, live Docker, registries, Coolify, provider consoles, or production traffic.

Safe verifier notes include command name, verifier name, PASS/FAIL line, checked file path, image target name, immutable tag policy, and forbidden-command exclusion result. Unsafe notes include real registry hosts, namespaces, credentials, Coolify URLs, UUID-like endpoints, bearer tokens, passwords, raw webhook values, or deploy logs with environment dumps.

Manual runtime deployment evidence outside the configured Gitea CI/deploy workflows is allowed only after a human explicitly approves the exact focused run. If approved, record target name, immutable tag ID, health/readiness status, request ID, queue/job correlation, timestamp, and pass/fail result. Keep ad hoc live Docker builds, ad hoc registry pushes, ad hoc Coolify execution, provider-console checks, browser flows, Playwright, Cypress, and E2E checks out of Auto-Mode verification.

Safe and unsafe diagnostic fields by incident type

Incident type	Safe fields	Unsafe fields
API request failure	Method, path, status, stable error code, x-request-id, JSON requestId, timestamp, generic origin label.	Authorization header, API key, bearer token, cookies, raw body, full headers, customer domain, tenant/customer identifiers.
Health response failure	Service label, endpoint, ready/unready, dependency diagnostic code, timestamp, request ID.	Environment dump, connection string, rejected secret value, stack trace with credentials, customer data.
Redis or queue issue	Selected key name, parse status, queue readiness, queue name, job counts, retry count, redacted dependency code.	Redis URL, username, password, host, database number, TLS material, payload contents.
Database readiness issue	Ready/unready, migration/status code, owning service, timestamp, diagnostic code.	DATABASE_URL, query output, usernames, passwords, hostnames, database names, customer rows.
Mailjet or webhook failure	Provider category, delivery status class, webhook signature pass/fail, request ID or job ID, timestamp.	API key, webhook secret, signing header value, provider payload, email address, email body, dashboard screenshot.
Worker retry exhaustion	Queue name, job type, job ID, attempts, retry limit, final failure timestamp, redacted error class, owner.	Raw payload, file contents, tenant/customer data, provider response, browser artifact, stack trace with secrets.
Deployment triage	Verifier name, PASS/FAIL summary, target name, immutable tag ID, deployment status label, health/readiness status.	Registry credentials, Coolify webhook URL, UUID endpoint, tokens, passwords, full environment values, raw deploy logs with secrets.

Failure triage flow

1. Identify the surface: web health, API readiness, API auth, CORS, worker queue, database, Redis, Mailjet, browser runtime, deployment, or documentation verifier.
2. Capture safe correlation facts: request ID, job ID, queue name, target name, timestamp, status, stable error code, and redacted diagnostic code.
3. Compare the symptom with the owning runbook: API, deployment, security, configuration, worker, database, mail, or health baseline documentation.
4. Run focused non-E2E verification only when the issue is a repository-contract regression. Use deterministic package or static verifier commands rather than live runtime checks.
5. Keep browser flows, Playwright, Cypress, provider-console inspection, ad hoc live Docker builds, ad hoc registry pushes, ad hoc Coolify execution, and E2E checks manual unless a human explicitly approves the exact focused run in the current session; configured Gitea CI/deploy workflow Docker, registry, Coolify, migration, and health steps are the automated exception.
6. If a secret or customer value was exposed, contain the artifact, rotate credentials when applicable, and publish a redacted incident note with only keys, statuses, codes, timestamps, and remediation steps.

Supported non-E2E verification

These commands are safe for Auto-Mode because they do not launch browser automation, E2E flows, live Docker services, registries, Coolify, provider consoles, or production traffic:

test -s docs/observability.md
python - <<'PY'
from pathlib import Path
text = Path('docs/observability.md').read_text()
for marker in ('TO' + 'DO', 'T' + 'BD', 'FIX' + 'ME'):
    assert marker not in text
PY
pnpm run verify:production-docs
pnpm exec vitest run scripts/verify-production-docs.test.mjs

Live runtime checks outside the configured Gitea CI/deploy workflows are manual-only unless a human explicitly approves the exact focused run. Manual-only activities include browser flows, Playwright, Cypress, E2E checks, ad hoc live Docker builds, ad hoc registry pushes, ad hoc Coolify execution, provider-console inspection, queue replay/drain operations, and production traffic probes beyond approved health/readiness inspection. When a manual check is approved, record only non-secret command purpose, service label, status, request ID or job ID, diagnostic code, timestamp, and pass/fail result.

Lighthouse-CI performance gate

A Lighthouse-CI gate is configured in .lighthouserc.json and invoked via pnpm run verify:lighthouse (wrapper at scripts/run-lighthouse.mjs). The gate is opt-in — it is not wired into the main CI workflow yet — and is SKIP-safe when the @lhci/cli binary cannot be installed in the current environment.

Thresholds against http://localhost:3000/ and http://localhost:3000/pricing (desktop preset, 3 runs):

• categories:performance >= 0.85 (error)
• categories:accessibility >= 0.90 (error)
• categories:best-practices >= 0.85 (warn)
• categories:seo >= 0.85 (warn)
• first-contentful-paint <= 2500ms (warn), largest-contentful-paint <= 3500ms (warn), total-blocking-time <= 300ms (warn), cumulative-layout-shift <= 0.1 (warn)

Run locally against a running web dev/prod server. The script exits 0 when lhci is unavailable so it can be safely added to local pre-merge sweeps without introducing flakes on hosts that block npx network installs.

Escalation notes

Escalate when redacted health/readiness, queue state, and verifier output do not identify an owner or when containment requires production access outside the current operator boundary. The escalation note should include service label, surface, status, correlation ID, timestamp, diagnostic code, retry state, attempted safe verification, and the requested owner. It must exclude secrets, customer data, provider payloads, raw logs, environment dumps, stack traces with credentials, screenshots, and payload JSON.

If a future diagnostic surface needs more data, add a narrowly scoped safe field to the implementation and update this runbook at the same time. Do not bypass redaction by copying raw production output into documentation or support threads.