HiveGrid-Wireframes/hivegrid-cloud/docs/troubleshooting.md

Production Troubleshooting Runbook

This runbook is the consolidated production entrypoint for safe failure triage across local setup, configuration, database, mail, worker, API, deployment, security, and observability surfaces. Use it with the focused guides it links to; do not copy raw runtime output into shared artifacts when a file-scoped document already owns the detailed contract.

Triage order

1. Identify the affected surface: local setup, environment configuration, database or migration, Mailjet or webhook, worker or queue, API auth/OpenAPI/request ID, deployment image or Coolify, security headers/CORS, or observability output.
2. Capture safe correlation facts first: x-request-id, JSON requestId, job ID, queue name, diagnostic code, status, retry count, timestamp, image target, or health/readiness state.
3. Compare the symptom with the owning runbook before changing runtime settings:
   • Quickstart for fresh local bootstrap.
   • Local development for day-to-day local PostgreSQL, Prisma, seed, and worker opt-in workflows.
   • Environment variables for required keys, optional integration keys, and redaction rules.
   • Database operations for PostgreSQL, Prisma, migration status, deploy migrations, backups, and restore expectations.
   • Mail operations for Mailjet delivery and webhook signing.
   • Worker operations for queue runtime, Redis configuration, retries, and failed-job handling.
   • API runbook for /v1 endpoints, authentication, OpenAPI, CORS, request IDs, and JSON errors.
   • Deployment for Docker targets, image tags, Coolify runner-side deployment, rollback, and health/readiness inspection.
   • Security for headers, CORS, API tokens, webhook boundaries, and incident containment.
   • Observability for safe diagnostics, structured logs, queue inspection, request IDs, and escalation.
4. Run focused non-E2E static or package verification only when the issue is a repository-contract regression. Keep browser flows, Playwright, Cypress, ad hoc live Docker builds, ad hoc registry pushes, ad hoc Coolify execution, provider-console inspection, queue replay, and production traffic probes manual unless a human explicitly approves the exact focused run; configured Gitea deploy workflow Docker, registry, Coolify, migration, and health steps are the automated exception.
5. Escalate with a redacted note when the owner is unclear, containment requires production access outside the current operator boundary, or an unsafe value was exposed.

Safe evidence collection

Collect the smallest structured evidence set that identifies the owner without revealing protected material.

Safe evidence includes:

• Service label, endpoint path class, queue name, job type, image target, verifier name, and checked file path.
• HTTP status, health/readiness state, queue status, worker enabled/disabled state, migration/status code, provider status class, and PASS/FAIL verifier line.
• x-request-id, JSON requestId, job ID, correlation ID, retry count, attempt number, immutable tag ID, and timestamps when they do not encode customer data.
• Stable error code, redacted dependency diagnostic code, missing/present/invalid key name, parse status, and remediation owner.

Unsafe evidence includes:

• Raw secret values, database URLs, Redis URLs, API keys, bearer tokens, cookies, passwords, registry credentials, Coolify webhook values, provider tokens, and secret-shaped examples.
• Customer data, tenant identifiers, customer domains, account names, email contents, uploaded file contents, database rows, raw request bodies, raw response bodies with customer data, raw job payloads, and provider payloads.
• Full environment dumps, authorization headers, webhook signature values, screenshots with credentials, browser artifacts, stack traces containing connection strings, and copied deploy logs with secrets.

If an unsafe value appears in a shared artifact, remove the artifact from the shared surface, rotate affected credentials when applicable, and replace the record with a redacted summary naming only keys, codes, statuses, timestamps, and containment actions.

Redaction rules

Use key names and diagnostic states, never raw values. Acceptable phrases include missing required env key DATABASE_URL, queue config missing BULLMQ_REDIS_URL, 401 invalid_webhook_signature, cors_origin_denied, database readiness failed, worker retry exhausted, and deployment verifier failed for worker target.

Do not include raw secret values, customer data, Redis URLs, database URLs, API keys, tokens, provider payloads, webhook bodies, webhook signatures, or full environment values in tickets, pull requests, logs, screenshots, documentation, or verification summaries. When evidence must refer to a runtime value, use a generic label such as service-token configured, example-api-origin, redacted-provider-error, request-id-for-correlation, queue-name, or immutable-tag-id.

Local setup failures

Start with Quickstart for first-run setup and Local development for ongoing workflows.

Safe checks:

• Confirm Corepack and the pinned pnpm@10.33.0 are active.
• Confirm dependencies installed, PostgreSQL is running locally, and Prisma commands target a disposable local database.
• Confirm .env was created from .env.example with placeholder-only local values.
• Capture command name, exit code, high-level phase, and timestamp.

Common owners:

• Package manager drift when install or script execution fails before application code starts.
• PostgreSQL not running, missing local database, or local Prisma migration drift.
• Seed order problems when pnpm seed runs before migrations are applied.
• Worker accidentally enabled during a normal web-only loop.

Do not copy local .env contents, seeded account credentials, database connection strings, or screenshots with private values into shared notes.

Environment configuration failures

Use Environment variables as the key ownership and redaction source of truth.

Safe checks:

• Identify the owning runtime: web/API, worker, mail provider, webhook handler, database operation, or deployment workflow.
• Record key names and validator states only: missing, present, invalid, parsed, enabled, disabled, or fallback selected.
• Confirm fallback order when Redis is involved: BULLMQ_REDIS_URL is preferred and REDIS_URL is fallback-only when the runtime contract allows it.

Escalate when a required key is missing from the secret store, when an optional key is being treated as required without documentation, or when runtime output reveals a rejected value instead of a redacted code.

Database and migration failures

Use Database operations for PostgreSQL, Prisma, migration status, migration deployment, generated client artifacts, backup, and restore expectations.

Safe checks:

• Record command name, diagnostic phase, exit code, migration/status code, sanitized Prisma excerpt, timestamp, and owning service.
• Distinguish read-only status checks from migration deployment.
• Confirm generated Prisma client artifacts are current when generated types or runtime queries appear stale.
• Confirm backup or restore-point policy before production migration deployment.

Common owners:

• Missing or invalid DATABASE_URL or required DIRECT_URL slot.
• Database service unavailable or authentication failure in the secret store.
• Migration drift between reviewed migrations and the target database.
• Stale generated Prisma client after schema or dependency changes.

Do not paste connection strings, usernames, passwords, hosts, database names, schemas containing customer data, SQL query output, or customer rows into shared artifacts.

Mailjet and webhook failures

Use Mail operations for outbound transactional mail and inbound webhook signing boundaries.

Safe checks:

• For outbound delivery, record provider category, sender-domain status class, request ID or job ID, queue status, retry count, and redacted Mailjet status class.
• For inbound webhooks, record whether required header classes were present, timestamp freshness status, signature pass/fail code, endpoint label, request ID, and HTTP status.
• Confirm MAILJET_API_KEY and MAILJET_FROM_EMAIL are configured together when delivery is enabled.
• Confirm MAILJET_WEBHOOK_SECRET is used only for inbound webhook verification and never as the outbound API key.

Common owners:

• Missing or expired Mailjet outbound key.
• Unverified sender domain or sender address configuration.
• Worker queue not processing outbound jobs.
• Proxy or middleware stripping webhook headers.
• Raw body transformed before signature verification.

Do not include Mailjet API keys, webhook signing secrets, provider payloads, webhook bodies, signature header values, email contents, tenant IDs, customer identifiers, or real attempt IDs in shared notes.

Worker and queue failures

Use Worker operations for worker enablement, Redis/BullMQ configuration, queue health, retry behavior, dead-letter handling, and manual runtime boundaries.

Safe checks:

• Record WORKER_ENABLED enabled/disabled state, selected Redis key name, Redis parse/readiness status, queue name, job type, job ID, status, attempt number, retry count, retry limit, timestamp, and redacted error class.
• Confirm the feature actually requires queue-backed work before enabling the worker locally or in production.
• Confirm failed-job or dead-letter views are treated as sensitive operational data and summarized only by classification.
• Classify ownership as configuration, provider availability, payload validation, code defect, timeout, idempotency, dependency outage, or replay decision.

Common owners:

• Worker disabled for a deployment that expects processors.
• Missing BULLMQ_REDIS_URL or unsupported fallback use of REDIS_URL.
• Database readiness failure inside worker health.
• Retry exhaustion after provider, payload, or dependency failures.
• Queue paused, delayed, stalled, or dead-lettered jobs.

Do not share Redis URLs, database URLs, mail tokens, browser runtime credentials, raw job payloads, stack traces with connection strings, provider responses, uploaded files, tenant identifiers, or customer data.

API auth, OpenAPI, and request-id failures

Use API runbook for the dedicated Hono API surface, authentication, OpenAPI, health, request IDs, CORS, and JSON error envelopes.

Safe checks:

• Confirm the path is part of the supported /v1 surface and compare with /v1/openapi.json.
• For auth failures, record method, path, status, stable error code, whether the expected credential channel was used, and x-request-id.
• Confirm API_ACCESS_TOKEN is configured when service-token endpoints are expected to authenticate.
• Confirm API errors include a safe JSON envelope with error.code, error.message, and error.requestId.
• Confirm responses include x-request-id; compare it with JSON requestId when both are present.

Common owners:

• Unsupported path or stale client schema.
• Missing API service token configuration.
• Wrong credential channel: bearer token, x-api-key, or JSON apiKey.
• Request ID middleware not applied to an error path.
• CORS allowlist mismatch for browser-origin calls.

Do not paste authorization headers, API keys, bearer tokens, cookies, raw request bodies, customer identifiers, tenant domains, or full headers into shared artifacts.

Deployment, Coolify, and image failures

Use Deployment for Docker target contracts, image tag policy, Coolify runner-side deployment, rollback, and health/readiness inspection.

Safe checks:

• Record static verifier name, PASS/FAIL line, checked target, image target name, immutable full-SHA tag ID, workflow run label, timestamp, and deployment status label.
• Confirm web, api, and worker image targets are built and published with immutable tags before any Coolify trigger.
• Confirm Coolify triggering runs from the self-hosted runner after image publication and reads endpoint/resource data from configured CI variables and secrets.
• During approved runtime rollback inspection, record target name, previous known-good tag, health/readiness status, request ID, queue/job correlation, and timestamp.

Common owners:

• Missing Docker target copy path or runtime command drift.
• Mutable tag used where an immutable tag is required.
• Coolify trigger attempted before all deployable images were published.
• Registry login or image push failure in CI secret configuration.
• Readiness failure after deployment caused by dependency configuration rather than image build.

Do not share registry hosts tied to private deployments, namespaces, credentials, passwords, Coolify URLs, UUID-like endpoints, bearer tokens, webhook values, full deploy logs with environment dumps, or container environment values.

Security, CORS, and header failures

Use Security for the production security entrypoint and its linked detailed header and CORS checklist.

Safe checks:

• Record affected surface, method, generic path class, status, header name, expected policy value, generic origin label, stable error code, and x-request-id.
• For CORS, compare expected allowed origin behavior with denied or malformed origin behavior; positive-only checks can miss over-broad allowlist regressions.
• For API token incidents, record configured/missing/invalid status and endpoint metadata only.
• For webhook security incidents, record signature verification pass/fail and request ID after confirming no downstream work occurred before verification.

Common owners:

• Missing or weakened web/API security header.
• CORS allowlist too narrow for a legitimate origin or too broad for denied origins.
• Token validation echoing raw credentials.
• Webhook handler trusting parsed or transformed bodies before signature verification.
• Logs or errors revealing secrets, tenant identifiers, customer data, or provider payloads.

Do not record customer domains, tenant hostnames, raw browser headers, cookies, authorization headers, webhook signatures, environment dumps, or screenshots containing personal data.

Observability and diagnostics failures

Use Observability for safe diagnostic fields, structured logs, queue inspection, deployment verifier output, request IDs, timestamps, and escalation.

Safe checks:

• Confirm logs and verifier output are file-scoped or surface-scoped enough to identify the owner.
• Confirm diagnostics include request IDs, job IDs, queue names, timestamps, retry counts, status, stable error codes, and redacted dependency codes where applicable.
• Confirm health/readiness output reports service label and dependency status without probing customer data.
• Confirm static verifier failures name the checked file, missing path, missing anchor, forbidden placeholder, or forbidden command class.

Common owners:

• Missing request ID on API errors.
• Queue diagnostics lacking retry count or timestamp.
• Health/readiness output too broad or too secret-bearing.
• Verifier output not naming the file or anchor that failed.
• Shared logs containing raw payloads, environment values, stack traces with credentials, or customer identifiers.

If a diagnostic surface needs more detail, add a narrowly scoped safe field to the implementation and update the owning runbook in the same change. Do not bypass redaction by copying raw production output into shared channels.

Escalation notes

Escalate with a concise, redacted note when safe triage cannot identify an owner or when containment requires production access outside the operator boundary.

Include:

• Service label or surface.
• Method/path class, queue name, image target, or verifier name.
• Status, stable error code, diagnostic code, health/readiness state, retry count, and timestamp.
• x-request-id, JSON requestId, job ID, or correlation ID when safe.
• Focused non-E2E verification already run and result.
• Requested owner: configuration, database, mail provider, worker/queue, API, deployment, security, observability, or product feature.

Exclude:

• Raw secrets, customer data, provider payloads, request/response bodies, environment dumps, stack traces with credentials, screenshots, database rows, Redis/database URLs, API keys, tokens, Coolify webhook values, and browser artifacts.

Supported non-E2E verification

These checks are safe for this documentation surface because they do not launch browser automation, E2E flows, live Docker services, registries, Coolify, provider consoles, or production traffic:

test -s docs/troubleshooting.md
pnpm run verify:production-docs
pnpm run verify:docs-links

Browser flows, Playwright, Cypress, live Docker builds, live registry pushes, Coolify execution, provider-console inspection, queue replay, and production traffic probes remain manual-only unless a human explicitly approves the exact focused run. When an approved manual check occurs, record only command purpose, service label, status, request ID or job ID, diagnostic code, timestamp, and pass/fail result.