Full-Text Search

This document describes the Meilisearch-backed search subsystem introduced in phase 04 (#48). It is the user-facing complement to docs/deploy/meilisearch.md (operator runbook).

Architecture

+------------------+     +--------------------+     +-----------------------+
|  Domain service  | --> |  OutboxEvent (DB)  | --> | dispatcher → BullMQ   |
|  (KB / CRM / …)  |     |  type = search.…   |     | outbox.search-index   |
+------------------+     +--------------------+     +-----------+-----------+
                                                                |
                                                                v
                                                  +--------------------------+
                                                  | search-indexer (general) |
                                                  |  upsert / delete / bulk  |
                                                  +-------------+------------+
                                                                |
                                                                v
                                                       +-----------------+
                                                       |  Meilisearch    |
                                                       |  (tenantId      |
                                                       |   isolated)     |
                                                       +-----------------+

Source of truth is Postgres. Meilisearch is a projected read-side populated asynchronously through the outbox pattern (#42).
Tenant isolation is enforced inside @hivegrid/search. Every read and write goes through the tenant-scoped package helpers with an explicit tenantId. The wrapper injects tenantId = "<id>" into the Meilisearch filter expression and escapes the value to defeat filter injection. Raw Meilisearch SDK access is not re-exported.
Estate namespace: schemas, outbox payloads and worker pagers use logical names such as projects. The SearchClient adapter is the only boundary that maps them to physical Meilisearch UIDs (hivegrid-projects). Callers never add the prefix themselves, and the adapter rejects already-prefixed input so no unprefixed or double-prefixed namespace can be touched.
Graceful degradation: every hot-path is wrapped in searchWithFallback, which routes to the legacy SQL LIKE implementation when Meilisearch raises a connectivity error. The search stays degraded-but-up instead of returning 500s.

User-Facing Contract

Since the 2026-07-12 scope cut, global search is exposed only as a Spotlight overlay. On desktop it opens through Cmd+K/Ctrl+K or the header search trigger; on mobile the header trigger opens a viewport-filling overlay. There is no user-facing /search page, shareable search URL, or full-page filter/pagination surface in v1. /search/api remains the protected, tenant-scoped internal resource used by the overlay. R-P11b continues to consume the same index and search contracts.

Index Schema Reference

The current schemas live in packages/search/src/schemas/. Each is a whitelist — only fields listed in allowedDocumentFields reach Meilisearch. Names in this table are logical contract names; the physical UID always has the estate prefix hivegrid-.

Index	Searchable	Filterable	Sortable
`kb-articles`	title, summary, content, tags	tenantId, knowledgeBaseId, language, isPublished, tags	createdAt, updatedAt, publishedAt
`crm-contacts`	firstName, lastName, email, company, jobTitle, tags	tenantId, entityType, status, ownerId, tags	createdAt, updatedAt, lastInteractionAt
`rows`	searchable, folio, entityName	tenantId, entityId, entityName, createdById	createdAt, updatedAt, folio
`entities`	name, title, titlePlural	tenantId, type	name, createdAt, updatedAt
`users`	firstName, lastName, email	tenantId, type, admin, active	createdAt, lastLoginAt

tenantId is filterable on every index (the package refuses to create one otherwise).

Tuning Tips

Ranking rules default to Meilisearch's recommendation: words → typo → proximity → attribute → sort → exactness. Per-tenant experimentation can shorten the chain to favour recency: words → typo → sort → attribute → exactness.
Synonyms are per-index in the schema. Add them sparingly — they expand every query and inflate index size.
Stop-words for full-text content (German der/die/das, English the/a/an) lift relevance noticeably; add them to the schema (stopWords).
Highlighting: pass attributesToHighlight through searchInIndex({ options: { ... } }) to receive <em>-wrapped snippets in the response.

Hot-Path Migrations

Path	Migrated	Notes
KB public search (`KnowledgeBaseService.search`)	yes (#48)	wraps `findKnowledgeBaseArticles` via `searchKnowledgeBaseArticles` helper with `searchWithFallback`. Tenant id resolved from KB record.
CRM contact helper	shipped	`apps/web/app/modules/crm/services/crmContactSearch.server.ts` wraps `crm-contacts` with `searchWithFallback` and a tenant-scoped SQL fallback.
Rows helper	shipped	`apps/web/app/modules/rows/services/rowSearch.server.ts` wraps the `rows` index with an `entityId` filter and tenant-scoped SQL fallback.
CRM/Rows producer and callsite integration	yes	`RowsApi.getAll` calls `searchCrmContacts` for tenant-scoped CRM contact-list free-text filters and `searchRows` for other tenant-scoped row filters. `RowsApi.create / createSimple / update / del` emit `search.index.*` events through `rowSearchOutbox.server.ts`. Contact rows fan out to both `crm-contacts` and `rows`.
HiveGrid global-search producers	yes	`apps/web/app/services/search/searchIndexProjection.server.ts` publishes `search.index.*` events and mirrors `SearchIndexEntry` for Kontakt/Firma/Lead/Projekt/File/Task/User lifecycle mutations. The Spotlight overlay reads through `/search/api` using tenant- and permission-aware SQL+Meili search.
Bulk reindex pagers	partial	`apps/worker/src/index.ts` registers `createSearchIndexerBulkPagers({ prisma })`; `BULK_REINDEX_SUPPORTED_INDEX_NAMES` exposes `kb-articles`, `crm-contacts`, `users`, and `projects`. File/task live producers are wired, but their disaster-recovery bulk pagers still need explicit coverage before `--index=all` can rebuild every projected index.

Open hot-path gaps still serve from the legacy SQL path. Until a specific producer emits index events, a specific callsite reads through the helper, and the indexer has the relevant backfill pager, relevance, typo-tolerance and disaster-recovery reindex coverage remain partial for that path. Rows, CRM contacts and the HiveGrid global-search entities now have read/CUD producer pieces wired. CRM contacts, users, projects and KB articles also have worker bulk-reindex pagers; files/tasks still depend on live producer replay until a dedicated DR pager is added. Entities still require a dedicated bulk pager before they are part of the disaster-recovery backfill. Row indexing is still published post-write rather than inside the row mutation transaction.

CLI Operations

# Reindex every supported bulk-reindex index for every tenant via outbox events.
# Today this means KB articles, CRM contacts, users, and projects; unsupported names are rejected.
pnpm exec tsx packages/cli/bin/hivegrid-cli.mjs search reindex --tenant=all --index=all

# Reindex one tenant's KB articles.
pnpm exec tsx packages/cli/bin/hivegrid-cli.mjs search reindex --tenant=tnt_abc123 --index=kb-articles

The CLI never writes to Meilisearch directly. It produces search.index.<entity>.bulkreindex outbox events; the worker-general service consumes them and pages over Postgres through registered bulkPagers. Today, kb-articles, crm-contacts, users, and projects are listed in BULK_REINDEX_SUPPORTED_INDEX_NAMES; the CLI and worker enforce that contract so operators cannot enqueue unsupported bulk jobs by accident. Operators should not rely on full-index backfills for entities until its pager is added. A full disaster-recovery rebuild estimate can only be validated after that pager is present in the target environment.

Rollback

A two-layer rollback strategy is built into the design:

Feature flag (SEARCH_BACKEND=sql) forces every hot-path onto the SQL fallback immediately. Worker indexing continues but is not read from. Flip the env in production, redeploy the web tier, done.
Disable the worker by setting SEARCH_INDEXER_ENABLED=false on the worker-general service. Outbox events queue up in Postgres (Outbox retention is handled by the cleanup job from phase 03) and resume when re-enabled — followed by a search reindex to catch up any drift.

Performance Expectations

P95 search latency under 50 ms for index sizes up to 1 M documents on a 1 vCPU / 2 GiB Meilisearch instance.
Indexer throughput around 10 k documents per minute per worker-general replica (CPU-bound — scale horizontally if needed).
Disk: roughly 30 MiB per 100 k documents on the KB-articles index, 20 MiB per 100 k contacts, less for the entity/user indexes.

Disaster Recovery

The Meilisearch volume is treated as cache. To rebuild from scratch:

docker volume rm meilisearch-data (after stopping the container).
Restart the container.
pnpm exec tsx packages/cli/bin/hivegrid-cli.mjs search reindex --tenant=all --index=all.

This rebuild currently covers the supported bulk-reindex set only: kb-articles, crm-contacts, users, and projects. Entities require a dedicated bulk pager before they are part of the disaster-recovery backfill. Expected time-to-recovery on a single replica: <30 minutes for 1 M KB-article documents.

12 KiB Raw Blame History