HiveGrid-Wireframes/hivegrid-cloud/docs/features/search.md

Full-Text Search

This document describes the Meilisearch-backed search subsystem introduced in phase 04 (#48). It is the user-facing complement to docs/improvement-plan/04-search-rag/meilisearch.md (technical spec) and docs/deploy/meilisearch.md (operator runbook).

Architecture

+------------------+     +--------------------+     +-----------------------+
|  Domain service  | --> |  OutboxEvent (DB)  | --> | dispatcher → BullMQ   |
|  (KB / CRM / …)  |     |  type = search.…   |     | outbox.search-index   |
+------------------+     +--------------------+     +-----------+-----------+
                                                                |
                                                                v
                                                  +--------------------------+
                                                  | search-indexer (general) |
                                                  |  upsert / delete / bulk  |
                                                  +-------------+------------+
                                                                |
                                                                v
                                                       +-----------------+
                                                       |  Meilisearch    |
                                                       |  (tenantId      |
                                                       |   isolated)     |
                                                       +-----------------+

• Source of truth is Postgres. Meilisearch is a projected read-side populated asynchronously through the outbox pattern (#42).
• Tenant isolation is enforced inside @toolctrl/search. Every read and write goes through searchInIndex / bulkIndex / deleteFromIndex with an explicit tenantId. The wrapper injects tenantId = "<id>" into the Meilisearch filter expression and escapes the value to defeat filter injection. Raw Meilisearch SDK access is not re-exported.
• Graceful degradation: every hot-path is wrapped in searchWithFallback, which routes to the legacy SQL LIKE implementation when Meilisearch raises a connectivity error. The search stays degraded-but-up instead of returning 500s.

Index Schema Reference

The current schemas live in packages/search/src/schemas/. Each is a whitelist — only fields listed in allowedDocumentFields reach Meilisearch.

Index	Searchable	Filterable	Sortable
kb-articles	title, summary, content, tags	tenantId, knowledgeBaseId, language, isPublished, tags	createdAt, updatedAt, publishedAt
crm-contacts	firstName, lastName, email, company, jobTitle, tags	tenantId, status, ownerId, tags	createdAt, updatedAt, lastInteractionAt
rows	searchable, folio, entityName	tenantId, entityId, entityName, createdById	createdAt, updatedAt, folio
entities	name, title, titlePlural	tenantId, type	name, createdAt, updatedAt
users	firstName, lastName, email	tenantId, type, admin, active	createdAt, lastLoginAt

tenantId is filterable on every index (the package refuses to create one otherwise).

Tuning Tips

• Ranking rules default to Meilisearch's recommendation: words → typo → proximity → attribute → sort → exactness. Per-tenant experimentation can shorten the chain to favour recency: words → typo → sort → attribute → exactness.
• Synonyms are per-index in the schema. Add them sparingly — they expand every query and inflate index size.
• Stop-words for full-text content (German der/die/das, English the/a/an) lift relevance noticeably; add them to the schema (stopWords).
• Highlighting: pass attributesToHighlight through searchInIndex({ options: { ... } }) to receive <em>-wrapped snippets in the response.

Hot-Path Migrations

Path	Migrated	Notes
KB public search (KnowledgeBaseService.search)	yes (#48)	wraps findKnowledgeBaseArticles via searchKnowledgeBaseArticles helper with searchWithFallback. Tenant id resolved from KB record.
CRM contact helper	shipped	apps/web/app/modules/crm/services/crmContactSearch.server.ts wraps crm-contacts with searchWithFallback and a tenant-scoped SQL fallback.
Rows helper	shipped	apps/web/app/modules/rows/services/rowSearch.server.ts wraps the rows index with an entityId filter and tenant-scoped SQL fallback.
CRM/Rows producer and callsite integration	yes	RowsApi.getAll calls searchCrmContacts for tenant-scoped CRM contact-list free-text filters and searchRows for other tenant-scoped row filters. RowsApi.create / createSimple / update / del emit search.index.* events through rowSearchOutbox.server.ts. Contact rows fan out to both crm-contacts and rows.
Bulk reindex pagers	partial	apps/worker/src/index.ts registers createSearchIndexerBulkPagers({ prisma }); BULK_REINDEX_SUPPORTED_INDEX_NAMES exposes kb-articles, crm-contacts, rows, and users. Entities still need an explicit pager before --index=all can rebuild every projected index.

Open hot-path gaps still serve from the legacy SQL path. Until a specific producer emits index events, a specific callsite reads through the helper, and the indexer has the relevant backfill pager, relevance, typo-tolerance and disaster-recovery reindex coverage remain partial for that path. Rows and CRM contacts have read/CUD producer pieces plus worker bulk-reindex pagers wired, but row indexing is still published post-write rather than inside the row mutation transaction. Entities still require a dedicated bulk pager; entity definitions are global platform-catalog records without a source tenantId, so their tenant-scoped projection needs an explicit design decision before production backfills include them.

CLI Operations

# Reindex every supported bulk-reindex index for every tenant via outbox events.
# Today this means KB articles, CRM contacts, rows, and users; unsupported names are rejected.
pnpm cli search reindex --tenant=all --index=all

# Reindex one tenant's KB articles.
pnpm cli search reindex --tenant=tnt_abc123 --index=kb-articles

The CLI never writes to Meilisearch directly. It produces search.index.<entity>.bulkreindex outbox events; the worker-general service consumes them and pages over Postgres through registered bulkPagers. Today, kb-articles, crm-contacts, rows, and users are listed in BULK_REINDEX_SUPPORTED_INDEX_NAMES; the CLI and worker enforce that contract so operators cannot enqueue unsupported bulk jobs by accident. Operators should not rely on full-index backfills for entities until its pager is added. A full disaster-recovery rebuild estimate can only be validated after that pager is present in the target environment.

Rollback

A two-layer rollback strategy is built into the design:

1. Feature flag (SEARCH_BACKEND=sql) forces every hot-path onto the SQL fallback immediately. Worker indexing continues but is not read from. Flip the env in production, redeploy the web tier, done.
2. Disable the worker by setting SEARCH_INDEXER_ENABLED=false on the worker-general service. Outbox events queue up in Postgres (Outbox retention is handled by the cleanup job from phase 03) and resume when re-enabled — followed by a search reindex to catch up any drift.

Performance Expectations

• P95 search latency under 50 ms for index sizes up to 1 M documents on a 1 vCPU / 2 GiB Meilisearch instance.
• Indexer throughput around 10 k documents per minute per worker-general replica (CPU-bound — scale horizontally if needed).
• Disk: roughly 30 MiB per 100 k documents on the KB-articles index, 20 MiB per 100 k contacts, less for the entity/user indexes.

Disaster Recovery

The Meilisearch volume is treated as cache. To rebuild from scratch:

1. docker volume rm meilisearch-data (after stopping the container).
2. Restart the container.
3. pnpm cli search reindex --tenant=all --index=all.

This rebuild currently covers the supported bulk-reindex set only: kb-articles, crm-contacts, rows, and users. Entities require a dedicated bulk pager before they are part of the disaster-recovery backfill. Expected time-to-recovery on a single replica: <30 minutes for 1 M KB-article documents.