# Full-Text Search This document describes the Meilisearch-backed search subsystem introduced in phase 04 (#48). It is the user-facing complement to `docs/improvement-plan/04-search-rag/meilisearch.md` (technical spec) and `docs/deploy/meilisearch.md` (operator runbook). ## Architecture ``` +------------------+ +--------------------+ +-----------------------+ | Domain service | --> | OutboxEvent (DB) | --> | dispatcher → BullMQ | | (KB / CRM / …) | | type = search.… | | outbox.search-index | +------------------+ +--------------------+ +-----------+-----------+ | v +--------------------------+ | search-indexer (general) | | upsert / delete / bulk | +-------------+------------+ | v +-----------------+ | Meilisearch | | (tenantId | | isolated) | +-----------------+ ``` - **Source of truth** is Postgres. Meilisearch is a projected read-side populated asynchronously through the outbox pattern (#42). - **Tenant isolation** is enforced inside `@toolctrl/search`. Every read and write goes through `searchInIndex / bulkIndex / deleteFromIndex` with an explicit `tenantId`. The wrapper injects `tenantId = ""` into the Meilisearch filter expression and escapes the value to defeat filter injection. Raw Meilisearch SDK access is not re-exported. - **Graceful degradation:** every hot-path is wrapped in `searchWithFallback`, which routes to the legacy SQL `LIKE` implementation when Meilisearch raises a connectivity error. The search stays degraded-but-up instead of returning 500s. ## Index Schema Reference The current schemas live in `packages/search/src/schemas/`. Each is a whitelist — only fields listed in `allowedDocumentFields` reach Meilisearch. | Index | Searchable | Filterable | Sortable | | -------------- | --------------------------------------------------- | ------------------------------------------------------ | --------------------------------------- | | `kb-articles` | title, summary, content, tags | tenantId, knowledgeBaseId, language, isPublished, tags | createdAt, updatedAt, publishedAt | | `crm-contacts` | firstName, lastName, email, company, jobTitle, tags | tenantId, status, ownerId, tags | createdAt, updatedAt, lastInteractionAt | | `rows` | searchable, folio, entityName | tenantId, entityId, entityName, createdById | createdAt, updatedAt, folio | | `entities` | name, title, titlePlural | tenantId, type | name, createdAt, updatedAt | | `users` | firstName, lastName, email | tenantId, type, admin, active | createdAt, lastLoginAt | `tenantId` is filterable on every index (the package refuses to create one otherwise). ## Tuning Tips - **Ranking rules** default to Meilisearch's recommendation: `words → typo → proximity → attribute → sort → exactness`. Per-tenant experimentation can shorten the chain to favour recency: `words → typo → sort → attribute → exactness`. - **Synonyms** are per-index in the schema. Add them sparingly — they expand every query and inflate index size. - **Stop-words** for full-text content (German `der/die/das`, English `the/a/an`) lift relevance noticeably; add them to the schema (`stopWords`). - **Highlighting**: pass `attributesToHighlight` through `searchInIndex({ options: { ... } })` to receive ``-wrapped snippets in the response. ## Hot-Path Migrations | Path | Migrated | Notes | | ------------------------------------------------ | --------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | KB public search (`KnowledgeBaseService.search`) | yes (#48) | wraps `findKnowledgeBaseArticles` via `searchKnowledgeBaseArticles` helper with `searchWithFallback`. Tenant id resolved from KB record. | | CRM contact helper | shipped | `apps/web/app/modules/crm/services/crmContactSearch.server.ts` wraps `crm-contacts` with `searchWithFallback` and a tenant-scoped SQL fallback. | | Rows helper | shipped | `apps/web/app/modules/rows/services/rowSearch.server.ts` wraps the `rows` index with an `entityId` filter and tenant-scoped SQL fallback. | | CRM/Rows producer and callsite integration | yes | `RowsApi.getAll` calls `searchCrmContacts` for tenant-scoped CRM contact-list free-text filters and `searchRows` for other tenant-scoped row filters. `RowsApi.create / createSimple / update / del` emit `search.index.*` events through `rowSearchOutbox.server.ts`. Contact rows fan out to both `crm-contacts` and `rows`. | | Bulk reindex pagers | partial | `apps/worker/src/index.ts` registers `createSearchIndexerBulkPagers({ prisma })`; `BULK_REINDEX_SUPPORTED_INDEX_NAMES` exposes `kb-articles`, `crm-contacts`, `rows`, and `users`. Entities still need an explicit pager before `--index=all` can rebuild every projected index. | Open hot-path gaps still serve from the legacy SQL path. Until a specific producer emits index events, a specific callsite reads through the helper, and the indexer has the relevant backfill pager, relevance, typo-tolerance and disaster-recovery reindex coverage remain partial for that path. Rows and CRM contacts have read/CUD producer pieces plus worker bulk-reindex pagers wired, but row indexing is still published post-write rather than inside the row mutation transaction. Entities still require a dedicated bulk pager; entity definitions are global platform-catalog records without a source `tenantId`, so their tenant-scoped projection needs an explicit design decision before production backfills include them. ## CLI Operations ```bash # Reindex every supported bulk-reindex index for every tenant via outbox events. # Today this means KB articles, CRM contacts, rows, and users; unsupported names are rejected. pnpm cli search reindex --tenant=all --index=all # Reindex one tenant's KB articles. pnpm cli search reindex --tenant=tnt_abc123 --index=kb-articles ``` The CLI never writes to Meilisearch directly. It produces `search.index..bulkreindex` outbox events; the worker-general service consumes them and pages over Postgres through registered `bulkPagers`. Today, `kb-articles`, `crm-contacts`, `rows`, and `users` are listed in `BULK_REINDEX_SUPPORTED_INDEX_NAMES`; the CLI and worker enforce that contract so operators cannot enqueue unsupported bulk jobs by accident. Operators should not rely on full-index backfills for entities until its pager is added. A full disaster-recovery rebuild estimate can only be validated after that pager is present in the target environment. ## Rollback A two-layer rollback strategy is built into the design: 1. **Feature flag (`SEARCH_BACKEND=sql`)** forces every hot-path onto the SQL fallback immediately. Worker indexing continues but is not read from. Flip the env in production, redeploy the web tier, done. 2. **Disable the worker** by setting `SEARCH_INDEXER_ENABLED=false` on the `worker-general` service. Outbox events queue up in Postgres (Outbox retention is handled by the cleanup job from phase 03) and resume when re-enabled — followed by a `search reindex` to catch up any drift. ## Performance Expectations - P95 search latency under 50 ms for index sizes up to 1 M documents on a 1 vCPU / 2 GiB Meilisearch instance. - Indexer throughput around 10 k documents per minute per `worker-general` replica (CPU-bound — scale horizontally if needed). - Disk: roughly 30 MiB per 100 k documents on the KB-articles index, 20 MiB per 100 k contacts, less for the entity/user indexes. ## Disaster Recovery The Meilisearch volume is treated as cache. To rebuild from scratch: 1. `docker volume rm meilisearch-data` (after stopping the container). 2. Restart the container. 3. `pnpm cli search reindex --tenant=all --index=all`. This rebuild currently covers the supported bulk-reindex set only: `kb-articles`, `crm-contacts`, `rows`, and `users`. Entities require a dedicated bulk pager before they are part of the disaster-recovery backfill. Expected time-to-recovery on a single replica: <30 minutes for 1 M KB-article documents.