152 lines
10 KiB
Markdown
152 lines
10 KiB
Markdown
# Full-Text Search
|
|
|
|
This document describes the Meilisearch-backed search subsystem introduced
|
|
in phase 04 (#48). It is the user-facing complement to
|
|
`docs/improvement-plan/04-search-rag/meilisearch.md` (technical spec) and
|
|
`docs/deploy/meilisearch.md` (operator runbook).
|
|
|
|
## Architecture
|
|
|
|
```
|
|
+------------------+ +--------------------+ +-----------------------+
|
|
| Domain service | --> | OutboxEvent (DB) | --> | dispatcher → BullMQ |
|
|
| (KB / CRM / …) | | type = search.… | | outbox.search-index |
|
|
+------------------+ +--------------------+ +-----------+-----------+
|
|
|
|
|
v
|
|
+--------------------------+
|
|
| search-indexer (general) |
|
|
| upsert / delete / bulk |
|
|
+-------------+------------+
|
|
|
|
|
v
|
|
+-----------------+
|
|
| Meilisearch |
|
|
| (tenantId |
|
|
| isolated) |
|
|
+-----------------+
|
|
```
|
|
|
|
- **Source of truth** is Postgres. Meilisearch is a projected read-side
|
|
populated asynchronously through the outbox pattern (#42).
|
|
- **Tenant isolation** is enforced inside `@toolctrl/search`. Every read
|
|
and write goes through `searchInIndex / bulkIndex / deleteFromIndex`
|
|
with an explicit `tenantId`. The wrapper injects
|
|
`tenantId = "<id>"` into the Meilisearch filter expression and
|
|
escapes the value to defeat filter injection. Raw Meilisearch SDK
|
|
access is not re-exported.
|
|
- **Graceful degradation:** every hot-path is wrapped in
|
|
`searchWithFallback`, which routes to the legacy SQL `LIKE`
|
|
implementation when Meilisearch raises a connectivity error. The
|
|
search stays degraded-but-up instead of returning 500s.
|
|
|
|
## Index Schema Reference
|
|
|
|
The current schemas live in `packages/search/src/schemas/`. Each is a
|
|
whitelist — only fields listed in `allowedDocumentFields` reach
|
|
Meilisearch.
|
|
|
|
| Index | Searchable | Filterable | Sortable |
|
|
| -------------- | --------------------------------------------------- | ------------------------------------------------------ | --------------------------------------- |
|
|
| `kb-articles` | title, summary, content, tags | tenantId, knowledgeBaseId, language, isPublished, tags | createdAt, updatedAt, publishedAt |
|
|
| `crm-contacts` | firstName, lastName, email, company, jobTitle, tags | tenantId, status, ownerId, tags | createdAt, updatedAt, lastInteractionAt |
|
|
| `rows` | searchable, folio, entityName | tenantId, entityId, entityName, createdById | createdAt, updatedAt, folio |
|
|
| `entities` | name, title, titlePlural | tenantId, type | name, createdAt, updatedAt |
|
|
| `users` | firstName, lastName, email | tenantId, type, admin, active | createdAt, lastLoginAt |
|
|
|
|
`tenantId` is filterable on every index (the package refuses to create
|
|
one otherwise).
|
|
|
|
## Tuning Tips
|
|
|
|
- **Ranking rules** default to Meilisearch's recommendation:
|
|
`words → typo → proximity → attribute → sort → exactness`. Per-tenant
|
|
experimentation can shorten the chain to favour recency:
|
|
`words → typo → sort → attribute → exactness`.
|
|
- **Synonyms** are per-index in the schema. Add them sparingly — they
|
|
expand every query and inflate index size.
|
|
- **Stop-words** for full-text content (German `der/die/das`,
|
|
English `the/a/an`) lift relevance noticeably; add them to the
|
|
schema (`stopWords`).
|
|
- **Highlighting**: pass `attributesToHighlight` through
|
|
`searchInIndex({ options: { ... } })` to receive `<em>`-wrapped
|
|
snippets in the response.
|
|
|
|
## Hot-Path Migrations
|
|
|
|
| Path | Migrated | Notes |
|
|
| ------------------------------------------------ | --------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
| KB public search (`KnowledgeBaseService.search`) | yes (#48) | wraps `findKnowledgeBaseArticles` via `searchKnowledgeBaseArticles` helper with `searchWithFallback`. Tenant id resolved from KB record. |
|
|
| CRM contact helper | shipped | `apps/web/app/modules/crm/services/crmContactSearch.server.ts` wraps `crm-contacts` with `searchWithFallback` and a tenant-scoped SQL fallback. |
|
|
| Rows helper | shipped | `apps/web/app/modules/rows/services/rowSearch.server.ts` wraps the `rows` index with an `entityId` filter and tenant-scoped SQL fallback. |
|
|
| CRM/Rows producer and callsite integration | yes | `RowsApi.getAll` calls `searchCrmContacts` for tenant-scoped CRM contact-list free-text filters and `searchRows` for other tenant-scoped row filters. `RowsApi.create / createSimple / update / del` emit `search.index.*` events through `rowSearchOutbox.server.ts`. Contact rows fan out to both `crm-contacts` and `rows`. |
|
|
| Bulk reindex pagers | partial | `apps/worker/src/index.ts` registers `createSearchIndexerBulkPagers({ prisma })`; `BULK_REINDEX_SUPPORTED_INDEX_NAMES` exposes `kb-articles`, `crm-contacts`, `rows`, and `users`. Entities still need an explicit pager before `--index=all` can rebuild every projected index. |
|
|
|
|
Open hot-path gaps still serve from the legacy SQL path. Until a specific
|
|
producer emits index events, a specific callsite reads through the helper, and
|
|
the indexer has the relevant backfill pager, relevance, typo-tolerance and
|
|
disaster-recovery reindex coverage remain partial for that path. Rows and CRM
|
|
contacts have read/CUD producer pieces plus worker bulk-reindex pagers wired,
|
|
but row indexing is still published post-write rather than inside the row
|
|
mutation transaction. Entities still require a dedicated bulk pager; entity
|
|
definitions are global platform-catalog records without a source `tenantId`, so
|
|
their tenant-scoped projection needs an explicit design decision before
|
|
production backfills include them.
|
|
|
|
## CLI Operations
|
|
|
|
```bash
|
|
# Reindex every supported bulk-reindex index for every tenant via outbox events.
|
|
# Today this means KB articles, CRM contacts, rows, and users; unsupported names are rejected.
|
|
pnpm cli search reindex --tenant=all --index=all
|
|
|
|
# Reindex one tenant's KB articles.
|
|
pnpm cli search reindex --tenant=tnt_abc123 --index=kb-articles
|
|
```
|
|
|
|
The CLI never writes to Meilisearch directly. It produces
|
|
`search.index.<entity>.bulkreindex` outbox events; the worker-general
|
|
service consumes them and pages over Postgres through registered
|
|
`bulkPagers`. Today, `kb-articles`, `crm-contacts`, `rows`, and `users` are listed in
|
|
`BULK_REINDEX_SUPPORTED_INDEX_NAMES`; the CLI and worker enforce that contract
|
|
so operators cannot enqueue unsupported bulk jobs by accident. Operators should
|
|
not rely on full-index backfills for entities until its pager is added. A full
|
|
disaster-recovery rebuild estimate can only be validated after that pager is
|
|
present in the target environment.
|
|
|
|
## Rollback
|
|
|
|
A two-layer rollback strategy is built into the design:
|
|
|
|
1. **Feature flag (`SEARCH_BACKEND=sql`)** forces every hot-path onto
|
|
the SQL fallback immediately. Worker indexing continues but is not
|
|
read from. Flip the env in production, redeploy the web tier, done.
|
|
2. **Disable the worker** by setting `SEARCH_INDEXER_ENABLED=false` on
|
|
the `worker-general` service. Outbox events queue up in Postgres
|
|
(Outbox retention is handled by the cleanup job from phase 03) and
|
|
resume when re-enabled — followed by a `search reindex` to catch up
|
|
any drift.
|
|
|
|
## Performance Expectations
|
|
|
|
- P95 search latency under 50 ms for index sizes up to 1 M documents
|
|
on a 1 vCPU / 2 GiB Meilisearch instance.
|
|
- Indexer throughput around 10 k documents per minute per
|
|
`worker-general` replica (CPU-bound — scale horizontally if needed).
|
|
- Disk: roughly 30 MiB per 100 k documents on the KB-articles index,
|
|
20 MiB per 100 k contacts, less for the entity/user indexes.
|
|
|
|
## Disaster Recovery
|
|
|
|
The Meilisearch volume is treated as cache. To rebuild from scratch:
|
|
|
|
1. `docker volume rm meilisearch-data` (after stopping the container).
|
|
2. Restart the container.
|
|
3. `pnpm cli search reindex --tenant=all --index=all`.
|
|
|
|
This rebuild currently covers the supported bulk-reindex set only:
|
|
`kb-articles`, `crm-contacts`, `rows`, and `users`. Entities require a
|
|
dedicated bulk pager before they are part of the disaster-recovery backfill.
|
|
Expected time-to-recovery on a single replica: <30 minutes for 1 M KB-article
|
|
documents.
|