Track HiveGrid Cloud planning reference
This commit is contained in:
david committed
1 parent
b3d5ea6764
commit
e341c5938a
5784 files changed
+1104929
-3
No files matched your search
@@ -0,0 +1,151 @@
|
||||
# Full-Text Search
|
||||
|
||||
This document describes the Meilisearch-backed search subsystem introduced
|
||||
in phase 04 (#48). It is the user-facing complement to
|
||||
`docs/improvement-plan/04-search-rag/meilisearch.md` (technical spec) and
|
||||
`docs/deploy/meilisearch.md` (operator runbook).
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
+------------------+ +--------------------+ +-----------------------+
|
||||
| Domain service | --> | OutboxEvent (DB) | --> | dispatcher → BullMQ |
|
||||
| (KB / CRM / …) | | type = search.… | | outbox.search-index |
|
||||
+------------------+ +--------------------+ +-----------+-----------+
|
||||
|
|
||||
v
|
||||
+--------------------------+
|
||||
| search-indexer (general) |
|
||||
| upsert / delete / bulk |
|
||||
+-------------+------------+
|
||||
|
|
||||
v
|
||||
+-----------------+
|
||||
| Meilisearch |
|
||||
| (tenantId |
|
||||
| isolated) |
|
||||
+-----------------+
|
||||
```
|
||||
|
||||
- **Source of truth** is Postgres. Meilisearch is a projected read-side
|
||||
populated asynchronously through the outbox pattern (#42).
|
||||
- **Tenant isolation** is enforced inside `@toolctrl/search`. Every read
|
||||
and write goes through `searchInIndex / bulkIndex / deleteFromIndex`
|
||||
with an explicit `tenantId`. The wrapper injects
|
||||
`tenantId = "<id>"` into the Meilisearch filter expression and
|
||||
escapes the value to defeat filter injection. Raw Meilisearch SDK
|
||||
access is not re-exported.
|
||||
- **Graceful degradation:** every hot-path is wrapped in
|
||||
`searchWithFallback`, which routes to the legacy SQL `LIKE`
|
||||
implementation when Meilisearch raises a connectivity error. The
|
||||
search stays degraded-but-up instead of returning 500s.
|
||||
|
||||
## Index Schema Reference
|
||||
|
||||
The current schemas live in `packages/search/src/schemas/`. Each is a
|
||||
whitelist — only fields listed in `allowedDocumentFields` reach
|
||||
Meilisearch.
|
||||
|
||||
| Index | Searchable | Filterable | Sortable |
|
||||
| -------------- | --------------------------------------------------- | ------------------------------------------------------ | --------------------------------------- |
|
||||
| `kb-articles` | title, summary, content, tags | tenantId, knowledgeBaseId, language, isPublished, tags | createdAt, updatedAt, publishedAt |
|
||||
| `crm-contacts` | firstName, lastName, email, company, jobTitle, tags | tenantId, status, ownerId, tags | createdAt, updatedAt, lastInteractionAt |
|
||||
| `rows` | searchable, folio, entityName | tenantId, entityId, entityName, createdById | createdAt, updatedAt, folio |
|
||||
| `entities` | name, title, titlePlural | tenantId, type | name, createdAt, updatedAt |
|
||||
| `users` | firstName, lastName, email | tenantId, type, admin, active | createdAt, lastLoginAt |
|
||||
|
||||
`tenantId` is filterable on every index (the package refuses to create
|
||||
one otherwise).
|
||||
|
||||
## Tuning Tips
|
||||
|
||||
- **Ranking rules** default to Meilisearch's recommendation:
|
||||
`words → typo → proximity → attribute → sort → exactness`. Per-tenant
|
||||
experimentation can shorten the chain to favour recency:
|
||||
`words → typo → sort → attribute → exactness`.
|
||||
- **Synonyms** are per-index in the schema. Add them sparingly — they
|
||||
expand every query and inflate index size.
|
||||
- **Stop-words** for full-text content (German `der/die/das`,
|
||||
English `the/a/an`) lift relevance noticeably; add them to the
|
||||
schema (`stopWords`).
|
||||
- **Highlighting**: pass `attributesToHighlight` through
|
||||
`searchInIndex({ options: { ... } })` to receive `<em>`-wrapped
|
||||
snippets in the response.
|
||||
|
||||
## Hot-Path Migrations
|
||||
|
||||
| Path | Migrated | Notes |
|
||||
| ------------------------------------------------ | --------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| KB public search (`KnowledgeBaseService.search`) | yes (#48) | wraps `findKnowledgeBaseArticles` via `searchKnowledgeBaseArticles` helper with `searchWithFallback`. Tenant id resolved from KB record. |
|
||||
| CRM contact helper | shipped | `apps/web/app/modules/crm/services/crmContactSearch.server.ts` wraps `crm-contacts` with `searchWithFallback` and a tenant-scoped SQL fallback. |
|
||||
| Rows helper | shipped | `apps/web/app/modules/rows/services/rowSearch.server.ts` wraps the `rows` index with an `entityId` filter and tenant-scoped SQL fallback. |
|
||||
| CRM/Rows producer and callsite integration | yes | `RowsApi.getAll` calls `searchCrmContacts` for tenant-scoped CRM contact-list free-text filters and `searchRows` for other tenant-scoped row filters. `RowsApi.create / createSimple / update / del` emit `search.index.*` events through `rowSearchOutbox.server.ts`. Contact rows fan out to both `crm-contacts` and `rows`. |
|
||||
| Bulk reindex pagers | partial | `apps/worker/src/index.ts` registers `createSearchIndexerBulkPagers({ prisma })`; `BULK_REINDEX_SUPPORTED_INDEX_NAMES` exposes `kb-articles`, `crm-contacts`, `rows`, and `users`. Entities still need an explicit pager before `--index=all` can rebuild every projected index. |
|
||||
|
||||
Open hot-path gaps still serve from the legacy SQL path. Until a specific
|
||||
producer emits index events, a specific callsite reads through the helper, and
|
||||
the indexer has the relevant backfill pager, relevance, typo-tolerance and
|
||||
disaster-recovery reindex coverage remain partial for that path. Rows and CRM
|
||||
contacts have read/CUD producer pieces plus worker bulk-reindex pagers wired,
|
||||
but row indexing is still published post-write rather than inside the row
|
||||
mutation transaction. Entities still require a dedicated bulk pager; entity
|
||||
definitions are global platform-catalog records without a source `tenantId`, so
|
||||
their tenant-scoped projection needs an explicit design decision before
|
||||
production backfills include them.
|
||||
|
||||
## CLI Operations
|
||||
|
||||
```bash
|
||||
# Reindex every supported bulk-reindex index for every tenant via outbox events.
|
||||
# Today this means KB articles, CRM contacts, rows, and users; unsupported names are rejected.
|
||||
pnpm cli search reindex --tenant=all --index=all
|
||||
|
||||
# Reindex one tenant's KB articles.
|
||||
pnpm cli search reindex --tenant=tnt_abc123 --index=kb-articles
|
||||
```
|
||||
|
||||
The CLI never writes to Meilisearch directly. It produces
|
||||
`search.index.<entity>.bulkreindex` outbox events; the worker-general
|
||||
service consumes them and pages over Postgres through registered
|
||||
`bulkPagers`. Today, `kb-articles`, `crm-contacts`, `rows`, and `users` are listed in
|
||||
`BULK_REINDEX_SUPPORTED_INDEX_NAMES`; the CLI and worker enforce that contract
|
||||
so operators cannot enqueue unsupported bulk jobs by accident. Operators should
|
||||
not rely on full-index backfills for entities until its pager is added. A full
|
||||
disaster-recovery rebuild estimate can only be validated after that pager is
|
||||
present in the target environment.
|
||||
|
||||
## Rollback
|
||||
|
||||
A two-layer rollback strategy is built into the design:
|
||||
|
||||
1. **Feature flag (`SEARCH_BACKEND=sql`)** forces every hot-path onto
|
||||
the SQL fallback immediately. Worker indexing continues but is not
|
||||
read from. Flip the env in production, redeploy the web tier, done.
|
||||
2. **Disable the worker** by setting `SEARCH_INDEXER_ENABLED=false` on
|
||||
the `worker-general` service. Outbox events queue up in Postgres
|
||||
(Outbox retention is handled by the cleanup job from phase 03) and
|
||||
resume when re-enabled — followed by a `search reindex` to catch up
|
||||
any drift.
|
||||
|
||||
## Performance Expectations
|
||||
|
||||
- P95 search latency under 50 ms for index sizes up to 1 M documents
|
||||
on a 1 vCPU / 2 GiB Meilisearch instance.
|
||||
- Indexer throughput around 10 k documents per minute per
|
||||
`worker-general` replica (CPU-bound — scale horizontally if needed).
|
||||
- Disk: roughly 30 MiB per 100 k documents on the KB-articles index,
|
||||
20 MiB per 100 k contacts, less for the entity/user indexes.
|
||||
|
||||
## Disaster Recovery
|
||||
|
||||
The Meilisearch volume is treated as cache. To rebuild from scratch:
|
||||
|
||||
1. `docker volume rm meilisearch-data` (after stopping the container).
|
||||
2. Restart the container.
|
||||
3. `pnpm cli search reindex --tenant=all --index=all`.
|
||||
|
||||
This rebuild currently covers the supported bulk-reindex set only:
|
||||
`kb-articles`, `crm-contacts`, `rows`, and `users`. Entities require a
|
||||
dedicated bulk pager before they are part of the disaster-recovery backfill.
|
||||
Expected time-to-recovery on a single replica: <30 minutes for 1 M KB-article
|
||||
documents.
|
||||
Reference in new issue
Block a user