188 lines
8.4 KiB
Markdown
188 lines
8.4 KiB
Markdown
# RAG Pipeline — Embeddings, Indexer, Retrieval
|
|
|
|
The RAG pipeline turns domain content (KB-Articles today, CRM-Notes and
|
|
embeddable rows next) into a tenant-scoped semantic-search store. The
|
|
`retrieveRelevantContext` helper exists, and Chatbot V2 has an
|
|
`enrichWithRAG` adapter, but production stream/tool wiring is still pending.
|
|
|
|
This document covers the runtime architecture, the embedding-provider
|
|
matrix, chunking + cost trade-offs, tenant-isolation guarantees, and the
|
|
backfill / disaster-recovery story.
|
|
|
|
## Architecture
|
|
|
|
```
|
|
+-----------------+
|
|
KB CRUD ──► | publishOutbox | (inside Prisma $transaction)
|
|
| rag.index.* |
|
|
+--------+--------+
|
|
│
|
|
▼
|
|
outbox.rag-index (BullMQ)
|
|
│
|
|
▼
|
|
+-----------------+
|
|
| rag-indexer | (apps/worker)
|
|
| |
|
|
| chunk ──────► splitForEmbedding (cl100k)
|
|
| embed ──────► EmbeddingProvider (provider-agnostic)
|
|
| upsert ─────► VectorStore.upsertVectors (tenant-scoped)
|
|
+-----------------+
|
|
|
|
Query path:
|
|
query string ──► retrieveRelevantContext({tenantId, query, collections})
|
|
│
|
|
├── embedSingle(query) via the same provider
|
|
├── searchForTenant(...) fan-out across collections
|
|
├── rank-merge by score
|
|
└── minScore filter + topK cap → snippets + sources
|
|
```
|
|
|
|
CRM notes and embeddable rows are planned sources, but they do not emit
|
|
`rag.index.*` events yet. The current production producer surface is
|
|
KB-article create/update/delete.
|
|
|
|
Modules in play:
|
|
|
|
- **`packages/ai/src/embeddings/`** — provider abstraction, chunker,
|
|
`embedAndUpsert`, `retrieveRelevantContext`, outbox publisher.
|
|
- **`apps/worker/src/jobs/rag-indexer.ts`** — BullMQ consumer for
|
|
`outbox.rag-index` events. Tenant scope enforced from envelope to
|
|
payload to vector-store call.
|
|
- **`apps/worker/src/outbox/event-router.ts`** — routes `rag.index.*`
|
|
events to the `outbox.rag-index` queue.
|
|
- **`packages/cli/src/commands/rag/`** — `toolctrl-cli rag reindex`
|
|
walks registered source tables and writes upsert events into the outbox.
|
|
The current registry includes KB articles only.
|
|
|
|
## Provider matrix
|
|
|
|
| Provider | Default model | Dimensions | EU region | Cost (per 1M tokens) | When to pick |
|
|
|---|---|---|---|---|---|
|
|
| `openai` | `text-embedding-3-small` | 1536 | No (US) | ~$0.02 | Cheapest + most documented. NOT suitable for tenants with hard EU residency. |
|
|
| `scaleway` | `bge-multilingual-gemma2` | 1024 | Yes (FR/NL) | ~€0.08 (managed inference) | **Default for EU-residency**. OpenAI-compatible payload, hosted in EU. |
|
|
| `aleph-alpha` | `luminous-base` | 5120 | Yes (DE) | varies (enterprise tiers) | German-residency / sovereignty requirements. Higher dim = higher storage + retrieval cost. |
|
|
| `anthropic` | — | — | — | — | Placeholder — no first-party endpoint at writing time. Adapter throws `EMBEDDING_PROVIDER_NOT_SUPPORTED` with a hint. |
|
|
|
|
Selecting the provider is a runtime decision driven by `AI_EMBEDDING_PROVIDER`.
|
|
See the env block in `.env.example` for every option.
|
|
|
|
## Tenant isolation
|
|
|
|
Tenant scope is enforced at four layers, each independent of the others:
|
|
|
|
1. **Outbox publisher** — `publishRagUpsertEvent` requires `tenantId` as
|
|
a typed argument and stamps it into the outbox `tenantId` column.
|
|
2. **Worker payload parsing** — `parseRagIndexJob` rejects events whose
|
|
payload `tenantId` is missing or empty (`MISSING_TENANT` error).
|
|
3. **`embedAndUpsert`** — writes `tenantId` into every vector payload
|
|
and explicitly drops a caller-supplied `extraPayload.tenantId`.
|
|
4. **`retrieveRelevantContext`** — `tenantId` is a required argument;
|
|
it is forwarded to `searchForTenant`, which injects a `must` filter
|
|
clause that callers cannot bypass.
|
|
|
|
Cross-tenant queries return zero hits because the vector store wrapper
|
|
appends the `tenantId` filter regardless of caller intent.
|
|
|
|
## Chunking strategy
|
|
|
|
`splitForEmbedding` (in `packages/ai/src/embeddings/chunking.ts`) is a
|
|
token-aware splitter:
|
|
|
|
- Token counting via `gpt-tokenizer` (cl100k BPE).
|
|
- Sentence-boundary splits where possible; hard cut as fallback for
|
|
super-long sentences.
|
|
- Defaults: 500 tokens per chunk, 50 tokens overlap.
|
|
- The worker indexer reads `RAG_CHUNK_SIZE_TOKENS` and
|
|
`RAG_CHUNK_OVERLAP_TOKENS` and passes them into `embedAndUpsert`.
|
|
Direct SDK/CLI callers can pass options via `chunkingOptions`.
|
|
|
|
Rule of thumb:
|
|
|
|
- **Knowledge-base articles**: 500 / 50 is right.
|
|
- **CRM-notes** (short): 300 / 30 — recall is higher because notes are
|
|
rarely long enough to need chunking.
|
|
- **Long rows / FAQ**: 800 / 80 — fewer chunks, better contextual
|
|
retrieval at the cost of slightly less precise hits.
|
|
|
|
## Cost management
|
|
|
|
Embedding cost scales linearly with token count. Defensive measures
|
|
already in place:
|
|
|
|
- **Async pipeline** — the CLI never embeds directly; the worker
|
|
controls concurrency (`RAG_INDEXER_CONCURRENCY`, default 2).
|
|
- **Backfill events use `dispatchKey: rag:<source>:<id>:backfill`** so
|
|
re-running `rag:reindex` is idempotent against an unfinished pass.
|
|
- **Per-chunk dedupe** — upsert deletes prior chunk ids first, so a
|
|
shrinking article does not accumulate orphan vectors.
|
|
|
|
A dedicated cost-tracking task (Phase 05 Task #54-#56) replaces the
|
|
current 2-concurrency cap with per-tenant token budgets. Until then,
|
|
keep `RAG_INDEXER_CONCURRENCY=2` and watch the provider dashboard
|
|
during the first backfill on each new tenant.
|
|
|
|
## Disaster recovery
|
|
|
|
The outbox is the durable log; the vector store is the projection. The
|
|
recovery sequence is therefore:
|
|
|
|
1. **Wipe the affected collection** with `vectorStore.deleteCollection()`.
|
|
2. **Run `pnpm cli rag reindex --tenant=<id|all> --source=<slug|all>`**.
|
|
3. **Observe** the worker logs / BullMQ dashboard until `outbox.rag-index`
|
|
drains.
|
|
|
|
For a provider switch (e.g. OpenAI → Scaleway because of an outage):
|
|
|
|
1. Update `AI_EMBEDDING_PROVIDER` + matching credential env on the
|
|
worker pod.
|
|
2. Restart the worker.
|
|
3. If dimensions differ, delete the collection (step 1 above) and
|
|
re-run `rag:reindex` — pgvector / Qdrant raise
|
|
`EMBEDDING_DIMENSION_MISMATCH` otherwise.
|
|
|
|
## Adding a new source-type
|
|
|
|
1. Implement a `RagBackfillSource` in `packages/cli/src/commands/rag/sources/`.
|
|
2. Add it to the registry in `reindex.ts` (default: include via `all`).
|
|
3. In the domain service, call `publishRagUpsertEvent` / `publishRagDeleteEvent`
|
|
inside the existing Prisma `$transaction` callback. The KB-Articles
|
|
helper in `apps/web/app/modules/knowledgeBase/services/kbArticleRag.server.ts`
|
|
is the reference implementation.
|
|
4. Decide on the source-type slug (kebab-case) — it becomes the prefix of
|
|
every chunk id (`<sourceType>:<sourceId>:<chunkIndex>`) and the dispatch
|
|
key.
|
|
|
|
## Onboarding checklist for a new tenant
|
|
|
|
- Confirm the embedding provider env is set on both `api` and `worker`.
|
|
- Choose the RAG collection slug (default `rag-kb-articles` for the
|
|
knowledge base).
|
|
- Run `pnpm cli rag reindex --tenant=<new-tenant-id>` after the first
|
|
KB-Article import. The worker drains in the background.
|
|
- Verify with `retrieveRelevantContext({ tenantId, query, collections })`
|
|
using a known phrase from one of the articles.
|
|
|
|
## Observability
|
|
|
|
Every job emits structured logs with `component: "rag-indexer"`, action,
|
|
collection, tenantId, sourceType, sourceId, chunks affected and (where
|
|
the provider reports it) input token count. Source text is never logged.
|
|
|
|
Failure modes:
|
|
|
|
- `EMBEDDING_PROVIDER_RATE_LIMITED` — BullMQ retries automatically.
|
|
- `EMBEDDING_DIMENSION_MISMATCH` — non-retriable; surface in the
|
|
dead-letter queue and run the disaster-recovery sequence above.
|
|
- `RagIndexerJobError: MISSING_TENANT` — bug in the upstream
|
|
publisher; events lacking `tenantId` are rejected at parse time.
|
|
|
|
## See also
|
|
|
|
- `docs/improvement-plan/04-search-rag/embedding-pipeline.md` — the
|
|
specification + acceptance-criteria checklist this implementation
|
|
satisfies.
|
|
- `docs/features/search.md` — the Meilisearch full-text counterpart;
|
|
the two pipelines share the same outbox-driven worker pattern.
|
|
- `docs/architecture/vector-store.md` — pgvector vs Qdrant trade-offs.
|