RAG Pipeline — Embeddings, Indexer, Retrieval

The RAG pipeline turns domain content (KB-Articles today, CRM-Notes and embeddable rows next) into a tenant-scoped semantic-search store. The retrieveRelevantContext helper exists, and Chatbot V2 has an enrichWithRAG adapter, but production stream/tool wiring is still pending.

This document covers the runtime architecture, the embedding-provider matrix, chunking + cost trade-offs, tenant-isolation guarantees, and the backfill / disaster-recovery story.

Architecture

                  +-----------------+
   KB CRUD ──►    | publishOutbox   | (inside Prisma $transaction)
                  |   rag.index.*   |
                  +--------+--------+
                           │
                           ▼
                  outbox.rag-index (BullMQ)
                           │
                           ▼
                  +-----------------+
                  | rag-indexer     | (apps/worker)
                  |                 |
                  |  chunk ──────► splitForEmbedding (cl100k)
                  |  embed ──────► EmbeddingProvider (provider-agnostic)
                  |  upsert ─────► VectorStore.upsertVectors (tenant-scoped)
                  +-----------------+

   Query path:
   query string ──► retrieveRelevantContext({tenantId, query, collections})
                       │
                       ├── embedSingle(query) via the same provider
                       ├── searchForTenant(...) fan-out across collections
                       ├── rank-merge by score
                       └── minScore filter + topK cap → snippets + sources

CRM notes and embeddable rows are planned sources, but they do not emit rag.index.* events yet. The current production producer surface is KB-article create/update/delete.

Modules in play:

packages/ai/src/embeddings/ — provider abstraction, chunker, embedAndUpsert, retrieveRelevantContext, outbox publisher.
apps/worker/src/jobs/rag-indexer.ts — BullMQ consumer for outbox.rag-index events. Tenant scope enforced from envelope to payload to vector-store call.
apps/worker/src/outbox/event-router.ts — routes rag.index.* events to the outbox.rag-index queue.
packages/cli/src/commands/rag/ — toolctrl-cli rag reindex walks registered source tables and writes upsert events into the outbox. The current registry includes KB articles only.

Provider matrix

Provider	Default model	Dimensions	EU region	Cost (per 1M tokens)	When to pick
`openai`	`text-embedding-3-small`	1536	No (US)	~$0.02	Cheapest + most documented. NOT suitable for tenants with hard EU residency.
`scaleway`	`bge-multilingual-gemma2`	1024	Yes (FR/NL)	~€0.08 (managed inference)	Default for EU-residency. OpenAI-compatible payload, hosted in EU.
`aleph-alpha`	`luminous-base`	5120	Yes (DE)	varies (enterprise tiers)	German-residency / sovereignty requirements. Higher dim = higher storage + retrieval cost.
`anthropic`	—	—	—	—	Placeholder — no first-party endpoint at writing time. Adapter throws `EMBEDDING_PROVIDER_NOT_SUPPORTED` with a hint.

Selecting the provider is a runtime decision driven by AI_EMBEDDING_PROVIDER. See the env block in .env.example for every option.

Tenant isolation

Tenant scope is enforced at four layers, each independent of the others:

Outbox publisher — publishRagUpsertEvent requires tenantId as a typed argument and stamps it into the outbox tenantId column.
Worker payload parsing — parseRagIndexJob rejects events whose payload tenantId is missing or empty (MISSING_TENANT error).
embedAndUpsert — writes tenantId into every vector payload and explicitly drops a caller-supplied extraPayload.tenantId.
retrieveRelevantContext — tenantId is a required argument; it is forwarded to searchForTenant, which injects a must filter clause that callers cannot bypass.

Cross-tenant queries return zero hits because the vector store wrapper appends the tenantId filter regardless of caller intent.

Chunking strategy

splitForEmbedding (in packages/ai/src/embeddings/chunking.ts) is a token-aware splitter:

Token counting via gpt-tokenizer (cl100k BPE).
Sentence-boundary splits where possible; hard cut as fallback for super-long sentences.
Defaults: 500 tokens per chunk, 50 tokens overlap.
The worker indexer reads RAG_CHUNK_SIZE_TOKENS and RAG_CHUNK_OVERLAP_TOKENS and passes them into embedAndUpsert. Direct SDK/CLI callers can pass options via chunkingOptions.

Rule of thumb:

Knowledge-base articles: 500 / 50 is right.
CRM-notes (short): 300 / 30 — recall is higher because notes are rarely long enough to need chunking.
Long rows / FAQ: 800 / 80 — fewer chunks, better contextual retrieval at the cost of slightly less precise hits.

Cost management

Embedding cost scales linearly with token count. Defensive measures already in place:

Async pipeline — the CLI never embeds directly; the worker controls concurrency (RAG_INDEXER_CONCURRENCY, default 2).
Backfill events use dispatchKey: rag:<source>:<id>:backfill so re-running rag:reindex is idempotent against an unfinished pass.
Per-chunk dedupe — upsert deletes prior chunk ids first, so a shrinking article does not accumulate orphan vectors.

A dedicated cost-tracking task (Phase 05 Task #54-#56) replaces the current 2-concurrency cap with per-tenant token budgets. Until then, keep RAG_INDEXER_CONCURRENCY=2 and watch the provider dashboard during the first backfill on each new tenant.

Disaster recovery

The outbox is the durable log; the vector store is the projection. The recovery sequence is therefore:

Wipe the affected collection with vectorStore.deleteCollection().
Run pnpm cli rag reindex --tenant=<id|all> --source=<slug|all>.
Observe the worker logs / BullMQ dashboard until outbox.rag-index drains.

For a provider switch (e.g. OpenAI → Scaleway because of an outage):

Update AI_EMBEDDING_PROVIDER + matching credential env on the worker pod.
Restart the worker.
If dimensions differ, delete the collection (step 1 above) and re-run rag:reindex — pgvector / Qdrant raise EMBEDDING_DIMENSION_MISMATCH otherwise.

Adding a new source-type

Implement a RagBackfillSource in packages/cli/src/commands/rag/sources/.
Add it to the registry in reindex.ts (default: include via all).
In the domain service, call publishRagUpsertEvent / publishRagDeleteEvent inside the existing Prisma $transaction callback. The KB-Articles helper in apps/web/app/modules/knowledgeBase/services/kbArticleRag.server.ts is the reference implementation.
Decide on the source-type slug (kebab-case) — it becomes the prefix of every chunk id (<sourceType>:<sourceId>:<chunkIndex>) and the dispatch key.

Onboarding checklist for a new tenant

Confirm the embedding provider env is set on both api and worker.
Choose the RAG collection slug (default rag-kb-articles for the knowledge base).
Run pnpm cli rag reindex --tenant=<new-tenant-id> after the first KB-Article import. The worker drains in the background.
Verify with retrieveRelevantContext({ tenantId, query, collections }) using a known phrase from one of the articles.

Observability

Every job emits structured logs with component: "rag-indexer", action, collection, tenantId, sourceType, sourceId, chunks affected and (where the provider reports it) input token count. Source text is never logged.

Failure modes:

EMBEDDING_PROVIDER_RATE_LIMITED — BullMQ retries automatically.
EMBEDDING_DIMENSION_MISMATCH — non-retriable; surface in the dead-letter queue and run the disaster-recovery sequence above.
RagIndexerJobError: MISSING_TENANT — bug in the upstream publisher; events lacking tenantId are rejected at parse time.

8.4 KiB Raw Blame History