# RAG Pipeline — Embeddings, Indexer, Retrieval

The RAG pipeline turns domain content (KB-Articles today, CRM-Notes and
embeddable rows next) into a tenant-scoped semantic-search store. The
`retrieveRelevantContext` helper exists, and Chatbot V2 has an
`enrichWithRAG` adapter, but production stream/tool wiring is still pending.

This document covers the runtime architecture, the embedding-provider
matrix, chunking + cost trade-offs, tenant-isolation guarantees, and the
backfill / disaster-recovery story.

## Architecture

```
                  +-----------------+
   KB CRUD ──►    | publishOutbox   | (inside Prisma $transaction)
                  |   rag.index.*   |
                  +--------+--------+
                           │
                           ▼
                  outbox.rag-index (BullMQ)
                           │
                           ▼
                  +-----------------+
                  | rag-indexer     | (apps/worker)
                  |                 |
                  |  chunk ──────► splitForEmbedding (cl100k)
                  |  embed ──────► EmbeddingProvider (provider-agnostic)
                  |  upsert ─────► VectorStore.upsertVectors (tenant-scoped)
                  +-----------------+

   Query path:
   query string ──► retrieveRelevantContext({tenantId, query, collections})
                       │
                       ├── embedSingle(query) via the same provider
                       ├── searchForTenant(...) fan-out across collections
                       ├── rank-merge by score
                       └── minScore filter + topK cap → snippets + sources
```

CRM notes and embeddable rows are planned sources, but they do not emit
`rag.index.*` events yet. The current production producer surface is
KB-article create/update/delete.

Modules in play:

- **`packages/ai/src/embeddings/`** — provider abstraction, chunker,
  `embedAndUpsert`, `retrieveRelevantContext`, outbox publisher.
- **`apps/worker/src/jobs/rag-indexer.ts`** — BullMQ consumer for
  `outbox.rag-index` events. Tenant scope enforced from envelope to
  payload to vector-store call.
- **`apps/worker/src/outbox/event-router.ts`** — routes `rag.index.*`
  events to the `outbox.rag-index` queue.
- **`packages/cli/src/commands/rag/`** — `toolctrl-cli rag reindex`
  walks registered source tables and writes upsert events into the outbox.
  The current registry includes KB articles only.

## Provider matrix

| Provider | Default model | Dimensions | EU region | Cost (per 1M tokens) | When to pick |
|---|---|---|---|---|---|
| `openai` | `text-embedding-3-small` | 1536 | No (US) | ~$0.02 | Cheapest + most documented. NOT suitable for tenants with hard EU residency. |
| `scaleway` | `bge-multilingual-gemma2` | 1024 | Yes (FR/NL) | ~€0.08 (managed inference) | **Default for EU-residency**. OpenAI-compatible payload, hosted in EU. |
| `aleph-alpha` | `luminous-base` | 5120 | Yes (DE) | varies (enterprise tiers) | German-residency / sovereignty requirements. Higher dim = higher storage + retrieval cost. |
| `anthropic` | — | — | — | — | Placeholder — no first-party endpoint at writing time. Adapter throws `EMBEDDING_PROVIDER_NOT_SUPPORTED` with a hint. |

Selecting the provider is a runtime decision driven by `AI_EMBEDDING_PROVIDER`.
See the env block in `.env.example` for every option.

## Tenant isolation

Tenant scope is enforced at four layers, each independent of the others:

1. **Outbox publisher** — `publishRagUpsertEvent` requires `tenantId` as
   a typed argument and stamps it into the outbox `tenantId` column.
2. **Worker payload parsing** — `parseRagIndexJob` rejects events whose
   payload `tenantId` is missing or empty (`MISSING_TENANT` error).
3. **`embedAndUpsert`** — writes `tenantId` into every vector payload
   and explicitly drops a caller-supplied `extraPayload.tenantId`.
4. **`retrieveRelevantContext`** — `tenantId` is a required argument;
   it is forwarded to `searchForTenant`, which injects a `must` filter
   clause that callers cannot bypass.

Cross-tenant queries return zero hits because the vector store wrapper
appends the `tenantId` filter regardless of caller intent.

## Chunking strategy

`splitForEmbedding` (in `packages/ai/src/embeddings/chunking.ts`) is a
token-aware splitter:

- Token counting via `gpt-tokenizer` (cl100k BPE).
- Sentence-boundary splits where possible; hard cut as fallback for
  super-long sentences.
- Defaults: 500 tokens per chunk, 50 tokens overlap.
- The worker indexer reads `RAG_CHUNK_SIZE_TOKENS` and
  `RAG_CHUNK_OVERLAP_TOKENS` and passes them into `embedAndUpsert`.
  Direct SDK/CLI callers can pass options via `chunkingOptions`.

Rule of thumb:

- **Knowledge-base articles**: 500 / 50 is right.
- **CRM-notes** (short): 300 / 30 — recall is higher because notes are
  rarely long enough to need chunking.
- **Long rows / FAQ**: 800 / 80 — fewer chunks, better contextual
  retrieval at the cost of slightly less precise hits.

## Cost management

Embedding cost scales linearly with token count. Defensive measures
already in place:

- **Async pipeline** — the CLI never embeds directly; the worker
  controls concurrency (`RAG_INDEXER_CONCURRENCY`, default 2).
- **Backfill events use `dispatchKey: rag:<source>:<id>:backfill`** so
  re-running `rag:reindex` is idempotent against an unfinished pass.
- **Per-chunk dedupe** — upsert deletes prior chunk ids first, so a
  shrinking article does not accumulate orphan vectors.

A dedicated cost-tracking task (Phase 05 Task #54-#56) replaces the
current 2-concurrency cap with per-tenant token budgets. Until then,
keep `RAG_INDEXER_CONCURRENCY=2` and watch the provider dashboard
during the first backfill on each new tenant.

## Disaster recovery

The outbox is the durable log; the vector store is the projection. The
recovery sequence is therefore:

1. **Wipe the affected collection** with `vectorStore.deleteCollection()`.
2. **Run `pnpm cli rag reindex --tenant=<id|all> --source=<slug|all>`**.
3. **Observe** the worker logs / BullMQ dashboard until `outbox.rag-index`
   drains.

For a provider switch (e.g. OpenAI → Scaleway because of an outage):

1. Update `AI_EMBEDDING_PROVIDER` + matching credential env on the
   worker pod.
2. Restart the worker.
3. If dimensions differ, delete the collection (step 1 above) and
   re-run `rag:reindex` — pgvector / Qdrant raise
   `EMBEDDING_DIMENSION_MISMATCH` otherwise.

## Adding a new source-type

1. Implement a `RagBackfillSource` in `packages/cli/src/commands/rag/sources/`.
2. Add it to the registry in `reindex.ts` (default: include via `all`).
3. In the domain service, call `publishRagUpsertEvent` / `publishRagDeleteEvent`
   inside the existing Prisma `$transaction` callback. The KB-Articles
   helper in `apps/web/app/modules/knowledgeBase/services/kbArticleRag.server.ts`
   is the reference implementation.
4. Decide on the source-type slug (kebab-case) — it becomes the prefix of
   every chunk id (`<sourceType>:<sourceId>:<chunkIndex>`) and the dispatch
   key.

## Onboarding checklist for a new tenant

- Confirm the embedding provider env is set on both `api` and `worker`.
- Choose the RAG collection slug (default `rag-kb-articles` for the
  knowledge base).
- Run `pnpm cli rag reindex --tenant=<new-tenant-id>` after the first
  KB-Article import. The worker drains in the background.
- Verify with `retrieveRelevantContext({ tenantId, query, collections })`
  using a known phrase from one of the articles.

## Observability

Every job emits structured logs with `component: "rag-indexer"`, action,
collection, tenantId, sourceType, sourceId, chunks affected and (where
the provider reports it) input token count. Source text is never logged.

Failure modes:

- `EMBEDDING_PROVIDER_RATE_LIMITED` — BullMQ retries automatically.
- `EMBEDDING_DIMENSION_MISMATCH` — non-retriable; surface in the
  dead-letter queue and run the disaster-recovery sequence above.
- `RagIndexerJobError: MISSING_TENANT` — bug in the upstream
  publisher; events lacking `tenantId` are rejected at parse time.

## See also

- `docs/improvement-plan/04-search-rag/embedding-pipeline.md` — the
  specification + acceptance-criteria checklist this implementation
  satisfies.
- `docs/features/search.md` — the Meilisearch full-text counterpart;
  the two pipelines share the same outbox-driven worker pattern.
- `docs/architecture/vector-store.md` — pgvector vs Qdrant trade-offs.