Track HiveGrid Cloud planning reference
This commit is contained in:
david committed
1 parent
b3d5ea6764
commit
e341c5938a
5784 files changed
+1104929
-3
No files matched your search
@@ -0,0 +1,187 @@
|
||||
# RAG Pipeline — Embeddings, Indexer, Retrieval
|
||||
|
||||
The RAG pipeline turns domain content (KB-Articles today, CRM-Notes and
|
||||
embeddable rows next) into a tenant-scoped semantic-search store. The
|
||||
`retrieveRelevantContext` helper exists, and Chatbot V2 has an
|
||||
`enrichWithRAG` adapter, but production stream/tool wiring is still pending.
|
||||
|
||||
This document covers the runtime architecture, the embedding-provider
|
||||
matrix, chunking + cost trade-offs, tenant-isolation guarantees, and the
|
||||
backfill / disaster-recovery story.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
+-----------------+
|
||||
KB CRUD ──► | publishOutbox | (inside Prisma $transaction)
|
||||
| rag.index.* |
|
||||
+--------+--------+
|
||||
│
|
||||
▼
|
||||
outbox.rag-index (BullMQ)
|
||||
│
|
||||
▼
|
||||
+-----------------+
|
||||
| rag-indexer | (apps/worker)
|
||||
| |
|
||||
| chunk ──────► splitForEmbedding (cl100k)
|
||||
| embed ──────► EmbeddingProvider (provider-agnostic)
|
||||
| upsert ─────► VectorStore.upsertVectors (tenant-scoped)
|
||||
+-----------------+
|
||||
|
||||
Query path:
|
||||
query string ──► retrieveRelevantContext({tenantId, query, collections})
|
||||
│
|
||||
├── embedSingle(query) via the same provider
|
||||
├── searchForTenant(...) fan-out across collections
|
||||
├── rank-merge by score
|
||||
└── minScore filter + topK cap → snippets + sources
|
||||
```
|
||||
|
||||
CRM notes and embeddable rows are planned sources, but they do not emit
|
||||
`rag.index.*` events yet. The current production producer surface is
|
||||
KB-article create/update/delete.
|
||||
|
||||
Modules in play:
|
||||
|
||||
- **`packages/ai/src/embeddings/`** — provider abstraction, chunker,
|
||||
`embedAndUpsert`, `retrieveRelevantContext`, outbox publisher.
|
||||
- **`apps/worker/src/jobs/rag-indexer.ts`** — BullMQ consumer for
|
||||
`outbox.rag-index` events. Tenant scope enforced from envelope to
|
||||
payload to vector-store call.
|
||||
- **`apps/worker/src/outbox/event-router.ts`** — routes `rag.index.*`
|
||||
events to the `outbox.rag-index` queue.
|
||||
- **`packages/cli/src/commands/rag/`** — `toolctrl-cli rag reindex`
|
||||
walks registered source tables and writes upsert events into the outbox.
|
||||
The current registry includes KB articles only.
|
||||
|
||||
## Provider matrix
|
||||
|
||||
| Provider | Default model | Dimensions | EU region | Cost (per 1M tokens) | When to pick |
|
||||
|---|---|---|---|---|---|
|
||||
| `openai` | `text-embedding-3-small` | 1536 | No (US) | ~$0.02 | Cheapest + most documented. NOT suitable for tenants with hard EU residency. |
|
||||
| `scaleway` | `bge-multilingual-gemma2` | 1024 | Yes (FR/NL) | ~€0.08 (managed inference) | **Default for EU-residency**. OpenAI-compatible payload, hosted in EU. |
|
||||
| `aleph-alpha` | `luminous-base` | 5120 | Yes (DE) | varies (enterprise tiers) | German-residency / sovereignty requirements. Higher dim = higher storage + retrieval cost. |
|
||||
| `anthropic` | — | — | — | — | Placeholder — no first-party endpoint at writing time. Adapter throws `EMBEDDING_PROVIDER_NOT_SUPPORTED` with a hint. |
|
||||
|
||||
Selecting the provider is a runtime decision driven by `AI_EMBEDDING_PROVIDER`.
|
||||
See the env block in `.env.example` for every option.
|
||||
|
||||
## Tenant isolation
|
||||
|
||||
Tenant scope is enforced at four layers, each independent of the others:
|
||||
|
||||
1. **Outbox publisher** — `publishRagUpsertEvent` requires `tenantId` as
|
||||
a typed argument and stamps it into the outbox `tenantId` column.
|
||||
2. **Worker payload parsing** — `parseRagIndexJob` rejects events whose
|
||||
payload `tenantId` is missing or empty (`MISSING_TENANT` error).
|
||||
3. **`embedAndUpsert`** — writes `tenantId` into every vector payload
|
||||
and explicitly drops a caller-supplied `extraPayload.tenantId`.
|
||||
4. **`retrieveRelevantContext`** — `tenantId` is a required argument;
|
||||
it is forwarded to `searchForTenant`, which injects a `must` filter
|
||||
clause that callers cannot bypass.
|
||||
|
||||
Cross-tenant queries return zero hits because the vector store wrapper
|
||||
appends the `tenantId` filter regardless of caller intent.
|
||||
|
||||
## Chunking strategy
|
||||
|
||||
`splitForEmbedding` (in `packages/ai/src/embeddings/chunking.ts`) is a
|
||||
token-aware splitter:
|
||||
|
||||
- Token counting via `gpt-tokenizer` (cl100k BPE).
|
||||
- Sentence-boundary splits where possible; hard cut as fallback for
|
||||
super-long sentences.
|
||||
- Defaults: 500 tokens per chunk, 50 tokens overlap.
|
||||
- The worker indexer reads `RAG_CHUNK_SIZE_TOKENS` and
|
||||
`RAG_CHUNK_OVERLAP_TOKENS` and passes them into `embedAndUpsert`.
|
||||
Direct SDK/CLI callers can pass options via `chunkingOptions`.
|
||||
|
||||
Rule of thumb:
|
||||
|
||||
- **Knowledge-base articles**: 500 / 50 is right.
|
||||
- **CRM-notes** (short): 300 / 30 — recall is higher because notes are
|
||||
rarely long enough to need chunking.
|
||||
- **Long rows / FAQ**: 800 / 80 — fewer chunks, better contextual
|
||||
retrieval at the cost of slightly less precise hits.
|
||||
|
||||
## Cost management
|
||||
|
||||
Embedding cost scales linearly with token count. Defensive measures
|
||||
already in place:
|
||||
|
||||
- **Async pipeline** — the CLI never embeds directly; the worker
|
||||
controls concurrency (`RAG_INDEXER_CONCURRENCY`, default 2).
|
||||
- **Backfill events use `dispatchKey: rag:<source>:<id>:backfill`** so
|
||||
re-running `rag:reindex` is idempotent against an unfinished pass.
|
||||
- **Per-chunk dedupe** — upsert deletes prior chunk ids first, so a
|
||||
shrinking article does not accumulate orphan vectors.
|
||||
|
||||
A dedicated cost-tracking task (Phase 05 Task #54-#56) replaces the
|
||||
current 2-concurrency cap with per-tenant token budgets. Until then,
|
||||
keep `RAG_INDEXER_CONCURRENCY=2` and watch the provider dashboard
|
||||
during the first backfill on each new tenant.
|
||||
|
||||
## Disaster recovery
|
||||
|
||||
The outbox is the durable log; the vector store is the projection. The
|
||||
recovery sequence is therefore:
|
||||
|
||||
1. **Wipe the affected collection** with `vectorStore.deleteCollection()`.
|
||||
2. **Run `pnpm cli rag reindex --tenant=<id|all> --source=<slug|all>`**.
|
||||
3. **Observe** the worker logs / BullMQ dashboard until `outbox.rag-index`
|
||||
drains.
|
||||
|
||||
For a provider switch (e.g. OpenAI → Scaleway because of an outage):
|
||||
|
||||
1. Update `AI_EMBEDDING_PROVIDER` + matching credential env on the
|
||||
worker pod.
|
||||
2. Restart the worker.
|
||||
3. If dimensions differ, delete the collection (step 1 above) and
|
||||
re-run `rag:reindex` — pgvector / Qdrant raise
|
||||
`EMBEDDING_DIMENSION_MISMATCH` otherwise.
|
||||
|
||||
## Adding a new source-type
|
||||
|
||||
1. Implement a `RagBackfillSource` in `packages/cli/src/commands/rag/sources/`.
|
||||
2. Add it to the registry in `reindex.ts` (default: include via `all`).
|
||||
3. In the domain service, call `publishRagUpsertEvent` / `publishRagDeleteEvent`
|
||||
inside the existing Prisma `$transaction` callback. The KB-Articles
|
||||
helper in `apps/web/app/modules/knowledgeBase/services/kbArticleRag.server.ts`
|
||||
is the reference implementation.
|
||||
4. Decide on the source-type slug (kebab-case) — it becomes the prefix of
|
||||
every chunk id (`<sourceType>:<sourceId>:<chunkIndex>`) and the dispatch
|
||||
key.
|
||||
|
||||
## Onboarding checklist for a new tenant
|
||||
|
||||
- Confirm the embedding provider env is set on both `api` and `worker`.
|
||||
- Choose the RAG collection slug (default `rag-kb-articles` for the
|
||||
knowledge base).
|
||||
- Run `pnpm cli rag reindex --tenant=<new-tenant-id>` after the first
|
||||
KB-Article import. The worker drains in the background.
|
||||
- Verify with `retrieveRelevantContext({ tenantId, query, collections })`
|
||||
using a known phrase from one of the articles.
|
||||
|
||||
## Observability
|
||||
|
||||
Every job emits structured logs with `component: "rag-indexer"`, action,
|
||||
collection, tenantId, sourceType, sourceId, chunks affected and (where
|
||||
the provider reports it) input token count. Source text is never logged.
|
||||
|
||||
Failure modes:
|
||||
|
||||
- `EMBEDDING_PROVIDER_RATE_LIMITED` — BullMQ retries automatically.
|
||||
- `EMBEDDING_DIMENSION_MISMATCH` — non-retriable; surface in the
|
||||
dead-letter queue and run the disaster-recovery sequence above.
|
||||
- `RagIndexerJobError: MISSING_TENANT` — bug in the upstream
|
||||
publisher; events lacking `tenantId` are rejected at parse time.
|
||||
|
||||
## See also
|
||||
|
||||
- `docs/improvement-plan/04-search-rag/embedding-pipeline.md` — the
|
||||
specification + acceptance-criteria checklist this implementation
|
||||
satisfies.
|
||||
- `docs/features/search.md` — the Meilisearch full-text counterpart;
|
||||
the two pipelines share the same outbox-driven worker pattern.
|
||||
- `docs/architecture/vector-store.md` — pgvector vs Qdrant trade-offs.
|
||||
Reference in new issue
Block a user