Track HiveGrid Cloud planning reference

hivegrid-cloud/docs/architecture/vector-store.md

@@ -0,0 +1,139 @@
+# Vector Store
+
+`@toolctrl/vector-store` is a thin adapter layer over two interchangeable
+nearest-neighbour stores. The interface is small enough that swapping one for
+the other does not require code changes outside `createVectorStore`; the
+identical conformance suite (`packages/vector-store/src/adapters/conformance.test.ts`)
+guarantees behavioural parity.
+
+## Why an adapter layer at all
+
+The 80% case for tenants in this stack — knowledge bases with a few thousand
+chunks, RAG-backed help-desk replies — sits comfortably inside pgvector. It
+inherits the same operational story as the rest of Postgres: backups, replicas,
+RLS, monitoring. No extra container, no extra credentials, no extra firewall
+rule.
+
+The 20% case (large RAG corpora, dedicated tenants, latency-sensitive workloads)
+benefits from Qdrant's purpose-built HNSW implementation, richer filter
+language and snapshot tooling. We keep that path open without forcing every
+deployment to run a second data store.
+
+## Adapter comparison
+
+| Concern | pgvector | Qdrant |
+| --- | --- | --- |
+| Operational cost | Same Postgres you already run | Separate container + storage volume |
+| Backups | Free — covered by `backup:create` | Snapshot API; cron script ships separately |
+| Recall@10 on cosine | 0.95+ once HNSW is built | 0.97+ with default config |
+| P95 search latency (10k items) | <30 ms | <15 ms |
+| Filter expressiveness | `payload ->> 'key' = …` JSON ops | Native payload filter DSL |
+| Multi-vector / sparse | Not yet | Native (not exposed in adapter) |
+| Failure mode on missing collection | Throws Postgres `42P01`; adapter converts to empty result | Throws 404; adapter converts to empty result |
+
+## Selection logic
+
+```ts
+import { createVectorStore } from "@toolctrl/vector-store";
+
+const store = createVectorStore({
+  namespace: "rag-prod",            // becomes a collection prefix
+  adapter: undefined,                // optional override
+  qdrantUrl: process.env.QDRANT_URL, // forward env at the boundary
+  qdrantApiKey: process.env.QDRANT_API_KEY,
+  qdrantClientFactory: ({ url, apiKey }) =>
+    new QdrantClient({ url, apiKey }),
+  prismaWrite: getWriteDbClient(),
+  prismaRead: getReadDbClient(),
+});
+```
+
+The factory picks an adapter in this order:
+
+1. `adapter` parameter when supplied (test seam, dual-running migrations).
+2. `qdrantUrl` truthy → Qdrant. Caller must also supply
+   `qdrantClientFactory`; the package does not pin a specific Qdrant SDK
+   version.
+3. Otherwise pgvector with `prismaWrite` / `prismaRead`.
+
+The package never reads `process.env` directly — it would break the
+`env-access-enforcement` boundary test. Callers thread env values in.
+
+## Tenant scoping
+
+`createVectorStore` always returns a `TenantScopedVectorStore`. The bare
+`searchVectors` method throws on this wrapper: searches must go through
+`searchForTenant({ tenantId, ... })`. The wrapper injects the tenantId as the
+first `must` filter clause and discards any caller-supplied `tenantId` clause
+before forwarding the call, so a buggy or hostile caller cannot escape the
+boundary.
+
+Writes are guarded at the type level: `VectorItem.payload` extends
+`{ tenantId: string }`, so omitting the tenant id at upsert time is a TS
+compile error.
+
+## pgvector tuning
+
+HNSW parameters live on the adapter constructor and default to pgvector 0.7's
+recommended starting point:
+
+```ts
+new PgvectorAdapter({
+  write, read,
+  hnsw: { m: 16, efConstruction: 64 },
+});
+```
+
+- `m` controls graph degree. Bump to 24 for collections >500k vectors when
+  recall lags.
+- `efConstruction` is the candidate-list size during index build. Bump to 128
+  for higher-recall corpora; build time roughly doubles.
+
+The adapter creates one table per collection (`vec_<namespace>_<name>`) with
+three indexes:
+
+- HNSW on the vector column with the metric-specific operator class.
+- GIN on the `payload` JSONB column for filter selectivity.
+- B-tree on the generated `tenant_id` column for the (almost always)
+  tenant-scoped read path.
+
+## Qdrant deployment notes
+
+- Pin to a 1.x release. Adapter is tested against `1.12`.
+- API key is mandatory whenever the URL is not localhost. The factory's caller
+  should reject `QDRANT_URL` without `QDRANT_API_KEY` at the boundary.
+- Use a dedicated PVC; Qdrant cannot recover from a wiped storage directory.
+- The adapter auto-creates collections on first upsert. Dimension mismatches
+  surface as `VectorDimensionMismatchError` so the embedding pipeline can
+  classify them for retry vs. dead-letter.
+
+## Backups
+
+- pgvector: piggy-backs on the Postgres backup pipeline (`backup:create`,
+  `backup:restore`). No additional work.
+- Qdrant: use the Snapshot API. A dedicated cron job + S3 upload script will
+  ship with task #50; until then run manual snapshots before each schema
+  rollout.
+
+## Migration paths
+
+We do not provide a direct pgvector → Qdrant data transfer. Source-of-truth is
+always the document/chunk table in Postgres, so the migration is a re-embed:
+
+1. Deploy with `VECTOR_STORE_DEFAULT_ADAPTER=qdrant` and `QDRANT_URL` set.
+2. Run `rag:reindex` (ships with #50). Embeddings are recomputed via the
+   embedding provider and written into Qdrant.
+3. Smoke-search to confirm parity, then drop the pgvector tables via
+   `vector-store:gc` once the warm-window is clear.
+
+Same procedure in reverse, with the obvious caveat that pgvector imposes a
+practical ceiling around 1M vectors per collection before HNSW build cost
+becomes painful.
+
+## Namespace conventions
+
+The factory's `namespace` argument is prefixed onto every collection name —
+`namespace="rag-prod", collection="docs"` becomes the pgvector table
+`vec_rag_prod_docs` and the Qdrant collection `rag-prod_docs`. Pick a
+namespace per workspace or per environment so multiple deployments can share
+the same store without colliding.