346 lines
33 KiB
Markdown
346 lines
33 KiB
Markdown
# Boilerplate-Inventur: Integrationen, KI & Apps-Engine
|
|
|
|
> Faktische Inventur des Boilerplate-Bereichs "Integrationen, KI & Apps-Engine".
|
|
> Quelle (NUR LESEN): `/Users/raffaelsheikh/Documents/GitHub/boilerplate`
|
|
> Erstellt fuer den Modul-Abgleich gegen HiveGrid 2.0. Alle Pfade absolut.
|
|
> Nichts erfunden — jeder Eintrag verweist auf konkrete Datei(en).
|
|
|
|
## Inhaltsverzeichnis / abgedeckte Pakete
|
|
|
|
| Paket | npm-Name | Zweck (kurz) |
|
|
|---|---|---|
|
|
| `packages/ai` | `@toolctrl/ai` | KI-Provider-Layer (Mistral/Scaleway/Aleph Alpha), Multi-Provider-Fallback, Embeddings/RAG-Pipeline, Cost-Tracking, Prompt-/Tool-Registry, Guards, Streaming |
|
|
| `packages/vector-store` | `@toolctrl/vector-store` | Vektor-Store-Adapter (pgvector + Qdrant) hinter einem Interface, tenant-scoped Wrapper |
|
|
| `packages/search` | `@toolctrl/search` | Typisierter Meilisearch-Wrapper, tenant-scoped Such-Index, Index-Schemas, Outbox-Publisher |
|
|
| `packages/plugins` | `@toolctrl/plugins` | Feature-Manifest-Schema + Loader (Apps-/Feature-Inventur-Manifeste) |
|
|
| `packages/plugin-runtime` | `@toolctrl/plugin-runtime` | WASM-Sandbox, Capability-Gateway, Manifest, Signatur, Hooks, Audit-Chain, Crash-Tracker, Migrations |
|
|
| `packages/plugin-sdk` | `@toolctrl/plugin-sdk` | Typdefinitionen + Settings-Iframe-Client fuer Plugin-Autoren (`defineHook`, `HostApi`) |
|
|
| `packages/storage` | `@toolctrl/storage` | S3/Scaleway-Storage-Adapter + lokaler Adapter, Signed-URLs, Virus-Scan (ClamAV/magic-bytes), Quarantaene |
|
|
| `packages/realtime` | `@toolctrl/realtime` | SSE/Realtime-Event-Serialisierung + Client/Server-Helfer (Notifications) |
|
|
| `packages/mail` | `@toolctrl/mail` | MailerSend-Provider, Queued-Mail-Provider, React-Email Transaktionstemplates |
|
|
| `packages/email-templates` | `@toolctrl/email-templates` | React-basierte Sequenz-Email-Templates (welcome, trial-reminder, re-activation) |
|
|
| `packages/email-sequences` | `@toolctrl/email-sequences` | Reine Logik: Sequenz-Engine, Condition-DSL, Step-Runner, Unsubscribe-Token |
|
|
| `packages/marketing-outbound` | `@toolctrl/marketing-outbound` | Outbound-Webhook-Adapter (Slack, Discord, Mailchimp, HubSpot, Brevo, Custom) |
|
|
| `packages/queue` | `@toolctrl/queue` | BullMQ-Wrapper: Queue-Namen, typisierte Producer/Jobs, Idempotenz, Prioritaeten, Redis-Connection |
|
|
|
|
Konvention durchgaengig: jedes Paket exportiert `exports` mit `.` -> `./src/index.ts` (Source-Exports, kein Build-Schritt zwingend), Scripts meist `test` / `typecheck` (teils `build`, `preview`). Pakete sind bewusst **dependency-arm** und lesen **nie `process.env` direkt** — Boundary-Code (apps/api, apps/worker) reicht env-Snapshots herein (Env-Access-Enforcement-Tests).
|
|
|
|
---
|
|
|
|
## 1. `@toolctrl/ai` — KI-Provider-Layer, Fallback, RAG, Cost
|
|
|
|
**Pfad:** `/Users/raffaelsheikh/Documents/GitHub/boilerplate/packages/ai`
|
|
**Deps:** `@ai-sdk/mistral`, `@ai-sdk/openai-compatible`, `ai` (Vercel AI SDK), `gpt-tokenizer`, `@toolctrl/entitlements`
|
|
**Subpath-Exports:** `.`, `./cost`, `./embeddings`, `./embeddings/providers/{openai,scaleway,aleph-alpha,anthropic}`
|
|
|
|
### 1.1 Provider-Adapter & Multi-Provider-Fallback
|
|
|
|
| Asset | Pfad | Was |
|
|
|---|---|---|
|
|
| Provider-Adapter-Interface + Registry | `packages/ai/src/index.ts` | `AIProviderAdapter`, `AIProviderName = "mistral" \| "scaleway" \| "aleph-alpha"`, `getAIProviderAdapter()`, `getAIProviderAdapterForModel()`, `getPrimaryProvider()`, `getAIProviderDiagnostics()`, Modell-Katalog `aiModels` (mit Preisen pro Mio Tokens), `defaultAIModel = "mistral-large-latest"`, `AIProviderAdapterError` mit typisierten Codes |
|
|
| Mistral-Provider | `packages/ai/src/providers/mistral.ts` | `createMistralLanguageModel()` via `@ai-sdk/mistral` `createMistral`, `resolveMistralModel(env)`, `MISTRAL_API_KEY`/`MISTRAL_MODEL` |
|
|
| Scaleway-Provider (OpenAI-kompatibel) | `packages/ai/src/providers/scaleway.ts` | `createScalewayLanguageModel()`, Default-BaseURL `https://api.scaleway.ai/v1`, Default-Modell `mistral-small-3.2-24b-instruct-2506`, `SCALEWAY_API_KEY`/`SCALEWAY_BASE_URL`/`SCALEWAY_MODEL` |
|
|
| Aleph-Alpha-Provider (OpenAI-kompatibel) | `packages/ai/src/providers/aleph-alpha.ts` | `createAlephAlphaLanguageModel()`, Default-BaseURL `https://api.aleph-alpha.com/v1`, Default-Modell `pharia-1-llm-7b-control`, `ALEPH_ALPHA_API_KEY`/`...BASE_URL`/`...MODEL` |
|
|
| OpenAI-Compatible-Bruecke | `packages/ai/src/providers/compatible.ts` | `createCompatibleLanguageModel()` ueber `@ai-sdk/openai-compatible` — gemeinsame Basis fuer Scaleway + Aleph Alpha |
|
|
| Health-Tracker + Fallback | `packages/ai/src/fallback.ts` | `AIProviderHealthTracker` (Failure-Window 30 min), `getFallbackProviderSelection()`, `fallbackProviderOrder = ["mistral","scaleway","aleph-alpha"]` — durchlaeuft Provider in fester Reihenfolge, ueberspringt ungesunde/unkonfigurierte |
|
|
|
|
**HiveGrid-Relevanz:** Kern fuer jedes KI-Modul (Chatbot, Prompt-Flows, Generieren, RAG-Antworten). EU-Provider Scaleway/Aleph Alpha = DSGVO-Argument. Multi-Provider-Fallback deckt Querschnitt "KI-Verfuegbarkeit/Resilienz" ab.
|
|
|
|
### 1.2 Embeddings-Pipeline (RAG-Schreib-/Leseseite)
|
|
|
|
**Pfad:** `packages/ai/src/embeddings` — Subpath `@toolctrl/ai/embeddings`
|
|
|
|
| Asset | Pfad | Was |
|
|
|---|---|---|
|
|
| Provider-Interface + Typen | `packages/ai/src/embeddings/types.ts` | `EmbeddingProvider` (`embed`/`embedSingle`, `dimensions`, `maxBatchSize`), `EmbeddingProviderName = "openai" \| "scaleway" \| "aleph-alpha" \| "anthropic"`, `EmbeddingProviderError` mit typisierten Codes, `EmbeddingProviderEnv`-Snapshot |
|
|
| Factory + Registry | `packages/ai/src/embeddings/factory.ts` | `createEmbeddingProvider()`, `registerEmbeddingProviderAdapter()` (Adapter registrieren sich selbst), `resolveEmbeddingProviderName(env)` (Default openai->scaleway->aleph-alpha), `resolveEmbeddingDimensions(env)` |
|
|
| Embed-Adapter (opt-in subpaths) | `packages/ai/src/embeddings/providers/{openai,scaleway,aleph-alpha,anthropic}.ts` | Scaleway-Default `bge-multilingual-gemma2` (1024 dim), Batch 64, max 4096 Tokens; Anthropic ist bewusst Stub/Platzhalter |
|
|
| Schreibseite | `packages/ai/src/embeddings/embed-and-upsert.ts` | `embedAndUpsert()` — chunked, embeds, upsert in `VectorUpsertSink`; erzwingt `tenantId` im Payload; deterministische `buildChunkId()`; optionale Kostenverfolgung ueber `executeLLMCall` (capability `embedding`) |
|
|
| Leseseite | `packages/ai/src/embeddings/retrieve.ts` | `retrieveRelevantContext()` — Query embedden, Fan-out ueber Collections, Rank-Merge, min-Score 0.7, top-K 8; tenant-scoped via `RagRetrievalStore.searchForTenant` |
|
|
| Chunking | `packages/ai/src/embeddings/chunking.ts` | `splitForEmbedding()`, `countTokens()`, `DEFAULT_CHUNK_TOKENS`/`...OVERLAP_TOKENS` |
|
|
| Outbox-Publisher (RAG-Index) | `packages/ai/src/embeddings/outbox-publisher.ts` | `publishRagUpsertEvent`/`publishRagDeleteEvent`, `ragEventType()` -> `rag.index.<entity>.<verb>` (mapped auf BullMQ-Queue `outbox.rag-index`), Idempotenz via `dispatchKey` |
|
|
|
|
**HiveGrid-Relevanz:** Komplette RAG-Grundlage (Knowledge-Base-Suche, KI-Antworten mit Quellen, semantische Suche). Outbox-Pattern erlaubt async Reindex bei jedem Domain-Write. Direkt nutzbar fuer HiveGrid-Module "Wissensdatenbank/RAG" und "KI-Assistent".
|
|
|
|
### 1.3 Cost-Tracking, Quota & Guards
|
|
|
|
| Asset | Pfad | Was |
|
|
|---|---|---|
|
|
| Cost-Barrel | `packages/ai/src/cost/index.ts` | Re-Export: `countTokens`, `countCallCost`, `executeLLMCall`, `estimateLLMCost`, Pricing-Tabelle (`getPricingTable`, `LATEST_PRICING_VERSION`), Spend-Cache, Quota |
|
|
| LLM-Wrapper | `packages/ai/src/cost/executeLLMCall.ts` | `executeLLMCall()` — wrappt jeden Provider-Call, leitet Token/Kosten ab, emittiert Outbox-Event `ai.cost.event.persist`, Quota-Pre-Check + Soft-Degradation + Concurrency-Slot; `setLLMCostPublisher/MetricsHook/ErrorHandler`; Flag `AI_COST_TRACKING_ENABLED=false` deaktiviert; capabilities `chat`/`chat-stream`/`embedding`/`completion`/`rerank`/`tool-call` |
|
|
| Quota-Guard | `packages/ai/src/cost/assertCredit.ts` | `assertTenantAICreditAccess()`, Concurrency-Counter (Redis-aehnlich), Soft-Degradation-Threshold |
|
|
| Spend-Cache | `packages/ai/src/cost/spend-cache.ts` | `createSpendCache()`, Monatsspend-Caching (TTL) |
|
|
| Token-Zaehlung | `packages/ai/src/cost/tokens.ts` | `countTokens`/`countMessageTokens` (gpt-tokenizer + Heuristik-Fallback) |
|
|
| Guards | `packages/ai/src/guards/index.ts` | `assertAICostBudget()`, `createFixedWindowAIRateLimiter()` (pro userId), `createAIAuditEvent()`/`redactAIAuditPayload()` (PII-Redaction, prompt/tool_call/mutation), `createReviewRequiredResult()`, `AIGuardError` |
|
|
|
|
**HiveGrid-Relevanz:** Querschnitt "Billing/Usage-Metering" (KI-Kosten pro Tenant), "Rate-Limiting", "Audit/DSGVO" (Redaction). Persistiert in Prisma `AICostEvent` (siehe unten). Direkt fuer HiveGrid-Modul "KI-Kontingente/Abrechnung".
|
|
|
|
### 1.4 Prompt-/Tool-Registry, Streaming, Eval
|
|
|
|
| Asset | Pfad | Was |
|
|
|---|---|---|
|
|
| Prompt-Registry | `packages/ai/src/prompts/index.ts` | `createAIPromptRegistry()` mit Versionierung, `render(selection, variables)`, `{{var}}`-Templates, `AIPromptRegistryError` |
|
|
| Tool-Registry | `packages/ai/src/tools/index.ts` | `createAIToolRegistry()` (scoped Tools fuer Vercel AI SDK `ToolSet`), `toToolSet(selection)`, Secret-Redaction in Fehlern |
|
|
| Streaming | `packages/ai/src/streaming/index.ts` | `createAITextStream()` ueber `streamText` (Vercel AI SDK), Guard-Evaluator, Prompt-/Tool-Registry-Integration, `getAITextStreamDiagnostics()` mit Redaction |
|
|
| Promptfoo-Provider + Evals | `packages/ai/src/promptfoo-provider.ts`, `packages/ai/test/evals/`, Repo-Root `promptfoo.config.yaml` | Eval-Harness (promptfoo-Baseline-Test) |
|
|
|
|
**HiveGrid-Relevanz:** "Prompt-Flows / Prompt-Bibliothek" (Versionierung), "KI-Tools/Agenten" (scoped Tool-Registry), Streaming-Chat-UX.
|
|
|
|
---
|
|
|
|
## 2. `@toolctrl/vector-store` — Vektor-Store-Adapter (pgvector + Qdrant)
|
|
|
|
**Pfad:** `/Users/raffaelsheikh/Documents/GitHub/boilerplate/packages/vector-store`
|
|
**Deps:** `@qdrant/js-client-rest`
|
|
**Subpath-Exports:** `.`, `./factory`, `./adapters/pgvector`, `./adapters/qdrant`, `./wrappers/tenant-scoped`
|
|
|
|
| Asset | Pfad | Was |
|
|
|---|---|---|
|
|
| Adapter-Interface + Typen | `packages/vector-store/src/types.ts` | `VectorStoreAdapter` (createCollection/upsert/search/delete/health), `VectorItem` mit Pflicht-`payload.tenantId`, `VectorFilter`/`FilterClause` (Qdrant-Filter-Sprache), `VectorMetric = "cosine"\|"euclidean"\|"dot"`, `VectorDimensionMismatchError` |
|
|
| Factory (Adapter-Auswahl) | `packages/vector-store/src/factory.ts` | `createVectorStore()` -> waehlt pgvector (Default) oder Qdrant (`adapter:` oder `qdrantUrl`); `resolveAdapter()`; Namespace-Pflicht; gibt tenant-scoped Store zurueck |
|
|
| pgvector-Adapter | `packages/vector-store/src/adapters/pgvector.ts` | `PgvectorAdapter` ueber `PrismaLike` (`$queryRawUnsafe`/`$executeRawUnsafe`), erstellt `vec_<namespace>_<collection>`-Tabellen, HNSW-Index (`m`, `efConstruction`), Payload-Filter via `payload ->> 'key'` |
|
|
| Qdrant-Adapter | `packages/vector-store/src/adapters/qdrant.ts` | `QdrantAdapter` ueber `QdrantClientLike`, Namespace-Praefix |
|
|
| Tenant-Scope-Wrapper | `packages/vector-store/src/wrappers/tenant-scoped.ts` | `withTenantScope()` -> `TenantScopedVectorStore`; `searchForTenant()` injiziert `tenantId`-must-Filter zwingend; bare `searchVectors()` wirft; `listVectorIds()` scoped nach tenant/sourceType/sourceId |
|
|
| Konformanztest | `packages/vector-store/src/adapters/conformance.test.ts` | Erzwingt byte-fuer-byte gleiches Verhalten beider Adapter |
|
|
|
|
**Prisma:** Kein Prisma-Modell — pgvector legt eigene `vec_*`-Tabellen via Raw-SQL an. Migration: `prisma/migrations/20260517085146_04_enable_pgvector/migration.sql` (aktiviert `pgvector`-Extension).
|
|
|
|
**HiveGrid-Relevanz:** Speicher-Backend fuer RAG/semantische Suche. pgvector = "kein Extra-Service" (laeuft auf Haupt-Postgres), Qdrant = Upgrade-Pfad. Tenant-Isolation built-in. Direkt fuer HiveGrid "RAG"/"semantische Suche".
|
|
|
|
---
|
|
|
|
## 3. `@toolctrl/search` — Meilisearch Such-Index
|
|
|
|
**Pfad:** `/Users/raffaelsheikh/Documents/GitHub/boilerplate/packages/search`
|
|
**Deps:** `meilisearch`, `zod`
|
|
**Subpath-Exports:** `.`, `./schemas`, `./testing`
|
|
|
|
| Asset | Pfad | Was |
|
|
|---|---|---|
|
|
| Client-Adapter | `packages/search/src/client.ts` | `createSearchClient()` — kapselt Meilisearch-SDK (nie re-exportiert), `SearchClient`/`IndexHandle` (ensure/search/upsert/delete/deleteByFilter), `meiliFactory`-Injection fuer Tests |
|
|
| Query-Layer (tenant-scoped) | `packages/search/src/query.ts` | `searchInIndex`, `searchWithFallback`, `bulkIndex`, `deleteFromIndex`, `health`, `TenantScopeError` — jede Suche erzwingt `tenantId`-Filter |
|
|
| Index-Helper | `packages/search/src/index-helper.ts` | `createSearchIndex()`, `validateIndexSchema()`, `IndexSchemaValidationError` |
|
|
| Index-Schemas | `packages/search/src/schemas/` (`kb-articles.ts`, `crm-contacts.ts`, `rows.ts`, `entities.ts`, `users.ts`, `index.ts`) | `ALL_SEARCH_INDEXES`, `BULK_REINDEX_SUPPORTED_INDEXES`, `getSearchIndexSchema()` |
|
|
| Outbox-Publisher | `packages/search/src/outbox-publisher.ts` | `publishSearchUpsertEvent`/`publishSearchDeleteEvent`/`publishSearchBulkReindexEvent`, `searchEventType()` |
|
|
| Sanitize | `packages/search/src/sanitize.ts` | `sanitizeDocument()` |
|
|
|
|
**HiveGrid-Relevanz:** Volltextsuche fuer KB-Artikel, CRM-Kontakte, generische Entity-Rows, User. Outbox-getriebener Reindex. Querschnitt "Suche" parallel zur Vektorsuche (hybrid moeglich).
|
|
|
|
---
|
|
|
|
## 4. Apps-Engine / Plugin-System (Kern-Fokus)
|
|
|
|
Wichtige Unterscheidung im Boilerplate:
|
|
- **`@toolctrl/plugins`** = Feature-/App-**Manifest-Inventur** (Klassifizierung von Features als core/plugin-candidate, Integration-Surfaces) — kein Laufzeit-Sandbox.
|
|
- **`@toolctrl/plugin-runtime` + `@toolctrl/plugin-sdk`** = echtes **WASM-Plugin-Laufzeitsystem** mit Capability-Modell (das "Apps-Engine"-Adapter-/Sandbox-Pattern).
|
|
- **Prisma `appsEngine.prisma`** = separate **Data-Sync-"App"-Engine** (App/DataConnection/AppSyncRun) fuer externe Datenquellen-Integrationen.
|
|
|
|
### 4.1 `@toolctrl/plugins` — Feature-Manifest-Schema + Loader
|
|
|
|
**Pfad:** `/Users/raffaelsheikh/Documents/GitHub/boilerplate/packages/plugins` (keine Deps)
|
|
|
|
| Asset | Pfad | Was |
|
|
|---|---|---|
|
|
| Manifest-Schema | `packages/plugins/src/manifest-schema.ts` | `FeatureManifest` (schemaVersion "1.0", classification core/plugin-candidate/remove-candidate/mixed, `integration.surfaces` admin/tenant/public/api-Routes, `integration.workers` queues/jobs, externalProviders, permissions, auditEvents, billingImpacts, uat, `boundaries`, `pruning`), `validateFeatureManifestShape()` (handgeschriebener Validator, kein zod) |
|
|
| Loader | `packages/plugins/src/manifest-loader.ts` | `loadFeatureManifests()` |
|
|
|
|
**HiveGrid-Relevanz:** Methodik-/Governance-Werkzeug — beschreibt, welche Features Plugin-Kandidaten sind und welche Surfaces/Worker/Provider sie brauchen. Nuetzlich als Vorlage fuer HiveGrid-Modul-/Plugin-Klassifizierung, weniger als Laufzeit.
|
|
|
|
### 4.2 `@toolctrl/plugin-runtime` — WASM-Sandbox + Capability-Gateway
|
|
|
|
**Pfad:** `/Users/raffaelsheikh/Documents/GitHub/boilerplate/packages/plugin-runtime`
|
|
**Deps:** `@toolctrl/logger`, `@toolctrl/shared-store`, `semver`, `zod`
|
|
**Subpath-Exports:** `.`, `./capabilities`, `./hooks/registry`, `./manifest`
|
|
**README:** `packages/plugin-runtime/README.md`
|
|
|
|
| Asset | Pfad | Was |
|
|
|---|---|---|
|
|
| WASM-Sandbox | `packages/plugin-runtime/src/runtime.ts` | `createPluginSandbox()` — Node-22 `WebAssembly`, Memory-Cap (Default 64 MB), Fuel-Budget (Default 1.000.000), JSON-Payload-ABI (`toolctrl_payload_len`/`...byte`), `invoke(hookName, payload)`, typisierte Failures (compile/instantiate/fuel_exhausted/memory_limit/trap). Upgrade-Pfad: Wasmer/Wasmtime hinter gleichem Interface |
|
|
| Capability-Enum | `packages/plugin-runtime/src/capabilities.ts` | `PLUGIN_CAPABILITY_NAMES` (http.outbound, db.read/write, storage.read/write, log, audit, tenant.info, secret.read), Risk-Metadata, **`PLUGIN_AVAILABLE_HOST_CAPABILITY_NAMES = ["log","audit"]`** (nur diese in Prod aktiv), `productionAvailable`-Flags |
|
|
| Capability-Gateway | `packages/plugin-runtime/src/capability-gateway.ts` | `createCapabilityGateway()` — deny-by-default; prueft manifest-deklariert + tenant-granted + Scope; `CapabilityDeniedError` mit Gruenden (not_declared/not_granted/scope_violation/no_host_implementation); laedt Fuel pro Call; `onCall`-Hook fuer Audit |
|
|
| Manifest (Plugin) | `packages/plugin-runtime/src/manifest.ts` | `PluginManifestSchema` (zod): Reverse-DNS-id, SemVer, `entry` (.wasm), `wasmSha256`, Ed25519 `signature`/`signedBy`, `allowedHosts` (SSRF-Allowlist, kein localhost/IP/.internal), `capabilities` mit `scope` (allowedHosts/keyPattern/tablePrefix), `hooks` (name->export), optional `ui.settings` (HTML + SHA256/Integrity), `min/maxHostVersion`; `parsePluginManifest()` (safe), `manifestDeclaresCapability()` |
|
|
| Signatur | `packages/plugin-runtime/src/signature.ts` | `verifyPluginSignature()` (Ed25519), `parseTrustedPublicKeys()`, `canonicalPluginSignaturePayload()` |
|
|
| Capability-Grants/Scope | `packages/plugin-runtime/src/capability-grants.ts`, `.../capability-scope.ts` | `PluginCapabilityGrant`, `isCapabilityGrantScopeSatisfied()`, `isCapabilityCallWithinScope()` |
|
|
| Hooks-Registry | `packages/plugin-runtime/src/hooks/registry.ts` | zod-validierte Hook-Payloads: `tenant.created`, `user.created`, `entity.{created,updated,deleted}`, `workflow.{created,updated,deleted,executed,failed}`; `registerHook()`, `validatePluginHookPayload()`, Prod-Available-Liste |
|
|
| Hook-Emit | `packages/plugin-runtime/src/hooks/emit.ts` | `emitPluginHook(tx, def, payload, publish)` — validiert + publiziert `plugin.hook.<name>` Outbox-Event in der DB-Transaktion (Fan-out auf Worker-Seite), `PluginHookValidationError` |
|
|
| Audit (Capability-Use) | `packages/plugin-runtime/src/audit.ts` | `createPluginCapabilityUseRecorder()`, `hashCapabilityPayload()` (kein Roh-Payload), Batch-Sink |
|
|
| Audit-Chain (Tamper-Evidence) | `packages/plugin-runtime/src/audit-chain.ts` | Per-Tenant Hash-Chain (`prevHash`/`rowHash`), `stampPluginCapabilityUseChain()`, `verifyPluginCapabilityUseChain()` |
|
|
| Crash-Tracker | `packages/plugin-runtime/src/crash-tracker.ts` | `createCrashTracker()` — auto-disable Plugin nach N Crashes |
|
|
| Migrations | `packages/plugin-runtime/src/migrations.ts` | `validatePluginMigration()`/`applyPluginMigration()` — Plugin-eigene SQL mit Keyword-Blocklist + Statement-Limit + `tablePrefix`-Scoping |
|
|
| Version-Check | `packages/plugin-runtime/src/version-check.ts` | `isPluginCompatibleWithHost()` (semver-Range gegen Host) |
|
|
| Production-Gate | `packages/plugin-runtime/src/production-gate.ts` | `assertPluginRuntimeProductionEnabled()`, Env `PLUGIN_RUNTIME_PRODUCTION_GATE_ENV` |
|
|
|
|
### 4.3 `@toolctrl/plugin-sdk` — Autoren-Typen + Settings-Client
|
|
|
|
**Pfad:** `/Users/raffaelsheikh/Documents/GitHub/boilerplate/packages/plugin-sdk` (keine Deps)
|
|
**README + Beispiele:** `packages/plugin-sdk/README.md`, `packages/plugin-sdk/examples/{hello-world,simple-hook-listener,capability-use,async-task}/` (jeweils `plugin.toml` + `handler.ts`)
|
|
|
|
| Asset | Pfad | Was |
|
|
|---|---|---|
|
|
| SDK-Typen + `defineHook` | `packages/plugin-sdk/src/index.ts` | `HookHandler`/`TypedHookHandler`, `defineHook()`, `PluginContext`, `HostApi` (log/audit/tenant/http/storage/secret), `PluginCapabilityName`/`PluginHookName` (gespiegelt aus runtime), Hook-Payload-Typen, ABI-Konstanten (`PLUGIN_PAYLOAD_ABI_VERSION`), Settings-Iframe-Protokoll: `createPluginSettingsClient()` (postMessage get/set, Timeout, Channel) |
|
|
|
|
**Prisma (Plugins):** `prisma/schema/plugins.prisma`
|
|
- `Plugin` (tenantId, pluginId, version, manifest Json, wasmSha256, signaturePublicKey, enabled)
|
|
- `TenantPluginCapabilityGrant` (per-tenant Consent pro Capability, scopeJson) — Install darf erst `enabled` werden, wenn alle deklarierten Capabilities Grants haben
|
|
- `PluginUserConfig` (per-user opt-in + configJson)
|
|
- `PluginInvocation` (Audit pro Invoke: outcome/durationMs/fuelUsed)
|
|
- `PluginCapabilityUse` (per-Capability-Call Audit mit `payloadHash`, Hash-Chain `prevHash`/`rowHash`)
|
|
|
|
**Queue:** Queue-Name `toolctrl.plugin-invoke`, Job `plugin.invoke` (siehe Queue-Abschnitt).
|
|
|
|
**HiveGrid-Relevanz (4.2/4.3):** Das ist das eigentliche **Apps-Engine/Plugin-Adapter-Pattern** fuer HiveGrid: sicheres, sandboxed Drittanbieter-/Erweiterungssystem mit Capability-Consent, Signatur-Verifikation, Hook-Lifecycle, Audit-Chain und Crash-Schutz. Direkt relevant fuer HiveGrid-Module "Apps/Marketplace", "Integrationen", "Erweiterbarkeit". Achtung: in Prod sind nur `log`/`audit`-Capabilities freigeschaltet — http/db/storage/secret sind Foundation-Stubs (`productionAvailable: false`).
|
|
|
|
### 4.4 Prisma Apps-Engine (Data-Sync) — `appsEngine.prisma`
|
|
|
|
**Pfad:** `prisma/schema/appsEngine.prisma`
|
|
|
|
| Modell | Was |
|
|
|---|---|
|
|
| `App` | Installierte App pro Tenant (`type`-Identifier z.B. "sample-data-sync", `config` JSON, isActive) |
|
|
| `DataConnection` | Verbindung zu externer Datenquelle (`type` "rest-api"/"oauth2"/"webhook", verschluesselte Credentials in `config`) |
|
|
| `AppSyncRun` | Sync-Lauf (status pending/running/completed/partial/failed/..., executionMode manual/scheduled/webhook, items processed/created/updated/failed) |
|
|
| `AppSyncRunLog` | Log-Eintraege pro Lauf |
|
|
| `AppSyncItem` | Einzel-Item-Mapping (sourceId/destinationId, action, rollback-Daten) |
|
|
|
|
**HiveGrid-Relevanz:** Separate "Apps-Engine" fuer **externe Datensynchronisation** (ETL-aehnlich, idempotent, mit Rollback + Audit). Relevant fuer HiveGrid-Module "Integrationen/Datenimport" und externe Connector-Apps — komplementaer zum WASM-Plugin-System (4.2).
|
|
|
|
---
|
|
|
|
## 5. `@toolctrl/storage` — S3/Scaleway-Storage, Signed-URLs, Virus-Scan
|
|
|
|
**Pfad:** `/Users/raffaelsheikh/Documents/GitHub/boilerplate/packages/storage`
|
|
**Deps:** `@aws-sdk/client-s3`, `clamscan`, `file-type`
|
|
|
|
| Asset | Pfad | Was |
|
|
|---|---|---|
|
|
| S3-/Local-Adapter | `packages/storage/src/index.ts` | `createS3StorageAdapter()` (endpoint/region/bucket/credentials, forcePathStyle, publicBaseUrl) ueber `@aws-sdk/client-s3` — Scaleway-/jeder-S3-kompatibel; `createLocalStorageAdapter()` (Dev, Pfad-Traversal-Schutz); `StorageAdapter` (putObject/deleteObject), `StorageAdapterError` mit typisierten Codes + Secret-Redaction |
|
|
| Signed-URLs | `packages/storage/src/signed/index.ts` | `createSignedUploadUrl`/`createSignedDownloadUrl` (PUT/GET, expiry, contentType, checksumSha256) |
|
|
| Virus-Scan-Service | `packages/storage/src/scan/index.ts` | `createStorageScanService()` (scan -> bei infected quarantaene), `ScanVerdict` clean/infected/error |
|
|
| ClamAV-Scanner | `packages/storage/src/scan/clamav-scanner.ts` | `createClamAVStorageScanner()` ueber `clamscan`, `ClamAVUnreachableError` |
|
|
| Magic-Bytes-Adapter | `packages/storage/src/scan/magic-bytes-adapter.ts` | `createMagicBytesScanAdapter()` ueber `file-type` (Content-Type-Verifikation) |
|
|
| Allow-List | `packages/storage/src/scan/allow-list.ts` | `DEFAULT_ALLOW_LIST`, `resolveAllowList()` |
|
|
| S3-Quarantaene | `packages/storage/src/scan/s3-quarantine.ts` | `createS3StorageQuarantineAdapter()` |
|
|
| Access-Control | `packages/storage/src/access/index.ts` | Storage-Zugriffspruefung |
|
|
|
|
**HiveGrid-Relevanz:** Datei-Uploads/Anhaenge (KB, CRM, Apps), DSGVO-EU-Storage via Scaleway-S3. Virus-Scan + magic-bytes-Pruefung = Sicherheits-Querschnitt fuer jedes Upload-Modul. Signed-URLs fuer direkten Client-Upload/Download.
|
|
|
|
---
|
|
|
|
## 6. `@toolctrl/realtime` — SSE/Realtime-Events
|
|
|
|
**Pfad:** `/Users/raffaelsheikh/Documents/GitHub/boilerplate/packages/realtime` (keine Deps)
|
|
**README:** `packages/realtime/README.md`
|
|
|
|
| Asset | Pfad | Was |
|
|
|---|---|---|
|
|
| Event-Typen | `packages/realtime/src/events.ts` | `REALTIME_EVENT_NAMES` (notification.snapshot/update, keepalive, error), `RealtimeEvent`-Union, `validateRealtimeEvent()`, Notification-Stream-Item-Typen |
|
|
| Server (SSE-Serialisierung) | `packages/realtime/src/server.ts` | `serializeRealtimeEvent()` (SSE `id/event/data`, retry), `serializeRealtimeComment()`, Newline-Sanitizing |
|
|
| Client | `packages/realtime/src/client.ts` | Client-seitige Event-Verarbeitung |
|
|
| Guards | `packages/realtime/src/guards.ts` | Type-Guards fuer Events |
|
|
|
|
**HiveGrid-Relevanz:** Live-Benachrichtigungen (In-App-Notifications) via SSE — kein WebSocket-Server, sondern SSE-Stream. Querschnitt "Realtime/Notifications". Hinweis: WebSocket im engeren Sinn ist hier **nicht** vorhanden; Realtime = SSE.
|
|
|
|
---
|
|
|
|
## 7. E-Mail-Stack
|
|
|
|
### 7.1 `@toolctrl/mail` — Provider + Templates
|
|
|
|
**Pfad:** `/Users/raffaelsheikh/Documents/GitHub/boilerplate/packages/mail`
|
|
**Deps:** `@react-email/components`, `@react-email/render`, `@toolctrl/brand`, `@toolctrl/queue`, `react`, `react-dom`
|
|
**README:** `packages/mail/README.md`; Preview-Server: `packages/mail/scripts/preview-server.ts`
|
|
|
|
| Asset | Pfad | Was |
|
|
|---|---|---|
|
|
| MailerSend-Provider | `packages/mail/src/mailersend.ts` | `createMailerSendProvider()`/`...FromEnv()`, API `https://api.mailersend.com`, Timeout, `MailMetricsHook`, Templates/Personalization/Attachments/Tracking |
|
|
| Queued-Mail-Provider | `packages/mail/src/queue.ts` | `createQueuedMailProvider()` — Versand via `@toolctrl/queue` (async) statt direkt |
|
|
| Mail-Typen | `packages/mail/src/types.ts` | `MailProvider`, `SendMailInput`, `MailAddress`, `MailAttachment`, `MailProviderError` |
|
|
| React-Email Render | `packages/mail/src/templates-react/render.ts` | `renderTransactionalEmail()`, `reactTransactionalTemplateNames`, Brand-Helfer |
|
|
| Transaktionstemplates | `packages/mail/src/templates-react/transactional/` | Welcome, MagicLink, PasswordReset, Invitation, InvoiceSent, InvoiceTenant, SubscriptionSuccess/Failed, TenantNotification |
|
|
| Layout-Primitives | `packages/mail/src/templates-react/_layout/` | BaseLayout, Header, Footer, Button (`assertSafeUrl`/`InvalidActionUrlError`), InfoBox, Eyebrow, theme |
|
|
|
|
### 7.2 `@toolctrl/email-templates` — Sequenz-Templates
|
|
|
|
**Pfad:** `/Users/raffaelsheikh/Documents/GitHub/boilerplate/packages/email-templates`
|
|
**Subpath-Exports:** `./sequences/{welcome,trial-reminder,re-activation,generic-step}`
|
|
|
|
| Asset | Pfad | Was |
|
|
|---|---|---|
|
|
| Render + Theme | `packages/email-templates/src/render.ts`, `.../theme.ts` | React-Email Render-Layer |
|
|
| Sequenz-Templates | `packages/email-templates/src/sequences/` | welcome, trial-reminder, re-activation, generic-step |
|
|
|
|
### 7.3 `@toolctrl/email-sequences` — Sequenz-Engine (reine Logik)
|
|
|
|
**Pfad:** `/Users/raffaelsheikh/Documents/GitHub/boilerplate/packages/email-sequences` (keine Deps)
|
|
**Subpath-Exports:** `.`, `./types`, `./engine`, `./step-runner`, `./conditions`, `./unsubscribe-token`
|
|
|
|
| Asset | Pfad | Was |
|
|
|---|---|---|
|
|
| Engine | `packages/email-sequences/src/engine.ts` | `startSequenceRun()`/`advanceSequenceRun()` — Store + Queue injiziert (kein Prisma/BullMQ direkt); `SequenceRunStatus`, idempotent ueber `(sequenceId, triggerEventId)` |
|
|
| Step-Runner | `packages/email-sequences/src/step-runner.ts` | `runStep()` — rendert + sendet einen Schritt |
|
|
| Condition-DSL | `packages/email-sequences/src/conditions.ts` | `evaluateCondition()`/`evaluateAllConditions()`, `resolvePath()` |
|
|
| Unsubscribe-Token | `packages/email-sequences/src/unsubscribe-token.ts` | `signUnsubscribeToken()`/`verifyUnsubscribeToken()`/`buildUnsubscribeUrl()`, Secret-Pflicht |
|
|
|
|
**Prisma:** `prisma/schema/emailSequences.prisma` — `EmailSequence`, `EmailSequenceRun`, `EmailSequenceStepDelivery`. `prisma/schema/email.prisma` (App-Email-Config).
|
|
|
|
**HiveGrid-Relevanz (7.x):** Transaktions-Mails (Auth, Billing, Notifications) + Marketing-/Lifecycle-Sequenzen (Onboarding, Trial-Reminder, Reaktivierung). EU-Provider MailerSend. Querschnitt "Kommunikation/E-Mail" fuer praktisch jedes HiveGrid-Modul; Sequenz-Engine direkt fuer "Marketing/Funnel/Lifecycle".
|
|
|
|
---
|
|
|
|
## 8. `@toolctrl/marketing-outbound` — Outbound-Webhook-Adapter
|
|
|
|
**Pfad:** `/Users/raffaelsheikh/Documents/GitHub/boilerplate/packages/marketing-outbound` (keine Deps)
|
|
**Subpath-Exports:** `./adapters/{slack,discord,mailchimp,hubspot,brevo,custom-webhook}`
|
|
|
|
| Asset | Pfad | Was |
|
|
|---|---|---|
|
|
| Adapter-Interface + Registry | `packages/marketing-outbound/src/index.ts` | `adapterRegistry` (slack/discord/mailchimp/hubspot/brevo/webhook), `resolveAdapter(type)`, `ADAPTER_TYPES` |
|
|
| Shared-Typen + SSRF-Guard | `packages/marketing-outbound/src/types.ts` | `MarketingOutboundAdapter<TConfig>` (`parseConfig`/`send`), `MarketingLeadPayload`/`MarketingLeadEventType` (captured/qualified/converted...), `adapterFetch()` (Timeout + SSRF-Allowlist, nur HTTPS, kein RFC1918/metadata), `maskEmailIfRequested()` (PII), `MarketingAdapterConfigError` |
|
|
| Konkrete Adapter | `packages/marketing-outbound/src/adapters/*.ts` | je `buildXPayload()` + Adapter-Objekt (Slack-Blocks, Discord-Embed, Mailchimp/HubSpot/Brevo-Payloads, Custom-Webhook) |
|
|
|
|
**Prisma:** `prisma/schema/marketingLeads.prisma` — `MarketingIntegration` (type, configJson verschluesselt, `triggerEvents` String[], enabled, lastDeliveredAt/lastError), `Lead`, `LeadEvent`, `TenantAcquisition`.
|
|
|
|
**HiveGrid-Relevanz:** Lead-/Marketing-Integrationen mit Drittsystemen (CRM, Chat, ESP). Worker dispatcht Outbox-Events nach `MarketingIntegration.type`. SSRF-/PII-Schutz im Adapter. Direkt fuer HiveGrid-Module "Marketing/Funnel" und "Integrationen (Outbound-Webhooks)".
|
|
|
|
---
|
|
|
|
## 9. `@toolctrl/queue` — BullMQ + Idempotenz + Prioritaeten
|
|
|
|
**Pfad:** `/Users/raffaelsheikh/Documents/GitHub/boilerplate/packages/queue`
|
|
**Deps:** `bullmq`
|
|
**Subpath-Exports:** `.`, `./runtime` (CJS `runtime.cjs`)
|
|
**README:** `packages/queue/README.md`
|
|
|
|
| Asset | Pfad | Was |
|
|
|---|---|---|
|
|
| Queue-Namen | `packages/queue/src/names.ts` | `queueNames`: `toolctrl.email`, `.notifications`, `.maintenance`, `.ai-chatbot`, `.browser-automation`, `.file-scan`, `.exports`, `.imports`, `.plugin-invoke`; `isQueueName()` |
|
|
| Job-Payloads (typisiert) | `packages/queue/src/payloads.ts` | `AiJob` (`ai.chat-completion`/`ai.embedding`/`ai.browser-task`), `PluginInvokeJob` (`plugin.invoke`), `TransactionalMailJob`, `WebhookDeliveryJob`, `NotificationJob`, `CleanupJob`, `FileScanJob`, `ExportJob`/`ImportJob`, Workflow-Outbox-Events; `QueueJobPayloads`-Map |
|
|
| Producer + Idempotenz | `packages/queue/src/producers.ts` | `enqueueJob()`, `build*Job()`-Builder, `buildIdempotentJobId()` (jobId/idempotencyKey/export-/importJobId/metadata.idempotencyKey), `buildRetryJobOptions()`, `buildIdempotentJobId`, `setEnqueueObserver()` |
|
|
| Prioritaeten | `packages/queue/src/priority.ts` | `JOB_PRIORITY_{CRITICAL,HIGH,NORMAL,LOW}`, `resolveJobPriorityForTenant()` (Plan-basiert), `priorityFromPlan/Override`, `setTenantPlanLookup()` |
|
|
| Redis-Connection | `packages/queue/src/connection.ts` | `parseRedisUrl()`, `resolveRedisConnectionOptions()`, `...FromProcessEnv()` |
|
|
| Queue-Gruppen | `packages/queue/src/groups.ts` | `WORKER_QUEUE_GROUPS`, `getGroupForQueue()`, `getQueueNamesForGroup()` |
|
|
| Runtime (BullMQ-Factory) | `packages/queue/src/runtime.ts` / `runtime.cjs` | `createQueue`/`createWorker`/`createQueueEvents` (typisiert) |
|
|
|
|
**Prisma:** `prisma/schema/queue.prisma` — `QueueFailedJob` (DLQ-Spiegel), `ExportJob` + `ExportJobStatus`-Enum. Outbox-Pattern: `prisma/schema/outbox.prisma`.
|
|
|
|
**HiveGrid-Relevanz:** Zentrale asynchrone Verarbeitung fuer KI-Jobs, Plugin-Invokes, Mail, Notifications, File-Scan, Exporte/Importe. **Idempotenz** ueber jobId/idempotencyKey + plan-basierte **Prioritaeten** = Querschnitt fuer alle HiveGrid-Worker-Module. AI/Plugin/Mail-Queues binden die obigen Pakete an den Worker an.
|
|
|
|
---
|
|
|
|
## Querschnitts-Konventionen (faktisch beobachtet)
|
|
|
|
- **Tenant-Isolation** ist ueberall am Seam verankert: `payload.tenantId`-Pflicht (vector-store), `searchForTenant`/`tenantId`-Filter (search, vector-store), `tenantId` Pflichtargument (embeddings), per-tenant Grants/Audit-Chain (plugin-runtime).
|
|
- **Outbox-Pattern** fuer asynchrone Seiteneffekte: RAG-Reindex (`rag.index.*`), Such-Reindex, Plugin-Hooks (`plugin.hook.*`), AI-Cost (`ai.cost.event.persist`), Marketing-Dispatch — alle in DB-Transaktion committet, Worker fan-out.
|
|
- **Kein direkter `process.env`-Zugriff** in den Paketen (Env-Snapshots vom Boundary) — Env-Access-Enforcement-Tests.
|
|
- **Secret-/PII-Redaction** durchgaengig in Fehlern/Audit (ai/guards, ai/tools, ai/streaming, storage, marketing-outbound).
|
|
- **Adapter-/Factory-Pattern** als roter Faden: `getAIProviderAdapter`, `createEmbeddingProvider`+Registry, `createVectorStore`+`resolveAdapter`, `createSearchClient`+`meiliFactory`, `adapterRegistry` (marketing), `createS3StorageAdapter`/`createLocalStorageAdapter`, Capability-`hostFns`-Map (plugin-runtime).
|
|
- **Tests:** jedes Paket hat `*.test.ts` neben den Quellen; Eval-Harness in `packages/ai/test/evals` + Repo-Root `promptfoo.config.yaml`; CI/verify ueber `pnpm` Workspace-Scripts (`test`/`typecheck`/`build`), Repo-Root `package.json` + `lefthook.yml`.
|
|
|
|
## Wichtige Abgrenzungen / Luecken (faktisch)
|
|
|
|
- **WebSocket:** Es gibt **kein** WebSocket-Paket; Realtime laeuft ueber **SSE** (`@toolctrl/realtime`).
|
|
- **Plugin-Capabilities in Prod:** Nur `log` + `audit` sind hostseitig verfuegbar (`PLUGIN_AVAILABLE_HOST_CAPABILITY_NAMES`); http/db/storage/secret/tenant.info sind deklarierbar, aber `productionAvailable: false` (Foundation).
|
|
- **WASM-Fuel:** Keine echte Per-Instruktions-Meterung (V8-Limit) — Fuel wird pro Capability-Call/Invoke geladen; Wall-Clock-Timeout liegt beim Aufrufer.
|
|
- **Anthropic-Embeddings:** Bewusster Stub/Platzhalter (`packages/ai/src/embeddings/providers/anthropic.ts`).
|
|
- **Zwei "Apps-Engine"-Bedeutungen:** WASM-Plugin-System (`plugin-runtime`/`plugin-sdk`) vs. Data-Sync-Apps (`prisma/schema/appsEngine.prisma`, `App`/`DataConnection`/`AppSyncRun`) — beim Modul-Abgleich nicht verwechseln.
|