164 lines
8.2 KiB
Markdown
164 lines
8.2 KiB
Markdown
# Chatbot MVP — Master Switch + 3 Base Capabilities
|
|
|
|
**Status:** Shipped (Phase 07, Task #13).
|
|
|
|
This document describes how the in-app chatbot is gated, which capabilities it
|
|
exposes, how to add new capabilities, and how to troubleshoot the most common
|
|
failure modes.
|
|
|
|
## Architecture overview
|
|
|
|
The chatbot is a streaming AI assistant exposed in the in-app shell. In MVP it
|
|
ships behind a **master feature flag** (`chatbot`) and exposes three read-only,
|
|
tenant-scoped capabilities. Future phases (V1, V2) extend the capability set
|
|
with per-user allow-lists, RAG, and write-tools.
|
|
|
|
```
|
|
+------------------+ flag off +------------+
|
|
| chatbot-stream | ───────────► | 403 (json) |
|
|
| (server) | +------------+
|
|
| | flag on, key missing
|
|
| | ───────────► +------------+
|
|
| | | 503 NO_AI_ |
|
|
| | | PROVIDER_ |
|
|
| | | KEY |
|
|
| | +------------+
|
|
| | flag on, key present
|
|
| | ───────────► capability resolver
|
|
| | │
|
|
| | ▼
|
|
| | tool registry (only active caps)
|
|
| | │
|
|
| | ▼
|
|
| | streamText → SSE
|
|
+------------------+
|
|
```
|
|
|
|
Source-of-truth modules:
|
|
|
|
| Concern | File |
|
|
| ------------------------ | ------------------------------------------------------------------------------- |
|
|
| Capability enum + metadata | `apps/web/app/modules/chatbot/lib/capabilities/capability-types.ts` |
|
|
| Capability resolver | `apps/web/app/modules/chatbot/lib/capabilities/resolver.server.ts` |
|
|
| System prompt builder | `apps/web/app/modules/chatbot/lib/system-prompt.ts` |
|
|
| `qa_help` tool | `apps/web/app/modules/chatbot/lib/capabilities/qa-help.server.ts` |
|
|
| `kb_search` tool | `apps/web/app/modules/chatbot/lib/capabilities/kb-search.server.ts` |
|
|
| `account_info` tool | `apps/web/app/modules/chatbot/lib/capabilities/account-info.server.ts` |
|
|
| Tool registry | `apps/web/app/modules/chatbot/lib/capabilities/tool-registry.server.ts` |
|
|
| Stream entry point | `apps/web/app/modules/chatbot/lib/chatbot-stream.server.ts` |
|
|
| Provider-key status | `apps/web/app/modules/chatbot/lib/provider-status.server.ts` |
|
|
| Tenant settings page | `apps/web/app/routes/app.$tenant/settings/chatbot.tsx` |
|
|
| CLI commands | `packages/cli/src/commands/chatbot/` |
|
|
|
|
## How the master flag works
|
|
|
|
A single global feature flag named `chatbot` controls availability across the
|
|
platform. The flag is evaluated per request via
|
|
`evaluateFlagForContext(CHATBOT_MASTER_FLAG_NAME, …)` using the Phase-05
|
|
feature-flag engine (`@toolctrl/feature-flags`). Result is cached at the
|
|
request level.
|
|
|
|
- **Flag off / not found** → tenant chats return
|
|
`403 { error: "chatbot-disabled", message: … }`.
|
|
- **Flag on + no AI-provider env key** → return
|
|
`503 { error: "ai-provider-not-configured", code: "NO_AI_PROVIDER_KEY", docs: … }`.
|
|
- **Flag on + key present** → resolver returns
|
|
`['qa_help', 'kb_search', 'account_info']`, the tool registry registers the
|
|
three corresponding tools, and `streamText` runs.
|
|
|
|
The flag is consulted only for **tenant** chats. Admin chats use a separate
|
|
static gate (`getChatbotContextConfig('admin').enabled`) enforced upstream.
|
|
|
|
### Toggling the flag
|
|
|
|
Three equivalent ways:
|
|
|
|
1. **Tenant Settings UI**: `/app/$tenant/settings/chatbot` → "Enable chatbot".
|
|
2. **Admin Feature Flags UI**: `/admin/feature-flags` → flip `chatbot` row.
|
|
3. **CLI**:
|
|
```bash
|
|
toolctrl-cli chatbot enable
|
|
toolctrl-cli chatbot disable
|
|
toolctrl-cli chatbot status
|
|
```
|
|
|
|
Every flip publishes on the `feature-flags:invalidate` channel — the web
|
|
replica's eval cache drops automatically.
|
|
|
|
## The 3 MVP capabilities
|
|
|
|
| ID | Tool name | Description | Risk |
|
|
| -------------- | ---------------------- | -------------------------------------------------------- | ---- |
|
|
| `qa_help` | `search_help` | Ranked snippets from bundled help docs (no DB, no net). | low |
|
|
| `kb_search` | `search_knowledge_base`| Tenant-scoped knowledge-base search. | low |
|
|
| `account_info` | `get_account_info` | Read-only plan / credits / limits / members projection. | low |
|
|
|
|
All three are **read-only**, **tenant-scoped via closure** (tenantId is
|
|
injected into the tool factory; the tool schema deliberately does not accept a
|
|
`tenantId` field), and never expose sensitive billing identifiers.
|
|
|
|
### Indirect-prompt-injection mitigation
|
|
|
|
`kb_search` wraps every snippet in explicit
|
|
`---KB ARTICLE START---` / `---KB ARTICLE END---` markers. The system prompt
|
|
contains an explicit rule that content between these markers is USER-DATA, not
|
|
instructions. This is a partial mitigation — V1 will add per-tool audit logs
|
|
and V2 will introduce a separated untrusted-context channel.
|
|
|
|
## How to add a new capability
|
|
|
|
1. Add the ID to the `CHATBOT_CAPABILITIES` union and metadata table in
|
|
`capability-types.ts`. Set `mvp: false`.
|
|
2. Implement the tool factory in `capabilities/<your-cap>.server.ts` exporting
|
|
a `build<YourCap>Tool(deps)` function. Use closure-bound tenantId; never
|
|
accept tenantId as a tool arg.
|
|
3. Add a case in `buildToolRegistry` mapping the capability to the new tool.
|
|
4. (V1+) Update the resolver so the new capability is included only for the
|
|
right users / plans.
|
|
5. Tests: unit-test the tool's projection, and add a registry case that
|
|
asserts the tool is exposed when the capability is active and NOT when it
|
|
is inactive.
|
|
|
|
## Provider-key onboarding
|
|
|
|
`provider-status.server.ts` checks for any of:
|
|
|
|
- `MISTRAL_API_KEY`
|
|
- `SCALEWAY_API_KEY`
|
|
- `ALEPH_ALPHA_API_KEY`
|
|
- `OPENAI_API_KEY`
|
|
- `ANTHROPIC_API_KEY`
|
|
|
|
If none are set but the chatbot flag is ON, the stream returns
|
|
`NO_AI_PROVIDER_KEY` (503). The settings UI surfaces a banner. The CLI
|
|
`chatbot status` command warns operators about this state.
|
|
|
|
## Troubleshooting
|
|
|
|
| Symptom | Diagnosis |
|
|
| -------------------------------------------------------- | ------------------------------------------------------------------------- |
|
|
| `403 chatbot-disabled` in browser network tab | Feature flag `chatbot` is off. Toggle in settings UI or via CLI. |
|
|
| `503 NO_AI_PROVIDER_KEY` after enabling | No AI-provider key in env. Set one of the 5 supported `*_API_KEY` vars. |
|
|
| Chatbot answers without using a tool | Question may not require a tool. Inspect Sentry tags / steps in DB. |
|
|
| KB search returns no results for known-good query | Verify tenant-id scoping. Cross-tenant negative test in `kb-search.server.test.ts`. |
|
|
| Stream succeeds but no cost event in DB | Check `AI_COST_TRACKING_ENABLED` env. MVP wires cost-tracking via the existing `streamText` path; full `executeLLMCall` integration arrives in V2. |
|
|
|
|
## Known MVP limitations (resolved in V1 / V2)
|
|
|
|
- Single global flag — no per-user allow-list (V1).
|
|
- Capability set is hard-wired to the three MVP ids (V1 will add the remaining
|
|
five: `data_read`, `data_write`, `workflow_trigger`, `admin_actions`,
|
|
`external_tools`).
|
|
- No RAG over tenant docs (V2).
|
|
- No per-capability rate limiting (V1 adds this on top of the global per-user
|
|
limiter).
|
|
- No audit-log entry per tool call (V1 adds this for high-risk tools).
|
|
|
|
## References
|
|
|
|
- Spec: `docs/improvement-plan/07-chatbot/chatbot-mvp.md`
|
|
- V1: `docs/improvement-plan/07-chatbot/chatbot-v1.md`
|
|
- V2: `docs/improvement-plan/07-chatbot/chatbot-v2.md`
|
|
- Feature flags engine: `docs/features/feature-flags.md`
|
|
- AI cost engine: `docs/features/ai-cost-tracking.md`
|