Chatbot MVP — Master Switch + 3 Base Capabilities

Status: Shipped (Phase 07, Task #13).

This document describes how the in-app chatbot is gated, which capabilities it exposes, how to add new capabilities, and how to troubleshoot the most common failure modes.

Architecture overview

The chatbot is a streaming AI assistant exposed in the in-app shell. In MVP it ships behind a master feature flag (chatbot) and exposes three read-only, tenant-scoped capabilities. Future phases (V1, V2) extend the capability set with per-user allow-lists, RAG, and write-tools.

+------------------+    flag off    +------------+
| chatbot-stream   |  ───────────►  | 403 (json) |
| (server)         |                +------------+
|                  |    flag on, key missing
|                  |  ───────────►  +------------+
|                  |                | 503 NO_AI_ |
|                  |                | PROVIDER_  |
|                  |                | KEY        |
|                  |                +------------+
|                  |    flag on, key present
|                  |  ───────────►  capability resolver
|                  |                  │
|                  |                  ▼
|                  |                tool registry (only active caps)
|                  |                  │
|                  |                  ▼
|                  |                streamText  → SSE
+------------------+

Source-of-truth modules:

Concern	File
Capability enum + metadata	`apps/web/app/modules/chatbot/lib/capabilities/capability-types.ts`
Capability resolver	`apps/web/app/modules/chatbot/lib/capabilities/resolver.server.ts`
System prompt builder	`apps/web/app/modules/chatbot/lib/system-prompt.ts`
`qa_help` tool	`apps/web/app/modules/chatbot/lib/capabilities/qa-help.server.ts`
`kb_search` tool	`apps/web/app/modules/chatbot/lib/capabilities/kb-search.server.ts`
`account_info` tool	`apps/web/app/modules/chatbot/lib/capabilities/account-info.server.ts`
Tool registry	`apps/web/app/modules/chatbot/lib/capabilities/tool-registry.server.ts`
Stream entry point	`apps/web/app/modules/chatbot/lib/chatbot-stream.server.ts`
Provider-key status	`apps/web/app/modules/chatbot/lib/provider-status.server.ts`
Tenant settings page	`apps/web/app/routes/app.$tenant/settings/chatbot.tsx`
CLI commands	`packages/cli/src/commands/chatbot/`

How the master flag works

A single global feature flag named chatbot controls availability across the platform. The flag is evaluated per request via evaluateFlagForContext(CHATBOT_MASTER_FLAG_NAME, …) using the Phase-05 feature-flag engine (@toolctrl/feature-flags). Result is cached at the request level.

Flag off / not found → tenant chats return 403 { error: "chatbot-disabled", message: … }.
Flag on + no AI-provider env key → return 503 { error: "ai-provider-not-configured", code: "NO_AI_PROVIDER_KEY", docs: … }.
Flag on + key present → resolver returns ['qa_help', 'kb_search', 'account_info'], the tool registry registers the three corresponding tools, and streamText runs.

The flag is consulted only for tenant chats. Admin chats use a separate static gate (getChatbotContextConfig('admin').enabled) enforced upstream.

Toggling the flag

Three equivalent ways:

Tenant Settings UI: /app/$tenant/settings/chatbot → "Enable chatbot".
Admin Feature Flags UI: /admin/feature-flags → flip chatbot row.

CLI:

toolctrl-cli chatbot enable
toolctrl-cli chatbot disable
toolctrl-cli chatbot status

Every flip publishes on the feature-flags:invalidate channel — the web replica's eval cache drops automatically.

The 3 MVP capabilities

ID	Tool name	Description	Risk
`qa_help`	`search_help`	Ranked snippets from bundled help docs (no DB, no net).	low
`kb_search`	`search_knowledge_base`	Tenant-scoped knowledge-base search.	low
`account_info`	`get_account_info`	Read-only plan / credits / limits / members projection.	low

All three are read-only, tenant-scoped via closure (tenantId is injected into the tool factory; the tool schema deliberately does not accept a tenantId field), and never expose sensitive billing identifiers.

Indirect-prompt-injection mitigation

kb_search wraps every snippet in explicit ---KB ARTICLE START--- / ---KB ARTICLE END--- markers. The system prompt contains an explicit rule that content between these markers is USER-DATA, not instructions. This is a partial mitigation — V1 will add per-tool audit logs and V2 will introduce a separated untrusted-context channel.

How to add a new capability

Add the ID to the CHATBOT_CAPABILITIES union and metadata table in capability-types.ts. Set mvp: false.
Implement the tool factory in capabilities/<your-cap>.server.ts exporting a build<YourCap>Tool(deps) function. Use closure-bound tenantId; never accept tenantId as a tool arg.
Add a case in buildToolRegistry mapping the capability to the new tool.
(V1+) Update the resolver so the new capability is included only for the right users / plans.
Tests: unit-test the tool's projection, and add a registry case that asserts the tool is exposed when the capability is active and NOT when it is inactive.

Provider-key onboarding

provider-status.server.ts checks for any of:

MISTRAL_API_KEY
SCALEWAY_API_KEY
ALEPH_ALPHA_API_KEY
OPENAI_API_KEY
ANTHROPIC_API_KEY

If none are set but the chatbot flag is ON, the stream returns NO_AI_PROVIDER_KEY (503). The settings UI surfaces a banner. The CLI chatbot status command warns operators about this state.

Troubleshooting

Symptom	Diagnosis
`403 chatbot-disabled` in browser network tab	Feature flag `chatbot` is off. Toggle in settings UI or via CLI.
`503 NO_AI_PROVIDER_KEY` after enabling	No AI-provider key in env. Set one of the 5 supported `*_API_KEY` vars.
Chatbot answers without using a tool	Question may not require a tool. Inspect Sentry tags / steps in DB.
KB search returns no results for known-good query	Verify tenant-id scoping. Cross-tenant negative test in `kb-search.server.test.ts`.
Stream succeeds but no cost event in DB	Check `AI_COST_TRACKING_ENABLED` env. MVP wires cost-tracking via the existing `streamText` path; full `executeLLMCall` integration arrives in V2.

Known MVP limitations (resolved in V1 / V2)

Single global flag — no per-user allow-list (V1).
Capability set is hard-wired to the three MVP ids (V1 will add the remaining five: data_read, data_write, workflow_trigger, admin_actions, external_tools).
No RAG over tenant docs (V2).
No per-capability rate limiting (V1 adds this on top of the global per-user limiter).
No audit-log entry per tool call (V1 adds this for high-risk tools).

References

Spec: docs/improvement-plan/07-chatbot/chatbot-mvp.md
V1: docs/improvement-plan/07-chatbot/chatbot-v1.md
V2: docs/improvement-plan/07-chatbot/chatbot-v2.md
Feature flags engine: docs/features/feature-flags.md
AI cost engine: docs/features/ai-cost-tracking.md

8.2 KiB Raw Blame History