crm-ai-demo/AGENTS.md

# CRM AI Demo - Project Guidelines

## Project
B2B stainless steel jewelry wholesaler (Yehwang, NL) CRM with AI-powered sales/service assistant bridge. Chatwoot webhook → LLM agent (tool-use) → Chatwoot reply + Twenty CRM opportunity creation.

## Tech Stack
- **Bridge**: Node.js 24, CommonJS (`"type": "commonjs"`), pure Node built-ins (`http`/`fetch`/`crypto`) + `jose` + `pg`. No Express. No test/lint/build scripts — verify by running the server and hitting `/healthz`.
- **LLM**: OpenAI-compatible provider via `bridge/llm.config.json`. Active: `aliyun` → DashScope `qwen-plus-latest` (`enableThinking: false`). `server.js` header comment says "glm-5.2/ark" — **ignore it, trust `llm.config.json`**.
- **Messaging**: Chatwoot (self-hosted, external source checkout at `./chatwoot/` — has its own `.git/`).
- **CRM**: Twenty CRM (self-hosted, external source checkout at `./twenty/` — may be absent).
- **Vector DB**: pgvector on shared `crm-postgres` (pg18). Three DBs: `chatwoot`, `twenty`, `products`. KB RAG table: `products.kb_docs`.
- **Product Search**: external recommend API (`RECOMMEND_URL`, default `http://recommend.internal.yw.com`).
- **Order/Inventory APIs**: trading-service (`ORDER_API_URL`) and eta-service (`INVENTORY_API_URL`). Mock server available: `node mock_services.js` (port 9505).
- **Embedding**: DashScope `text-embedding-v4` (dim 1024), configured in `kb.config.json`. Query translation fallback uses `qwen-turbo`.

## Architecture
```
Chatwoot (external) ──webhook──► Bridge (:4000) ──LLM tool-use──► Chatwoot reply
                                       │
                                       ├── pgvector (products.kb_docs)   KB RAG            (search_kb)
                                       ├── recommend API                  product catalog    (search_products)
                                       ├── trading-service                order tracking     (search_orders)
                                       ├── eta-service                    inventory/stock    (search_inventory)
                                       └── Twenty CRM GraphQL             create Opportunity
```

Bridge routes (`bridge/server.js`):
- `GET /healthz` (and `/`) — liveness, returns `agentBot: !!AGENT_BOT_SECRET`
- `GET /conversation/:id` and `GET /conversation/:id/html` — debug views
- `POST /agent-bot` — Chatwoot AgentBot outgoing webhook (HMAC-signed)
- `POST /chatwoot-webhook` — Chatwoot `message_created` webhook (fallback entry)

Agent tool-use loop: LLM autonomously calls `search_kb`, `search_products`, `search_orders`, `search_inventory`, or `escalate_human`. Single `AGENT_SYSTEM` persona — no prompt switching. Per-conversation lock (`locks` Set) prevents concurrent LLM calls; `createdOpps` Set prevents duplicate Opportunities.

## Startup
```bash
# Full stack (requires Docker Desktop):
bash start.sh

# Shared infra only (no Chatwoot/Twenty source needed):
docker compose up -d --build

# Local dev (no Docker):
cd bridge && node server.js  # Node 24+; needs PG_URL etc. as env vars

# Mock business APIs (for dev without real trading/eta services):
node mock_services.js  # Listens on :9505
```

`start.sh` conditionally starts Twenty (if `./twenty/docker-compose.yaml` exists) and Chatwoot (if `./chatwoot/docker-compose.yml` exists — note: currently runs unconditionally, will error if absent). Shared infra always starts.

Logs / health:
```bash
docker logs -f crm-bridge
curl http://localhost:4000/healthz
```

**No rebuild needed for code changes** — `docker-compose.yml` volume-mounts `bridge/*.js` and `*.config.json` read-only. `docker restart crm-bridge` picks up edits. Only rebuild the image when changing `package.json` deps.

## Key Files
```
bridge/
  server.js              Main bridge: routes, LLM agent loop, tool impls, Chatwoot/Twenty API calls (~1250 lines)
  index_kb.js            KB vectorizer: customer-service-kb/*.md → embed → products.kb_docs (DROPS + recreates table each run)
  register_agent_bot.js  Registers Chatwoot AgentBot (idempotent). Run: node register_agent_bot.js
  llm.config.json        LLM provider config. Active provider key: "aliyun"
  kb.config.json         Embedding config (model, dim, translateModel) + DashScope creds
  chatwoot.config.json   Chatwoot API connection (baseURL, accountId, accessToken, inboxId)
  twenty.config.json     Twenty CRM connection (baseURL, workspaceId, apiKey JWT)
  Dockerfile             node:24-alpine, npm install --omit=dev, copies server.js + configs
customer-service-kb/     FAQ markdown (source for index_kb.js). Chunks split on ### Q:. English; AI translates per customer language.
infra/init-db.sh         postgres init: creates DBs chatwoot/twenty/products + enables pgvector on chatwoot & products.
docker-compose.yml       Shared infra: pgvector/pgvector:pg18 + redis:8-alpine + bridge
mock_services.js         Mock trading/eta API server (:9505) for dev without real backends
add_note.js              CLI utility: attach a text note to a Twenty opportunity
create_workflow.js       CLI utility: create Twenty automation workflow (new opp → auto follow-up task)
find_keys.js             CLI utility: find Twenty signing key tables in postgres
gen_apikey.js            CLI utility: generate Twenty API key JWT (needs ENCRYPTION_KEY + PG_DATABASE_URL)
landing.html             Multilingual landing page
start.sh                 One-shot launcher for all compose stacks
docs/twenty-crm-guide.md Twenty CRM integration guide: which GraphQL ops bridge uses, data model, extension ideas
```

## Environment Variables
Env vars always override `*.config.json` values. Key overrides in `server.js`:
- `CW_BASE_URL` — overrides `chatwoot.config.json` baseURL
- `TWENTY_BASE_URL` — overrides `twenty.config.json` baseURL
- `PG_URL` — postgres connection (default `postgres://postgres:postgres@localhost:5432/products`)
- `AGENT_BOT_SECRET` — HMAC signing key for AgentBot webhook verification
- `RECOMMEND_URL` — product recommendation API base URL
- `ORDER_API_URL` — trading-service base URL (order queries)
- `ORDER_API_KEY` — auth key for trading-service
- `INVENTORY_API_URL` — eta-service base URL (stock/ETA queries)
- `INVENTORY_API_KEY` — auth key for eta-service
- `DASHSCOPE_API_KEY` — overrides `kb.config.json` apiKey (avoids hardcoding in config)
- `PORT` — bridge listen port (default 4000)

Docker-compose sets these for the bridge container (using container names for inter-service routing, `host.docker.internal` for host-side services).

## Conventions & Gotchas
- **Config loading**: `loadCfg()` reads `*.config.json` from `__dirname` (the bridge dir). Env vars always win — check `process.env.*` overrides in `server.js` before editing config files.
- **KB re-indexing**: `index_kb.js` **DROPS and recreates** `kb_docs` every run (no incremental update). Any model/dimension change requires a full re-vectorize. Run `node index_kb.js` after editing any `customer-service-kb/*.md` file.
- **KB doc format**: YAML frontmatter (`title`/`category`/`keywords`/`source_url`) + `### Q:`/answer blocks. Embedding text = `keywords + Q + A`. Source URLs are mapped per-section in `index_kb.js` `SECTION_URL_MAP`.
- **Agent persona is the source of truth** for sales policy (discount tiers, minimum order €30, contacts). `AGENT_SYSTEM` in `server.js` encodes business rules; `customer-service-kb/` encodes policy FAQs. Keep them consistent.
- **Language rule**: replies must mirror the customer's language, even when tool results are English. Enforced in `AGENT_SYSTEM`.
- **Multi-language KB recall**: when cosine similarity < 0.55 and query isn't English, `searchKB` auto-translates to English via `qwen-turbo` and merges results. Rarely triggers — `text-embedding-v4` handles most EU languages well.
- **Duplicate `addChatToOpportunity` call**: `server.js` lines ~1086-1088 call this function twice for the same conversation/opportunity. Likely a bug — second call creates a duplicate note.
- **`start.sh` Chatwoot unconditional**: the Twenty section checks for `docker-compose.yaml` existence first, but the Chatwoot section runs `docker compose up` unconditionally — will fail if `./chatwoot/` is absent.
- **Not a git repo**: the project root has no `.git/`. `./chatwoot/` has its own `.git/` (separate checkout). `./twenty/` may or may not be present.
- **Secrets in configs**: `*.config.json` and `docker-compose.yml` contain real API keys/tokens (DashScope, Chatwoot access token, Twenty JWT, `AGENT_BOT_SECRET`). Treat as sensitive; do not commit to public repos.
- **Resource limits**: postgres 0.5 CPU/512M, redis 0.3 CPU/256M, bridge 0.5 CPU/512M (in `docker-compose.yml`). LLM/embedding calls are external network — watch latency under load.
- **Dockerfile base image**: `node:24-alpine`. The comment says "Node 18+" but `FROM node:24-alpine`. Node 24 has built-in `fetch`/`FormData`/`Blob`.
- **Bridge auto-creates follow-up tasks**: after creating an Opportunity, `createLeadInCRM` auto-creates a Task due in 3 days, linked to the opportunity. This is in bridge code, not Twenty's workflow engine (Twenty API key lacks permission to create workflows via GraphQL).
- **Opportunity extraction logic**: `EXTRACT_SYSTEM` requires 3 elements to set `ready=true`: company name + contact info (email/phone/name) + product/need. Missing any → `ready=false`, no CRM record created.