Initial commit: CRM AI Demo bridge + KB knowledge base + infrastructure
- Bridge server with LLM agent tool-use (search_kb, search_products, search_orders, search_inventory, escalate_human) - pgvector RAG knowledge base (95 FAQ chunks) - Auto opportunity creation in Twenty CRM - Auto follow-up task workflow - Chatwoot AgentBot integration (HMAC, webhook) - Docker compose infrastructure (PG18, Redis 8.8, node24-alpine) - Configuration templates (example files) - real secrets excluded via .gitignore
This commit is contained in:
AI Bridge Dev
114
AGENTS.md
Normal file
114
AGENTS.md
Normal file
@@ -0,0 +1,114 @@
|
||||
# CRM AI Demo - Project Guidelines
|
||||
|
||||
## Project
|
||||
B2B stainless steel jewelry wholesaler (Yehwang, NL) CRM with AI-powered sales/service assistant bridge. Chatwoot webhook → LLM agent (tool-use) → Chatwoot reply + Twenty CRM opportunity creation.
|
||||
|
||||
## Tech Stack
|
||||
- **Bridge**: Node.js 24, CommonJS (`"type": "commonjs"`), pure Node built-ins (`http`/`fetch`/`crypto`) + `jose` + `pg`. No Express. No test/lint/build scripts — verify by running the server and hitting `/healthz`.
|
||||
- **LLM**: OpenAI-compatible provider via `bridge/llm.config.json`. Active: `aliyun` → DashScope `qwen-plus-latest` (`enableThinking: false`). `server.js` header comment says "glm-5.2/ark" — **ignore it, trust `llm.config.json`**.
|
||||
- **Messaging**: Chatwoot (self-hosted, external source checkout at `./chatwoot/` — has its own `.git/`).
|
||||
- **CRM**: Twenty CRM (self-hosted, external source checkout at `./twenty/` — may be absent).
|
||||
- **Vector DB**: pgvector on shared `crm-postgres` (pg18). Three DBs: `chatwoot`, `twenty`, `products`. KB RAG table: `products.kb_docs`.
|
||||
- **Product Search**: external recommend API (`RECOMMEND_URL`, default `http://recommend.internal.yw.com`).
|
||||
- **Order/Inventory APIs**: trading-service (`ORDER_API_URL`) and eta-service (`INVENTORY_API_URL`). Mock server available: `node mock_services.js` (port 9505).
|
||||
- **Embedding**: DashScope `text-embedding-v4` (dim 1024), configured in `kb.config.json`. Query translation fallback uses `qwen-turbo`.
|
||||
|
||||
## Architecture
|
||||
```
|
||||
Chatwoot (external) ──webhook──► Bridge (:4000) ──LLM tool-use──► Chatwoot reply
|
||||
│
|
||||
├── pgvector (products.kb_docs) KB RAG (search_kb)
|
||||
├── recommend API product catalog (search_products)
|
||||
├── trading-service order tracking (search_orders)
|
||||
├── eta-service inventory/stock (search_inventory)
|
||||
└── Twenty CRM GraphQL create Opportunity
|
||||
```
|
||||
|
||||
Bridge routes (`bridge/server.js`):
|
||||
- `GET /healthz` (and `/`) — liveness, returns `agentBot: !!AGENT_BOT_SECRET`
|
||||
- `GET /conversation/:id` and `GET /conversation/:id/html` — debug views
|
||||
- `POST /agent-bot` — Chatwoot AgentBot outgoing webhook (HMAC-signed)
|
||||
- `POST /chatwoot-webhook` — Chatwoot `message_created` webhook (fallback entry)
|
||||
|
||||
Agent tool-use loop: LLM autonomously calls `search_kb`, `search_products`, `search_orders`, `search_inventory`, or `escalate_human`. Single `AGENT_SYSTEM` persona — no prompt switching. Per-conversation lock (`locks` Set) prevents concurrent LLM calls; `createdOpps` Set prevents duplicate Opportunities.
|
||||
|
||||
## Startup
|
||||
```bash
|
||||
# Full stack (requires Docker Desktop):
|
||||
bash start.sh
|
||||
|
||||
# Shared infra only (no Chatwoot/Twenty source needed):
|
||||
docker compose up -d --build
|
||||
|
||||
# Local dev (no Docker):
|
||||
cd bridge && node server.js # Node 24+; needs PG_URL etc. as env vars
|
||||
|
||||
# Mock business APIs (for dev without real trading/eta services):
|
||||
node mock_services.js # Listens on :9505
|
||||
```
|
||||
|
||||
`start.sh` conditionally starts Twenty (if `./twenty/docker-compose.yaml` exists) and Chatwoot (if `./chatwoot/docker-compose.yml` exists — note: currently runs unconditionally, will error if absent). Shared infra always starts.
|
||||
|
||||
Logs / health:
|
||||
```bash
|
||||
docker logs -f crm-bridge
|
||||
curl http://localhost:4000/healthz
|
||||
```
|
||||
|
||||
**No rebuild needed for code changes** — `docker-compose.yml` volume-mounts `bridge/*.js` and `*.config.json` read-only. `docker restart crm-bridge` picks up edits. Only rebuild the image when changing `package.json` deps.
|
||||
|
||||
## Key Files
|
||||
```
|
||||
bridge/
|
||||
server.js Main bridge: routes, LLM agent loop, tool impls, Chatwoot/Twenty API calls (~1250 lines)
|
||||
index_kb.js KB vectorizer: customer-service-kb/*.md → embed → products.kb_docs (DROPS + recreates table each run)
|
||||
register_agent_bot.js Registers Chatwoot AgentBot (idempotent). Run: node register_agent_bot.js
|
||||
llm.config.json LLM provider config. Active provider key: "aliyun"
|
||||
kb.config.json Embedding config (model, dim, translateModel) + DashScope creds
|
||||
chatwoot.config.json Chatwoot API connection (baseURL, accountId, accessToken, inboxId)
|
||||
twenty.config.json Twenty CRM connection (baseURL, workspaceId, apiKey JWT)
|
||||
Dockerfile node:24-alpine, npm install --omit=dev, copies server.js + configs
|
||||
customer-service-kb/ FAQ markdown (source for index_kb.js). Chunks split on ### Q:. English; AI translates per customer language.
|
||||
infra/init-db.sh postgres init: creates DBs chatwoot/twenty/products + enables pgvector on chatwoot & products.
|
||||
docker-compose.yml Shared infra: pgvector/pgvector:pg18 + redis:8-alpine + bridge
|
||||
mock_services.js Mock trading/eta API server (:9505) for dev without real backends
|
||||
add_note.js CLI utility: attach a text note to a Twenty opportunity
|
||||
create_workflow.js CLI utility: create Twenty automation workflow (new opp → auto follow-up task)
|
||||
find_keys.js CLI utility: find Twenty signing key tables in postgres
|
||||
gen_apikey.js CLI utility: generate Twenty API key JWT (needs ENCRYPTION_KEY + PG_DATABASE_URL)
|
||||
landing.html Multilingual landing page
|
||||
start.sh One-shot launcher for all compose stacks
|
||||
docs/twenty-crm-guide.md Twenty CRM integration guide: which GraphQL ops bridge uses, data model, extension ideas
|
||||
```
|
||||
|
||||
## Environment Variables
|
||||
Env vars always override `*.config.json` values. Key overrides in `server.js`:
|
||||
- `CW_BASE_URL` — overrides `chatwoot.config.json` baseURL
|
||||
- `TWENTY_BASE_URL` — overrides `twenty.config.json` baseURL
|
||||
- `PG_URL` — postgres connection (default `postgres://postgres:postgres@localhost:5432/products`)
|
||||
- `AGENT_BOT_SECRET` — HMAC signing key for AgentBot webhook verification
|
||||
- `RECOMMEND_URL` — product recommendation API base URL
|
||||
- `ORDER_API_URL` — trading-service base URL (order queries)
|
||||
- `ORDER_API_KEY` — auth key for trading-service
|
||||
- `INVENTORY_API_URL` — eta-service base URL (stock/ETA queries)
|
||||
- `INVENTORY_API_KEY` — auth key for eta-service
|
||||
- `DASHSCOPE_API_KEY` — overrides `kb.config.json` apiKey (avoids hardcoding in config)
|
||||
- `PORT` — bridge listen port (default 4000)
|
||||
|
||||
Docker-compose sets these for the bridge container (using container names for inter-service routing, `host.docker.internal` for host-side services).
|
||||
|
||||
## Conventions & Gotchas
|
||||
- **Config loading**: `loadCfg()` reads `*.config.json` from `__dirname` (the bridge dir). Env vars always win — check `process.env.*` overrides in `server.js` before editing config files.
|
||||
- **KB re-indexing**: `index_kb.js` **DROPS and recreates** `kb_docs` every run (no incremental update). Any model/dimension change requires a full re-vectorize. Run `node index_kb.js` after editing any `customer-service-kb/*.md` file.
|
||||
- **KB doc format**: YAML frontmatter (`title`/`category`/`keywords`/`source_url`) + `### Q:`/answer blocks. Embedding text = `keywords + Q + A`. Source URLs are mapped per-section in `index_kb.js` `SECTION_URL_MAP`.
|
||||
- **Agent persona is the source of truth** for sales policy (discount tiers, minimum order €30, contacts). `AGENT_SYSTEM` in `server.js` encodes business rules; `customer-service-kb/` encodes policy FAQs. Keep them consistent.
|
||||
- **Language rule**: replies must mirror the customer's language, even when tool results are English. Enforced in `AGENT_SYSTEM`.
|
||||
- **Multi-language KB recall**: when cosine similarity < 0.55 and query isn't English, `searchKB` auto-translates to English via `qwen-turbo` and merges results. Rarely triggers — `text-embedding-v4` handles most EU languages well.
|
||||
- **Duplicate `addChatToOpportunity` call**: `server.js` lines ~1086-1088 call this function twice for the same conversation/opportunity. Likely a bug — second call creates a duplicate note.
|
||||
- **`start.sh` Chatwoot unconditional**: the Twenty section checks for `docker-compose.yaml` existence first, but the Chatwoot section runs `docker compose up` unconditionally — will fail if `./chatwoot/` is absent.
|
||||
- **Not a git repo**: the project root has no `.git/`. `./chatwoot/` has its own `.git/` (separate checkout). `./twenty/` may or may not be present.
|
||||
- **Secrets in configs**: `*.config.json` and `docker-compose.yml` contain real API keys/tokens (DashScope, Chatwoot access token, Twenty JWT, `AGENT_BOT_SECRET`). Treat as sensitive; do not commit to public repos.
|
||||
- **Resource limits**: postgres 0.5 CPU/512M, redis 0.3 CPU/256M, bridge 0.5 CPU/512M (in `docker-compose.yml`). LLM/embedding calls are external network — watch latency under load.
|
||||
- **Dockerfile base image**: `node:24-alpine`. The comment says "Node 18+" but `FROM node:24-alpine`. Node 24 has built-in `fetch`/`FormData`/`Blob`.
|
||||
- **Bridge auto-creates follow-up tasks**: after creating an Opportunity, `createLeadInCRM` auto-creates a Task due in 3 days, linked to the opportunity. This is in bridge code, not Twenty's workflow engine (Twenty API key lacks permission to create workflows via GraphQL).
|
||||
- **Opportunity extraction logic**: `EXTRACT_SYSTEM` requires 3 elements to set `ready=true`: company name + contact info (email/phone/name) + product/need. Missing any → `ready=false`, no CRM record created.
|
||||
Reference in New Issue
Block a user