Support teams do not need an AI that answers everything. They need one that handles the routine questions well, recognises the ones it should not touch, and hands those over with enough context that the person does not start from scratch.
This post describes a triage workflow with those three jobs: classify, answer when safe, escalate when not.
Design for the handoff first#
Most triage projects start with the answering and treat escalation as an afterthought. Reverse that. The quality of your handoff determines how much people trust the system.
A good handoff gives the human:
- The customer's message, in full.
- What the agent understood, in one line.
- What it already tried or looked up.
- Why it escalated: a rule, low confidence, a sensitive topic, or an explicit request for a person.
- A suggested draft reply, clearly marked as a draft.
Without that, escalation means the customer repeats themselves, which is the thing they are already annoyed about.
Step 1: Classify#
A first step labels each message from a short list you define: order status, returns, billing, technical fault, complaint, account change, other. Keep the list short and mutually exclusive, and include an "other" so the model has somewhere to put what does not fit.
Also extract the fields you will need later: order number, product, language, and whether the customer is asking for a person.
Run a small test set before going live. Take a few hundred real past messages, label them by hand, and compare. Where the model and your team disagree, you will find either a vague category or a prompt that needs an example.
Step 2: Decide who handles it#
Routing should combine the classification with rules that do not depend on the model:
| Situation | Route |
|---|---|
| Routine category, answer found in the knowledge base | Draft an answer |
| Complaint, legal threat, safety issue, or mention of a regulator | Human, always |
| Billing disputes and refunds above your limit | Human approval |
| Customer asks for a person | Human, immediately |
| Message in a language you do not support well | Human or a translation-first path |
| Classified as "other" | Human |
Notice that sensitive categories are a rule, not a prediction. Do not ask the model whether this seems like a legal threat and trust its confidence. Match keywords and categories, and let the model add to that list, not replace it.
About confidence#
It is tempting to have the model report a confidence score and escalate below some number. Be careful. A number a model writes about itself is not a calibrated probability, and it varies between runs. Use it as one signal among several rather than the gate.
Signals that hold up better:
- Whether the knowledge base returned a relevant passage at all. No match means no answer.
- Whether the draft quotes or uses the retrieved text, or goes beyond it.
- Whether a second, cheaper check agrees: for example, a step that asks "is every claim in this reply supported by the passages below?"
- Whether the customer has already written back about the same issue.
Tune the gate with your test set, not by feel.
Step 3: Answer from your knowledge, not from memory#
Connect the agent to a knowledge base of your help articles, policies and product details. Tell it to answer only from retrieved passages and to say so when it cannot find an answer. Include the source article in the reply or in the internal note so the reviewer can check it.
Keep the knowledge base current. An agent answering from last year's returns policy is worse than no agent. Assign someone to own it, and review the cases where the agent found nothing, because those show you the gaps.
Step 4: Review before sending#
For the first weeks, put a person on every drafted reply. Track how often they send it unchanged, edit it, or discard it. When the unchanged rate is high on a category and the stakes are low, let that category go straight through. Keep a sample of those for review.
If your channel is WhatsApp, remember the platform's rules. Meta's documentation describes a 24-hour customer service window that starts when the customer messages you; inside it you can send free-form replies, and after it closes you can only send pre-approved template messages. A slow human queue can push a reply outside the window, so design the escalation to respond within it, or to use an approved template.
Step 5: Escalate with context#
When the route is a person, create the handoff described earlier. Post it where the team already works, with the customer's details, the summary, the draft and the reason. Then let the person reply, and send that reply through the same channel.
Building it in Cadrix#
- Trigger from your channel: a WhatsApp message, a mailbox, or a webhook from your helpdesk.
- LLM step that classifies the message and extracts fields into a fixed structure.
- Condition applies your rules: sensitive categories, explicit requests for a person, unsupported languages. These go to step 7.
- Agent with a knowledge base attached drafts an answer from retrieved passages, and notes which passages it used. For order status, give it a tool or an HTTP step to look up the order.
- Condition checks that retrieval found something and that the check step passed.
- Human Input shows the customer message, the draft and the sources; the agent's draft can be approved, edited or rejected. After you trust a category, route low-risk ones around this step.
- Escalation branch sends the full handoff to the team and notes the reason.
- Send the reply by WhatsApp or email.
You can follow the whole thing in the run trace, including which passages the agent was given. When a reply was wrong, you can see whether the knowledge was missing, the retrieval missed it, or the model ignored it.
Measure what matters#
Track four things: the share of messages resolved without a person and not reopened within a few days, the edit rate on drafts, the time from message to first reply, and the number of escalations that were unnecessary. Be wary of "deflection" as a goal on its own, because a customer who gave up is not a resolved customer.
