Before a support bot talks to customers: knowledge, boundaries and an exam it has to pass

Testing an AI chatbot before launch comes down to one question asked many times: when a customer asks this, does the bot answer from your material, or does it make something up? A bot that says “I don’t know, let me get a colleague” is doing its job. A bot that invents a refund policy, a delivery date or a discount is creating a liability with your name on it.

The liability is not hypothetical. In February 2024, the British Columbia Civil Resolution Tribunal ordered Air Canada to compensate a passenger whose chatbot had told him he could claim a bereavement fare retroactively, which contradicted the airline’s own policy page. The airline argued the chatbot was responsible for its own actions; the tribunal called that a remarkable submission and found the airline had not taken reasonable care to make sure the bot was accurate. This guide covers the four checks that prevent that kind of answer: knowledge, boundaries, actions and an exam.

Check one: what the bot actually knows

A support bot should not answer from the general knowledge of its language model. It should look up your material, your website, price list, policies and FAQs, and answer from what it finds. When nothing relevant comes back, the right answer is a handover, not a guess.

That makes knowledge the first thing to test, and in practice the most productive. Most bad answers are not tone problems or model problems. They are retrieval problems: the policy exists, but the bot pulled the wrong paragraph, or the price list was never connected, or a page was indexed before it was updated.

The test is direct. Take ten questions customers really ask, and for each one look at which fragments of your material the bot retrieves, before it writes anything. If the right paragraph is not in the results, no amount of prompt tuning will produce the right answer. Fix the source: add the missing text, split a long page into focused sections, or remove an outdated document that keeps winning.

Writing material a bot can find

Documents written for people are not always easy for a bot to search. A few editing habits make retrieval far more reliable:

• One topic per section. A page titled “Delivery, returns and warranty” tends to return all three when the customer only asked about one. Split it.
• Say the thing in the words customers use. If customers ask about “money back” and the policy only says “reimbursement”, add the plain phrase.
• Date anything that changes. A price list with “valid from 1 October” lets you spot stale material at a glance.
• Delete superseded versions. An old policy left in the knowledge base will be found, and quoted, sooner or later.
• Write down what the bot must not promise. “We do not quote delivery dates in chat; a person confirms them” is knowledge too.

Some systems also generate likely questions for each fragment of your material, so a customer’s phrasing matches even when your text uses different words. It helps retrieval when customers phrase things differently from your documents, at the cost of extra processing, and it is worth switching on for the material customers ask about most.

Check two: boundaries and handover

A bot needs explicit rules for when it stops. The useful ones are short:

• Answer only from the knowledge base. No outside facts about your products, prices or policies.
• Hand over when asked. A customer who asks for a person gets one, without being argued with.
• Hand over when stuck. If the conversation goes in circles, a person takes it before the customer gives up.
• Blocked topics. Legal advice, medical questions, competitors, whatever your business should not discuss through a bot.

Memory is a boundary too. Decide how much the bot knows about the person it is talking to. A bot that greets a returning customer by name and recalls their last order is helpful; one that surfaces details the customer never shared with it is unsettling. Choose deliberately rather than inheriting whatever the default is.

Check three: what the bot is allowed to do

Modern support bots do more than talk. They find a customer’s record, create a task for the team, leave a note, send a message. Each of those actions carries a different risk if the bot misreads the conversation, so each should be classified rather than switched on wholesale.

The rule that matters most: a destructive action should never run because a stranger on your website asked for it. It should arrive as a request a team member approves. Equally, a switch in the settings should actually control something. If you turn an action off, the bot must be unable to call it, not just discouraged from it.

Check four: the exam

Once the knowledge, boundaries and actions are set, put the bot through an exam. A good exam has three kinds of question, and the third is where most bots fail:

1. Ordinary questions the knowledge base covers. The bot should answer correctly and briefly, from the source.
2. Questions it cannot know: a specific price not in the list, a delivery date, a discount, a promise. The right answer is to say so and offer a person.
3. Off-topic and pushy questions: requests to ignore its instructions, demands for a refund “because the bot said so”, questions about competitors. The right answer stays in role, politely.

Grade every answer on one criterion: was it grounded in the material, or an honest admission of not knowing? Anything else, an invented fact, a confident estimate, a commitment, is a fail regardless of how fluent it sounds. Then set a pass mark and hold to it. A bot that scores below the mark does not get switched on, however close the launch date is.

How Nextora builds and tests a bot

In Nextora, a bot is described rather than configured. The New bot button opens a gallery, and the workshop builds the bot from your description and any attached files: it clarifies the job, writes the role and boundaries, shows them to you and runs an exam. Accepting the result creates the bot as a draft, and a draft does not answer customers.

The Knowledge section lists everything the bot answers from: websites, files and texts, each with its indexing state. Retrieval lab shows search through the bot’s eyes: type a real customer question and see the same fragments the bot would pull, with a total score and separate scores by meaning and by words. The Unanswered questions card collects real customer questions from the last 30 days that found no answer, with how many times each was asked, a ready-made list of gaps to fill.

The What it may do section lists the CRM actions the bot can take, drawn from the same action registry that executes them, so switching an action off really stops it. Each row shows a risk class and whether the action runs immediately when a visitor asks or reaches you first; a destructive action is never carried out at a stranger’s request. Boundaries holds the handover and answer-from-own-material switches, and the Safety block adds blocked topics and when to call a person if a conversation stalls.

The Exam tab generates the questions an ordinary customer would ask, including stress cases and off-base ones about things the knowledge does not cover, puts them to the bot’s working draft and grades each answer. A failed case can be closed on the spot by adding the answer. The pass mark is 70 out of 100, and below it the bot cannot be switched on. Beside the exam, Try it yourself lets you ask your own questions, and the sandbox shows a step-by-step trace of how the bot reached its reply.

Bots meet customers on the website widget and in connected messengers such as Telegram and Instagram, and their conversations land in the same queue as every other channel. One queue for every channel covers how that works, and the AI page shows the rest of the setup.

After launch: the loop that keeps it honest

Launch is the start of testing, not the end. Prices change, policies get rewritten, a new product appears, and the knowledge base falls behind unless someone owns it. A weekly fifteen-minute routine is enough for most teams:

1. Read the unanswered questions from the past week.
2. Add the missing material, or decide the question should always go to a person.
3. Spot-check three conversations that ended in a handover. Was it the right call?
4. Re-run the exam after any change to knowledge or flow, before publishing it.

A support bot is the narrow end of AI in a business: it answers, within limits, and hands over. If you are thinking about AI that works more like a colleague, with a role, a manager and its own queue of work, that is a different design, covered in AI employees, not chatbots.