How to regression-test your AI chatbot's answers

Updated July 26, 2026

Every knowledge edit and prompt tweak can silently change what your bot tells customers. Golden questions catch the regression before a visitor does.

Support AI has a failure mode software teams will recognize: the regression. You fix one article, tighten one prompt line, swap one model version — and a question the bot answered correctly for months quietly starts getting a different answer. Nobody notices, because nobody was asking. The first person to find the regression is a customer, and their report arrives as a churn signal, not a bug ticket.

The fix is the same one software found decades ago: a test suite. For support AI it's called golden questions — a saved set of real customer questions with known-good expectations, re-run against the live answering path after every change that could move an answer. Not a benchmark, not a model eval: a regression suite for your specific bot on your specific knowledge.

Building the set takes an hour. Pull your ten most-asked questions from real transcripts — pricing, refunds, setup, the works — phrased the way customers actually phrase them. For each, decide the assertion: some questions have a checkable fact (‘the answer must contain 29'), others just need the bot to answer confidently without escalating. Then add the trap questions: things your bot should NOT know — a fax number you don't have, a discount that doesn't exist, a competitor's feature. A trap that passes by refusing to invent is the most reassuring green check in the suite.

Run the suite at the moments regressions are born: after editing knowledge articles, after any system-prompt change, after switching models or providers, and on a schedule as your provider silently updates theirs. The runs must go through the same grounded answering path visitors hit — testing a different code path tests nothing — and must never leak into real conversations or usage metrics.

Read failures as work items, not verdicts. An expected-phrase miss usually means the article changed meaning (fix the article) or the phrasing drifted (fix the assertion). An unexpected escalation means grounding got thinner. A trap that starts 'passing' by inventing an answer is the emergency: your bot got more confident than its knowledge. Each failure names its own fix, which is what makes the suite compound in value while ad-hoc spot-checking never does.

Dchat ships this as 'Answer tests' in the AI Sandbox: saved questions with optional must-contain phrases, re-run in one click through the live grounded path, pass/fail persisted per question, nothing written to conversations or quota. It is deliberately not automated prompt rollback or provider-certified QA — it's the regression suite, and the regression suite is the part that catches the customer-facing surprise.

Golden questions are a regression suite for your bot: real questions, known-good expectations, re-run after every knowledge or prompt change.

Include trap questions the bot should refuse — a trap that passes by not inventing is the strongest signal in the suite.

Run through the live grounded path, never a test-only path, and treat every failure as a named work item.