Blog · Method
How to find what your AI voice agent can’t answer
To find what an AI agent can’t answer, pull out every moment where a customer asked for something the agent couldn’t give, take the customer’s request rather than the agent’s refusal, group requests that one answer would cover, and rank the groups by how often they happen. On a public run of 10,437 dialogues, this turned 5,153 unanswered moments into 24 substantive gaps that cover 49.5% of them.
Step 1: Decide what counts as a miss
A miss is a moment where the customer asked for something and did not get it: the agent declined, said it didn’t know, or found no match. Write the rule down before you look at the data, so you aren’t tempted to move it later.
Step 2: Keep the customer’s words, not the agent’s
Refusals all sound alike (“I’m sorry, I don’t have that”), so grouping on them tells you nothing. The customer’s request carries the information: what they wanted to know. Group on that.
Step 3: Group by meaning
Customers ask the same thing in many ways. Turn each request into a sentence embedding and group requests that are close in meaning. In the public run we used the all-MiniLM-L6-v2 model and a cosine similarity cutoff of 0.58, fixed before the corpus was read and never tuned.
Then apply one plain rule when naming groups: two requests are the same gap if one knowledge-base entry would answer both.
Step 4: Rank, then sort by how each gap can be fixed
Rank gaps by how often they occur. Then label each one: ENTRY if one knowledge entry would answer it, TOOL if it needs a live lookup or an action. Only ENTRY gaps should go to a person as a question. TOOL gaps are integration work.
Step 5: Set pass marks before you look
Decide in advance what result would mean the method works. Ours were: at least 400 candidate moments to proceed, then at least 8 gaps, at least 15 recurrences in the largest gap, and at least 40% of moments covered.
What did the public run find?
- Corpus: MultiWOZ 2.2, train, dev and test: 10,437 dialogues and 143,044 turns.
- Unanswered moments: 5,153, well above the 400 needed to proceed.
- Grouping: 680 micro-clusters, named into 62 gaps. 38 of those are “No [cuisine] restaurant” rows, an artifact of the dataset, which leaves 24 substantive gaps.
- Coverage: the 24 substantive gaps hold 49.5% of all moments (2,550 of 5,153); 63.9% counting all 62.
- Largest gap: 1,065 moments (a booking slot that was unavailable, a TOOL gap).
Every merge, and every cluster left out with its reason, is published on the method page.
What this run does not show
MultiWOZ is a research corpus with a bounded database; its misses are mostly database misses, not production knowledge-base gaps. 38 of the 62 named gaps are absent-cuisine rows from an ontology that lists cuisines with no venues. This demonstrates that the engine works on data it has never seen. It does not demonstrate demand.
Common questions
Do I need embeddings to do this?
For a few hundred conversations, a careful manual read works. Past that, grouping by meaning is what makes the list short enough to act on.
Why fix the settings in advance?
If you tune the cutoff after seeing results, you can make almost any number look good. Fixing it first makes the result believable.
Does this work for chat and SMS as well as voice?
Yes. The method works on transcripts, so the channel doesn’t matter. The public run used text dialogues.
Connect read-only, or let us run a free 48-hour gap report for you.