# Method — Unknown Ops gap engine on MultiWOZ 2.2

**Public data only.** No client data, client counts, client gap labels or results from any other engagement were used, referenced or compared against. The pipeline (`run.py`, `step0.py`) was written fresh for this demonstration.

## Corpus
MultiWOZ 2.2 — github.com/budzianowski/multiwoz, `data/MultiWOZ_2.2`, train + dev + test: 10,437 human–human dialogues, 143,044 turns. The public MultiWOZ venue database (`db/`) was used only to decide ENTRY vs TOOL.

## Step 0 — candidate population (go / no-go set before counting: E ≥ 400)
B (NoOffer / NoBook acts) 4,397 · C (lexical, no such act) 130 · D (user question with no schema slot — heuristic) 684 · **E = 5,153 → GO**. See `step0_report.md`. D's precision is low (most are in-schema questions the annotators left untagged); E without D is 4,527, so the go decision does not depend on it.

## Clustering — parameters fixed before any output was seen
- Model: `sentence-transformers/all-MiniLM-L6-v2`, normalised embeddings
- Text embedded: the **user's request** (for B/C, the user turn immediately before the refusal)
- Agglomerative clustering, average linkage, cosine; similarity threshold **0.58**, locked, not swept
- Only micro-clusters with ≥ 5 members go to the naming pass; smaller ones are the long tail

**Raw micro-clusters: 680** (180 with ≥ 5 members, covering 4,320 moments; 304 singletons). **Grouped: 62 named gaps.**

## The locked same-gap rule
> Two moments are the same gap if one single knowledge-base entry would have answered both.

For TOOL-class moments, where no entry can answer, this is read as "one fix (one tool) would have served both". That reading is disclosed because it is what allows the largest gap to merge hotel and restaurant booking failures.

**ENTRY vs TOOL** (decided against the public venue database, not against the transcripts): ENTRY if the requested thing is absent from the database, exists in ≤ 3 venues (one entry can list them), or is an attraction type confined to known areas; TOOL if the answer depends on live availability or a multi-attribute search.

## Every merge
All in `merged_raw_clusters.json` — each named gap lists its raw cluster ids and the reason. Reviewed clusters that were **not** named, with the reason:
- **mixed — the locked rule cannot unify the members (different facts would answer them):** 14 clusters, 429 moments
- **context-dependent fragment — the request only makes sense with earlier turns (an area, a price, a day, a party size):** 29 clusters, 352 moments
- **not a gap — in-schema request the annotators left untagged (step-0 D noise: reference numbers, addresses, star rating, cost, travel time, 'help me find a place'):** 14 clusters, 244 moments

## Thresholds (set in the brief before the data was touched)
| Check | Result | Threshold | |
|---|---|---|---|
| Named gap clusters | 62 | ≥ 8 | met |
| Occurrences in largest cluster | 1065 | ≥ 15 | met |
| Share of candidates in a named cluster | 63.9% | ≥ 40% | met |

No persistence metric: MultiWOZ dialogues carry no meaningful timestamps, so none is reported.

## What this demonstrates — and what it does not

**What this demonstrates:** the engine ingests a corpus it has never seen, identifies the moments where a user asked for something the system could not satisfy, and resolves many phrasings into a small number of named, ranked, fix-classified gaps — using a pipeline whose model, threshold and grouping rule were locked before the corpus was read.

**What it does not demonstrate:** that MultiWOZ has a knowledge-base problem. It is a research corpus with a bounded database, and its misses are mostly database misses. The wizards had a database and mostly could answer; the dataset was built so tasks complete. Nothing here is drawn from, or compared with, any production deployment.

## Known weaknesses, stated plainly
- The naming / merging pass is a human judgment applied with the locked rule; it is fully listed so it can be checked.
- The largest gap merges hotel and restaurant booking failures under the TOOL reading of the rule — the loosest merge in the table.
- D-sourced moments inside named gaps may be untagged ordinary questions rather than failures.
- Per brief, the worked cluster is the second-largest gap; at 711 members it is a TOOL gap (multi-attribute hotel search), not a single-fact entry.

## Addendum: figures corrected after the first pass

The figures above are the raw run. Two adjustments were made after reviewing
the named gaps; the corrected figures are the ones reported publicly.

**The absent-cuisine artifact.** 38 of the 62 named gaps are "no [cuisine]
restaurant" rows: the MultiWOZ ontology lists cuisines the venue database has
no entries for at all. These are artifacts of a research corpus, not gaps a
real business would have. **Substantive gaps: 24.**

**Coverage, stated both ways.** 63.9% of candidate moments land in one of the
62 named gaps. Counting only the 24 substantive gaps, coverage is 49.5%
(2,550 of 5,153).

**ENTRY vs TOOL, by occurrence.** The threshold table counts gaps, not moments.
By moment: ENTRY 1,339 (41%), TOOL 1,956 (59%). By gap count the ratio is far
more favourable; occurrence is reported instead because it is the honest
measure: most unanswered moments need a live lookup, not one knowledge-base
entry.