Treating retrieval itself as the classifier

Here is a query that broke our RAG routers:

"How do we validate the incrementality estimate our MMM is producing?"

Read it once and the answer seems obvious — it says "MMM" right there. Read it twice and the problem shows up. Incrementality is a causal quantity. Validation of a marketing mix model, in practice, means a geo test or a switchback — an experiment. And the model producing the estimate is - the MMM. One sentence, three domains, and a correct answer requires all three corpora.

Our knowledge base spans exactly those three domains: Marketing Mix Modeling (MMM), Experimentation (EXP), and Causal Inference (CAU).

They are separate disciplines with separate literatures. They also share an enormous amount of vocabulary. Incrementality, Geo lift test, confounding, attribution, Contribution, holdout, effect size — every one of those words appears across all three corpora, meaning something adjacent but not identical each time.

So the router has to be multi-label. Any of the following seven non-empty subsets is a legitimate answer: MMM, EXP, CAU, MMM+EXP, EXP+CAU, CAU+MMM, or all three.

This research study is about how we built that router, why we stopped asking an LLM to do it, and the specific thing that made it work: treating retrieval itself as the classifier.

---

Why the obvious approaches didn't work

Ask an LLM to label the query. This is the first thing everyone tries, and it works fine right up until the domains overlap. The failure mode is subtle: a prompted classifier answers "does this query sound like MMM?" — a judgment rendered against the model's pretraining priors about what MMM means. That prior is generic. It has no idea what is actually in our corpus. So it confidently routes an incrementality question to CAU because the causal-inference literature it saw during training owns that word, while the doc that actually answers the question sits in our MMM collection. The label is defensible in the abstract and useless in practice.

Fine-tune a classifier. Now you need labeled data for seven classes, the combination classes are rare, and the whole thing has to be retrained every time the corpus changes shape. We add documents weekly sometimes even daily.

Skip routing entirely — one big index, search everything. We tried this. It fails for a reason worth stating plainly, because it's the reason the rest of the design exists.

Corpus imbalance eats the small domains. Our MMM corpus is several times the size of the other two. On a plain top-k search across the whole store, MMM documents win most of the slots — not because they're more relevant, but because there are more of them competing. A pure-EXP query would come back with a top-10 that was majority MMM. The generator, handed that context, produced answers that were plausible, fluent, and quietly wrong.

That failure is what pointed at the fix. The problem wasn't that the router didn't know what the domains meant. The problem was that we were asking the wrong question.

---

The reframe: retrieval as the classifier

We attacked the problem differently. We brainstormed and said "Don't ask "which domain does this query belong to?".

That's a question about taxonomy, and it needs a model with an opinion about your domains.

Instead ask "which domains have documents that can actually answer this?" That's a question about your corpus, and your corpus can answer it directly.

The second question is also the one that matters.

Routing to a domain that holds nothing relevant is a wasted retrieval no matter how philosophically correct the label was.

Routing away from a domain that holds the answer is a broken response no matter how confident the classifier sounded.

So: our light bulb momemt was to run the query against the store, and read each domain's retrieval as a weak binary detector for that domain's relevance. If the MMM documents come back strong, MMM is in. If the EXP documents come back weak, EXP is out. The evidence is drawn from the same index the generator will read from, which means the classifier and the retriever can never disagree about what exists.

One architectural note, because it matters for the rest: we run a single vector database, not three. Every chunk carries a domain label in its metadata. This is deliberate — one embedding space, one index, one place to write. Domain separation happens at query time through metadata filters, not at storage time through separate collections. Three collections would have forced us to compare scores across three independently-populated indexes, which, as we'll get to, is a harder problem than it looks.

---

The cascade

Three stages, cheap to expensive, each one narrowing what the next has to consider.

Stage 1 — Elimination, not selection

One unfiltered search across the whole index. Bucket the hits by their domain label. Then, for each domain, compare its evidence against that domain's own threshold and knock out the ones that fall short.

The direction here is the whole trick. We are not trying to find the right domain. We are trying to rule out the wrong ones. We even wrote about it in our article "via negativa".

That's not a stylistic preference — negative evidence is genuinely more reliable than positive evidence under corpus imbalance.

"Almost nothing from EXP cleared the bar" is a strong, trustworthy signal. "Some MMM stuff came back" is nearly meaningless when MMM outnumbers everything else and will always put something in the top-k. So Stage 1 only ever removes. If EXP is eliminated, we've gone from seven possible answers to three (MMM, CAU, MMM+CAU) using one search.

Why the thresholds are per-domain

This is the part that took the most iteration, and it's the direct consequence of the imbalance problem.

A single global threshold cannot work. Each domain's score distribution has its own shape, set by how many documents it has, how densely it covers its own subject matter, and how broad its vocabulary is. A large, dense corpus clears any fixed bar easily. A small, tight one rarely does. Set one global τ and you've just rebuilt the failure you were trying to escape: the biggest domain always survives, the smallest never does, and the threshold has silently become a popularity contest.

So each domain gets its own: τ_MMM, τ_EXP, τ_CAU, tuned separately. The threshold isn't asking "is this score high?" — it's asking "is this score high for this domain?" Read that way, a per-domain cutoff is a crude distribution normalizer. Each τ_d encodes what a good retrieval looks like for domain d specifically, so the comparison across domains becomes something closer to apples-to-apples.

Stage 2 — Discrimination among survivors

Stage 1 leaves us with candidates, say MMM and CAU. Now we re-query the index once per survivor, with a metadata filter pinning the domain.

The point is fairness. In Stage 1's unfiltered search, a domain's evidence is whatever managed to fight its way into a shared top-k — so a domain can look weak simply because it got crowded out. Filtering removes the crowding. Each surviving domain now competes only against itself and returns its genuine best evidence for this query. A domain that looked marginal because MMM ate its slots gets to make its actual case. A domain that looked marginal because it has nothing to say still has nothing to say, and now that's unambiguous.

Comparing those results tells us whether we're looking at MMM, CAU, or MMM+CAU.

Stage 3 — The relevance vote

Finally, a lightweight LLM looks at the surviving domains' retrieved chunks and answers one narrow question per document: does this actually address the query? Those judgments become a vote on whether the domain stays.

Note what the model is and isn't doing here. It is not classifying the query — we established that it's bad at that, because it reasons from priors instead of from our corpus. It's doing something much smaller and much more grounded: reading a specific document, reading a specific query, and judging fit. Both objects are in front of it. There's nothing to hallucinate about.

This catches the case that pure vector similarity cannot: chunks that are topically close but functionally irrelevant. A document explaining what a confounder is will embed close to a query about handling confounders in a specific MMM specification, and contribute nothing to answering it. Similarity sees a match. The judge sees a definition where a method was needed, and votes no.

Small model, tight prompt, runs in parallel across survivors. It's a filter on the tail of the distribution, not a reasoner.

---

The principle underneath: your errors are not symmetric

If there's one thing to take from this research study, it's this. It's the principle every threshold in the system is set against.

A false positive and a false negative do not cost the same.

• False positive (routing to a domain the query didn't need): you pull in some extra context. The generator mostly ignores it. You paid tokens and a little latency. Recoverable.
• False negative (dropping a domain the query did need): the generator answers with a hole in its context. It doesn't know the hole is there. It produces something fluent, complete-sounding, and wrong. Not recoverable, and not visible.

That asymmetry dictates the tuning. Every τ_d is set toward recall — deliberately permissive, biased to let domains survive Stage 1 when the evidence is ambiguous. We accept a pile of false positives at the elimination stage, because Stages 2 and 3 exist specifically to claw the precision back at a point where we can afford a more expensive look.

Cheap and permissive first. Expensive and strict second. Never the reverse.

---

What we'd do differently

We'll be straight about where this is room for improvement.

The thresholds are hand-fitted constants. τ_MMM, τ_EXP, and τ_CAU are entangled with the current corpus size, the current embedding model, and the current chunking strategy. Add five hundred documents to one domain and the distribution shifts underneath them. Nothing in the system will announce this. The router will just start quietly dropping a domain, and the only symptom will be answers that are subtly incomplete. This is the same corpus-imbalance failure we set out to fix, wearing a hat.

The fix is to stop hard-coding the cutoff and start normalizing the distribution. Score each domain's retrieval as a z-score against that domain's own background score distribution, or go rank-based instead of score-based and drop absolute magnitudes entirely. Then the threshold becomes a statement about distance from that domain's normal, which survives corpus growth. This is on the roadmap.

Long compositional queries need decomposing, not classifying. Something like "compare our MMM's channel contribution against last quarter's geo test and tell me whether the discrepancy is a confounder" isn't one query wearing three hats.

It's three questions in a trench coat. Classifying it whole gets you all three labels and an undifferentiated pile of context. The right move is to split it first and route each piece.

None of the mechanism here is novel, and it shouldn't be. Stripped down, Stage 1 is a kNN classifier over embeddings, and semantic-router libraries have done retrieval-based intent classification for a while. What we'd claim is the composition: multi-label elimination across domains that genuinely overlap, per-domain thresholds as a defense against corpus imbalance, and a relevance judge used as a voter rather than a decider. The pieces are old. The arrangement is where the innovation is.

---

The takeaway

The instinct when building a query router is to reach for the smartest model you have and ask it to make the call. We think that's backwards.

Your LLM knows what MMM means in general. It has no idea what's in your index. And "what's in your index" is the only thing that determines whether a retrieval succeeds.

So don't ask the model. Ask the corpus.