Thirty-One Questions, One Retriever, No Embeddings

Retrieval Is the Layer Under the Answer

Ask an AI assistant a question about your own documents and something has to find the right page first. That something is retrieval. The model only writes the answer. The retriever decides what the model gets to read.

Retrieval-augmented generation is the name for this pattern. It gets sold with a lot of vocabulary: embeddings, vector databases, semantic search. Most of that is a vendor’s shopping list, not a requirement for a small corpus.

We built a small one. Ask the Field Notes is a live tool on this site. It answers questions using only the ten Field Notes published before it, and it cites a source for every sentence. This article covers the layer under the conversation. It explains how the articles were chunked and how a question finds the right chunk. Then it shows how we measured whether that works. The conversation layer gets its own piece under Active Chat.

The Questions Came First

The Active Eval piece set the rule: decide the right answer before anyone looks at the output. We applied it here before writing a line of retrieval code.

On September 14, 2026, we read the extracted text of all ten articles and wrote 37 questions. Thirty-one have a known answer in one named article and one named section. Six do not. Each question is phrased the way an operator would ask it, not by copying the page’s words. “Do local restaurants actually block AI crawlers?” has to find the restaurant audit. “Is GPTBot the same thing as ChatGPT-User?” has to find the robots.txt piece.

The six out-of-scope questions are the hard part. Four were chosen to trip a lazy retriever. “What does Active Reach do?” shares the word Active with every page. “How do I fine-tune Llama on my support tickets?” shares “support tickets” with the NPS article. “What’s the weather in Denver today?” shares Denver with the restaurant piece. The right answer to all six is “that is not in the Field Notes.”

The file was frozen before the first run. Nothing in it changed after. Two entries, quoted as written.

Ten Articles, 75 Chunks

A retriever ranks pieces of text, so the first decision is what a piece is. We chose one chunk per section, split at each article’s own headings. Sections past 220 words split again at a paragraph break. Each article also gets a one-chunk summary made of its title and lead paragraph. That gives “which piece covers X” questions somewhere to land.

The result was 75 chunks from ten articles, 6,368 words in total. A script extracts them from the live pages and drops the banners and download forms. It writes a plain JavaScript file the server imports at startup. That file is committed to the repository, because nothing regenerates it at deploy time. Re-running the script is now a step in the checklist every time a Field Note ships.

No Embeddings, on Purpose

The retriever is BM25, a lexical ranking function from the 1990s. It scores a chunk by how many of the question’s words it contains. Words that are rare across the corpus count for more. There is no embedding model, no vector store, and no network call. The whole index builds in memory at cold start and answers a query in well under a millisecond.

That was a decision, not a shortcut. Ten documents do not need a vector database. Anyone who tells you otherwise is selling one. But a decision like that has to be measured, not assumed. That is what the question set is for. If the plain retriever had failed the set, embeddings would have been the next step.

Three Runs, One Table

Each run scores the same 37 questions. Doc@1 means the top chunk came from the right article. Section@3 means the right article and section both appear in the top three. A false reject is a real question the retriever scored too low to answer. A leak is an out-of-scope question that scored high enough to get an answer.

The baseline was plain BM25 over the chunk text. It put the right article on top for 27 of 31 questions. It also refused two real questions. “What is a Claude Skill?” scored 5.67 against a threshold of 6. A two-word question cannot pile up much score, so the threshold alone would have turned it away.

The tuned run added three things. Light stemming, so “crawlers” matches “crawler.” The section heading and article title, folded into each chunk’s words once. Adjacent word pairs as extra tokens, so “menu schema” outranks a chunk that mentions both words far apart. One more fix came out of a failed test. “Who won the 2026 World Cup?” was matching the phrase “ranking first won’t matter.” The tokenizer had split “won’t” into “won” and “t.” Contractions now fold into one token.

That run moved Doc@1 to 29 of 31 and cleared both false rejects. It also made the one leak worse. The fine-tuning question now scored 10.11 on the NPS article, on the strength of “support” and “ticket” alone.

The third run added a coverage rule. The top chunk has to share at least half of the question’s content words. A one-word question passes on coverage alone. That closed the Llama leak. It opened another. “What does Active Reach do?” has two content words, and one chunk of the Active Forge article contains both. Coverage is 100 percent, so the low score of 3.15 no longer matters. Five of six before, five of six after. The rule moved the leak. It did not remove it.

The Misses, Named

Two in-scope questions still put the wrong article on top.

• “Which tools does the PDF verification pipeline use?” ranks a section of the Active Forge article first. That section names the PDF verification technique. The Active Ship section that lists the tools is second. The model still receives it.
• “How do I test that my structured data is valid?” ranks the GEO Readiness guide’s Structured Data section first. That section does name a validator, so the answer is still right. The section written for the question, in the schema-types article, is not in the top three.

Three more find the right article but the wrong section. The worst is “Show me an example of FAQ schema markup.” It lands on the section that measured the FAQ gap, not the one holding the markup. The chunk with the example ranks eighth. The model receives at most five, so it never sees the example. It can only say where to look.

One more limit belongs in the open. The threshold of 6 and the coverage rule were both chosen by looking at these six out-of-scope questions. They were tuned on the questions they are scored on. The honest rejection rate is the one from the next batch of real questions, which does not exist yet.

What the Model Sees

After ranking, the top chunks go to the model as numbered sources. Up to five make it, and any chunk below 40 percent of the top score is left out. If the top chunk fails the grounding rule, no model is called at all. The tool answers “that is not in the Field Notes” itself, for free.

The model gets one rule that matters: answer only from the sources, with a number after every sentence. The reply comes back with the citations. The page turns each one into a link to the section it came from. Every answer also shows its own retrieval trace, the top score and how many chunks were passed. You can watch the retriever work.

Where Else This Pattern Applies

Grounding answers in a small, known corpus generalizes past a firm’s own articles. None of the four examples below are real Active AI Advisors client work. They show where the same build applies.

• HR. A 40-page benefits handbook. An employee asks how many weeks of parental leave they get and receives the paragraph, cited, not a guess.
• Field service. Two hundred pages of equipment manuals. A technician asks about a fault code and gets the page for that code, not a summary of the manual.
• Sales. A product spec library. A rep asks whether the product supports a named integration and gets the line from the spec, with its date.
• Finance. Month-end close procedures. An analyst asks which account a specific accrual posts to and gets the procedure that says so.

In every case the question set comes first. Twenty or thirty questions with known answers, written before the retriever, decide whether the plain version is enough. Each example is illustrative, not an engagement. Ask the Field Notes is the instance of this pattern that is real and running today.

What This Article Proves

This article’s PDF was generated and rendered to a real image before it shipped, using the Active Ship pipeline. The article also produced one more result on its own. The corpus builder indexes every published Field Note, so the first draft of this piece went straight into the corpus. The frozen question set was run again, this time over eleven articles and 87 chunks. Doc@1 fell from 29 to 27. Out-of-scope rejection fell from five of six to two of six.

The cause is on this page. This article quotes the test questions word for word. It outranked the real answers for two of its own questions. It also handed “Who won the 2026 World Cup?” a chunk that matched every word. The test set had ended up inside the index. The fix was a rule, not a rewrite. A page can now mark itself excluded from the corpus, and this one does. The corpus is back to ten articles and 75 chunks, and every number in the table holds. The run with the leaks is kept as the fourth results file. Keep the answer key out of the index.

Active Retrieval is the name for this practice area. It means grounding answers in your own documents, with a question set that says how well the grounding works. Next is Active Chat, the conversation layer on this same build. That piece covers what the tool refuses, how it handles follow-ups, and when it hands off to a person.