Finding Before Answering

A good answer about a document depends on finding the right part of it first. This guide follows one question through each step, using a fictional online shop. The shop, its policies, and the replies below are an authored illustration, not a recorded system.

The shop keeps a handful of policy documents. Among them are these passages:

A. Items can be returned within 30 days of delivery with a receipt.

B. Orders placed from November 15 to December 24 can be returned until January 31.

C. Sale items are final sale.

D. Standard shipping takes 3 to 5 business days.

A visitor asks: "I ordered boots on November 20 for the holidays. Does the holiday return window cover them, and until when?"

The system searches first and keeps a shortlist of passages that might help.

Keyword search looks for the words in the question. It finds "return" in passages A and B. It is good at exact terms such as an order number. Embedding search turns the question and each passage into numbers that represent meaning, then looks for passages close in meaning. It can connect "for the holidays" with "November 15 to December 24" even though they share no words. Anthropic's contextual retrieval article explains both, including how embeddings can miss exact matches.

Hybrid search runs both and merges the results into one ranked list. Elastic's hybrid search documentation describes this. For our question, a hybrid search might return A, B, and D.

Finding passages and then writing from them is often called retrieval-augmented generation, or RAG, after a 2020 research paper that combined a retriever with a language model.

A shortlist is not an answer. A choosing step, sometimes called ranking or reranking, decides which passage answers this question. Here it should pick B, since A is the general rule and D is about shipping. Choosing from a fixed set is a different job from writing, and Writing an Answer or Choosing One covers when each one fits.

A generator then writes the reply from B: "Yes. Orders placed between November 15 and December 24 can be returned until January 31."

Code can check some parts exactly: that November 20 falls inside the window, and that the reply states the deadline. A required field is either present or missing.

Whether the sentence means what passage B says still takes a reader. The answer-checking guide walks through that check.

Now delete passage B from the documents and ask again. Search returns A and D. The choosing step still picks one, since picking is its job. A looks closest, so the reply becomes: "Returns are accepted within 30 days of delivery."

That reply restates a real policy and cites a passage it was given. It also answers a question the visitor did not ask, because nothing on the shortlist mentions a holiday window. A more confident chooser cannot help: it can rank only what search handed it. As the bounded decisions guide puts it, a score over the candidates cannot tell you whether the right candidate was missing.

One way to get there is to give the choosing step an allowed "none of these answers it" option that routes to a fallback. The right result here is a clear "need more information": "I couldn't find a holiday return policy. The general policy allows 30 days from delivery. I'll pass your question to a person who can confirm holiday orders."

One score for the final reply hides where things went wrong. Check each stage separately, using test questions whose answering passage you already know.

• Coverage. Did the answering passage reach the shortlist? Ragas calls a version of this context recall.
• Choice. When it was there, did the choosing step pick it?
• Staying within the evidence. Did the reply say only what the chosen passage supports? Ragas measures something similar as faithfulness. It uses a model as the judge, so treat it as an estimate.

Include questions the documents cannot answer, and count a clear "need more information" as a pass for them.

Widening the shortlist can improve coverage. It also gives the choosing step more to sort, puts more text in front of the generator, and takes time. Measure how wide your shortlist needs to be on your own questions instead of borrowing a number.

A visitor asks whether a discounted jacket can be returned. The reply says: "Yes, within 30 days of delivery," citing passage A. Passage C, "Sale items are final sale," is in the documents but was not on the shortlist. Which stage needs fixing?

It is tempting to blame the generator. But the reply stayed within passage A, so a faithfulness check would pass it, and the choosing step picked the best passage it had. The miss happened at search: the question said "discounted" and the policy says "sale," so keyword search found no match, and the embedding search did not rank C highly enough. Fix coverage first: add test questions that use other words for sale, and check whether C reaches the shortlist.

To check one reply against its sources, try checking an AI answer. For when a model should choose rather than write, read writing an answer or choosing one, then should a model make this decision? for error limits. Designing the system around a model covers search, tools, and testing in a larger system.