Why an AI agent can't find an answer that's already in its knowledge base

You ask your AI agent something you know is documented: in the product manual, an internal policy, the ticket history. And the agent says "I don't have that information," or worse, makes up something close but wrong. The first suspicion is usually that the model is bad. The real cause, almost always, is something else: the model never got to read the right passage.

Why does an agent fail when the answer is already in your documents?

Because most agents connected to a knowledge base don't read every document every time someone asks something. They first search for the passages that seem most relevant to that specific question (the technique is called RAG, retrieval augmented generation), and only those passages reach the model that generates the answer. If the search pulls the wrong passages, or none at all, the model never gets a chance to answer well. It doesn't matter how good that model is.

A Deakin University paper studied this across three real cases: two systems already in use (one for researchers, one for students) and a third, larger-scale experiment with 15,000 documents and 1,000 questions on biomedical topics. That produced a catalog of seven distinct failure points. Three of those seven happen before the generating model reads a single word: the information is missing from the base, the right passage exists but didn't make it into the top search results, or it made the results but got lost in the step that assembles and trims everything before sending it to the model. The mistake happened in the search, not in the answer.

Searching by keyword or by meaning: both fail, differently

Before LLMs, searching for information across many documents was already a solved problem, just solved differently: by keyword, indexing which term appears in which document (the classic technique is called BM25). That method fails when the customer writes something different from how the document is worded: they ask to "cancel my subscription" and the document says "downgrade the plan."

The semantic search a RAG uses solves exactly that: it turns the question and each document passage into a vector, and searches for the vectors closest in meaning, not exact text. The cost shows up somewhere else: how each document gets cut into passages. A passage that's too small doesn't have enough context to answer questions that span two ideas. One that's too long dilutes the relevant part among text that doesn't matter, and that extra text ends up competing against the real answer when the model has to decide what to use. (How to pick the right embedding model for each domain is a separate discussion, worth its own article.)

Does adding more context fix this?

Sometimes. The same study found that widening the context passed to the model, from 4,000 to 8,000 tokens in its case, improved the answers. But there's a limit, and it isn't just about size: separate research on how language models use long contexts (Liu et al., 2023, "Lost in the Middle: How Language Models Use Long Contexts") found that accuracy drops when the relevant information sits in the middle of a long context, neither at the start nor the end. Stuffing in more documents "just in case" doesn't make up for a bad search. In some cases it makes things worse, because it adds more places where the correct answer can get lost.

What we already knew before LLMs existed

Finding the right piece of information among many documents isn't a problem language models invented. It existed in search engines, in recommendation systems, in any machine learning pipeline from before 2020. And the lesson from that era still holds exactly: the quality of what goes into a system (how it's indexed, how it's chunked, how well it's preprocessed) sets the ceiling on what that system can answer, no matter how sophisticated the final step is. Swapping "statistical model" for "language model" doesn't change that dependency; it just hides it better, because the final step now writes convincing prose even when it had nothing to work with.

We used to hand-train the model that had to correct that kind of error. Today a tidier prompt gets written hoping for the same effect. The underlying problem, whether the right data made it through or not, is the same as always.

What to check before switching models

When an agent gets something wrong that's already documented, it's worth checking one step earlier than the prompt or the model: which passage actually reached that answer. Most of the time, the right answer was never on the table. That step (what got searched, what got found, what of it got shown to the model) is easier to audit than to guess, and that's where it's worth looking first.

---

Sources: Barnett, Kurniawan, Thudumu, Brannelly, and Abdelrazek, "Seven Failure Points When Engineering a Retrieval Augmented Generation System," Deakin University, 2024 (arxiv.org/abs/2401.05856). Liu et al., "Lost in the Middle: How Language Models Use Long Contexts," 2023.