--- title: "Why your AI search gives wrong answers | All Stack Labs" description: "Why a company chatbot or AI search gives wrong answers from your own documents, the retrieval fixes that work, and how to measure them." url: "https://allstacklabs.com/blog/why-your-ai-search-gives-wrong-answers/" --- # Why your AI search gives wrong answers. Most wrong answers start before the model writes a word, in what your search hands it. All Stack Labs, 12 October 2026 In February 2024, a tribunal in British Columbia ordered Air Canada to pay a passenger $812.02. The airline's website chatbot had told him he could claim a bereavement fare within 90 days of buying his ticket. The bereavement page that the chatbot linked to said the opposite. Air Canada argued the chatbot was responsible for its own words. The tribunal called that "a remarkable submission" and found the airline had not taken reasonable care to make its chatbot accurate. The sum was small, but the ruling was clear: when an AI answers from your documents, its answers are yours. And the research says the cause of a wrong answer is often upstream of the model. ## Where it breaks A 2024 study of three retrieval systems, two of them in use, listed seven ways they fail. Three happen before the model writes anything: - The answer is not there. Your documents do not cover the question, and the system answers anyway instead of saying so. - It ranks too low. The right passage exists, but other passages outrank it. - It gets cut. The passage is retrieved, then dropped to fit the prompt. The other four happen in the model. It misses an answer buried among noise or contradictions, or it gets the format, the level of detail or the completeness wrong. ## More context can make it worse The instinct is to hand the model more. Google Research found that when the retrieved text was not enough to answer the question, one model's rate of wrong answers rose from 10.2% with no retrieval at all to 66.1%. Partial evidence made it less willing to say it did not know. Other studies point the same way. In one, models followed incorrect retrieved text over their own correct knowledge more than 60% of the time. In another, answers placed in the middle of a long prompt were found less often than ones at the start or the end. And at 32,000 tokens of context, 11 of the 13 models in one benchmark dropped below half of their short-prompt scores. A bigger context window is not a fix on its own. ## Semantic search misses exact lookups Vector search is good at meaning and poor at exact strings. In Microsoft's tests on customer data, vector search scored 11.7 on keyword-style queries where keyword search scored 79.2. Think part numbers, policy codes, product SKUs and people's names, which is a large share of what customers and staff actually ask. Searching both ways is the starting point. On the same customer datasets, keyword and vector search combined, with a reranker on top, scored 60.1 against 43.8 for vectors alone. ## What fixes it - Hybrid search and a reranker. Anthropic added a short line of context to each chunk before indexing it, then combined vector search, keyword search and reranking. Failed retrievals in their tests fell by 67%, from 5.7% to 1.9%. - Clean documents. Retire old versions, remove duplicates and put effective dates on policies. A model will repeat an outdated page as confidently as a current one. - A tested way to say "I don't know". When the answer is not in your documents, the right response is to say so and hand off to a person. Treat that as a feature and measure it. - Chunks that fit the questions. NVIDIA found page-sized chunks gave the most consistent accuracy across datasets, while short factual questions did best with chunks of 256 to 512 tokens. - No retrieval when you don't need it. If the whole knowledge base is under about 200,000 tokens, roughly 500 pages, Anthropic suggests putting all of it in the prompt. ## Measure it, then keep measuring Start with a set of real questions your users ask, each paired with the passage that answers it. Anthropic's engineering team suggests 20 to 50 cases drawn from real failures as a first set. Add to it every time something new goes wrong. Score retrieval and answering separately. Context recall tells you whether the right passage came back. Faithfulness tells you whether every claim in the answer is supported by what came back. Open source tools such as Ragas compute both. OpenAI's evaluation guide works through an example that targets context recall of at least 0.85, and recommends running the evals on every change. Once that set exists, a model upgrade, a new batch of documents or a prompt edit stops being a guess. You can see what got better and what broke. ## Where to start When your AI search is wrong, look at retrieval and the documents behind it before you look at the model, and measure retrieval before and after every change. That is how we build retrieval for clients, and it is what lets you put the system in front of customers. ## Sources 1. [Moffatt v. Air Canada, 2024 BCCRT 149, Civil Resolution Tribunal of British Columbia (2024)](https://decisions.civilresolutionbc.ca/crt/crtd/en/525448/1/document.do) 2. [Barnett et al., Seven Failure Points When Engineering a Retrieval Augmented Generation System (2024)](https://arxiv.org/abs/2401.05856) 3. [Google Research, Deeper insights into retrieval augmented generation: the role of sufficient context (2025)](https://research.google/blog/deeper-insights-into-retrieval-augmented-generation-the-role-of-sufficient-context/) 4. [Wu, Wu and Zou, ClashEval: Quantifying the tug-of-war between an LLM's internal prior and external evidence (2024)](https://arxiv.org/abs/2404.10198) 5. [Liu et al., Lost in the Middle: How Language Models Use Long Contexts (2023)](https://arxiv.org/abs/2307.03172) 6. [NoLiMa: Long-Context Evaluation Beyond Literal Matching (2025)](https://arxiv.org/abs/2502.05167) 7. [Microsoft, Azure AI Search: Outperforming vector search with hybrid retrieval and ranking capabilities (2023)](https://techcommunity.microsoft.com/blog/azure-ai-services-blog/azure-ai-search-outperforming-vector-search-with-hybrid-retrieval-and-ranking-ca/3929167) 8. [Anthropic, Introducing Contextual Retrieval (2024)](https://www.anthropic.com/news/contextual-retrieval) 9. [NVIDIA, Finding the Best Chunking Strategy for Accurate AI Responses (2025)](https://developer.nvidia.com/blog/finding-the-best-chunking-strategy-for-accurate-ai-responses/) 10. [Anthropic, Demystifying evals for AI agents (2026)](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) 11. [OpenAI, Evaluation best practices](https://platform.openai.com/docs/guides/evaluation-best-practices) 12. [Ragas documentation](https://docs.ragas.io/en/stable/) Related [AI infrastructure](https://allstacklabs.com/services/ai-infrastructure/) [Book a strategy call](https://allstacklabs.com/contact/) ## Get answers you can stand behind. Send us a few questions your AI gets wrong. We'll show you where it breaks. [Book a strategy call](https://allstacklabs.com/contact/) [hello@allstacklabs.com](mailto:hello@allstacklabs.com) --- All Stack Labs LLC. hello@allstacklabs.com. (508) 505-4961. Book a strategy call: https://allstacklabs.com/contact/