RAG gives you the machinery for grounding. It does not give you grounding.

I gave a plenary and a concurrent talk on this at NAACCR 2026 last week. This is the written version of one lesson from those sessions, for anyone who could not make it.

It cited the right table and still got the code wrong

I piloted a retrieval-augmented assistant to help with cancer registry coding. The shape is the standard one: someone asks a question, the system searches the indexed guidelines and coding tables, and answers with citations back to the source. And it worked. On a 20-question test set, 10 site and 10 histology, it got 18 right. So this is not a story about RAG falling over. It mostly does the job, and the numbers say so. It is a story about the two it missed, because the way it missed one of them was not what I expected.

The question was to code Acute leukemia, NOS in ICD-O-3. This was a histology lookup, the kind of thing I would have bet on it getting right. The correct code is 9801/3, Acute leukemia, NOS. The assistant returned 9800/3, Leukemia, NOS, the more generic neighbor. And it cited its source: the WHO ICD-O-3 site and histology lookup, the public table that lists both codes. So the answer arrived with a real, relevant citation attached, and that citation pointed at the document that contains the right answer. It was still wrong.

Here is the part that bugged me. The document was retrieved and cited; you can see it in the output, sitting right next to the code. The correct entry, 9801/3, is in that document. The model returned 9800/3 anyway, which is also in the table, just the wrong line for this term. When a stricter instruction forced it to surface the exact entry first, it returned 9801/3. I want to be careful about what I can and cannot claim: I cannot see why it landed on the wrong row, whether it read past 9801/3 or never weighed it at all. What I can say is that the source was right there, correctly cited, and the answer still did not match the entry the question called for. The citation was real. The grounding was not.

Below is that situation, reconstructed. The question is fixed. The cited source is fixed, and it contains the correct value. The only thing you change is the instruction the assistant runs under. Notice what does not change: the citation. Notice what does: whether the answer matches the document it points to.

The model points at the right table either way. What changes is whether the answer matches the entry the question calls for, or a different line from the same table. A citation cannot tell you which one you got.

One more thing, because one bug is not a pattern. I ran the same question a few more times (even after I "fixed" it with a stricter prompt..), and it missed the same way each time, citation attached each time, so it is repeatable. What I found interesting was asking the assistant why it got it wrong. BTW. If you haven't asked AI why it gets something wrong, you totally should. The first time, it explained with total confidence that it had anchored on the first matching row and not read far enough to catch the "acute" qualifier. I asked again and got a different confident explanation: it had grabbed an adjacent leukemia row by mistake. A third time, a third story.. every time, a new tidy explanation, and they cannot all be what happened. A model has no privileged view of why it produced a given answer, so each of those accounts is itself a fluent, ungrounded generation, the same failure one level up. The practical point: you cannot debug this from the model's story about itself, only from what it does. That's probably obvious but worth saying.

This has a name, and the research helps clarify why

What you just saw is a knowledge conflict: the source points one way, the model's parametric memory, the statistical residue of everything it absorbed in pretraining, points another, and the model has to pick. Or at least, when I dug into the literature, this seemed to be an interesting theory to explore. It helps to keep two things separate here. Retrieval gets information into the context. Whether the generated answer is controlled by that context, rather than by what the model already believed, is a different question, and it's possible that that's the one that bit us. The survey of this area (Xu et al., 2024) sorts these conflicts into three kinds: context-memory conflict, which is ours, plus inter-context conflict where two retrieved passages disagree, and intra-memory conflict where the model contradicts itself across rephrasings of the same question.

Researchers land on similar observations: retrieved context and the model's prior compete as the answer is generated, and which one wins is not guaranteed. The early work leaned pessimistic. Longpre et al. (2021) showed that when a retrieved document contradicts what a model already believes, the model often clings to its own memory and overlooks the document, and argues that we should measure a model's tendency to hallucinate rather than read.

The more useful finding is that it depends, and what it depends on is something you can act on. Xie et al. (2023), in a paper with the excellent title Adaptive Chameleon or Stubborn Sloth, found two behaviors at once. Give a model coherent, convincing external evidence as the only thing in front of it, and it will often follow that evidence even against its prior. But let the context contain anything that agrees with what the model already believes, and a strong confirmation bias kicks in: it leans back toward memory.

Maybe that fits our bug... or at least it's one reading of it. 9800/3, Leukemia NOS, is a high-frequency, default-feeling code the model has seen constantly. If it leaned on that prior, this is exactly where it would land, even with the right table cited beside it. The faithfulness research runs straight through this: work like Zhou et al. (2023) and Shi et al. (2023) is about whether an output is determined by the provided context or by the model's priors. A correct citation next to a wrong value is the textbook unfaithful but attributed answer, and it is the version most likely to slip past a reviewer.

Why this is dangerous: the citation buys trust

Fluent and wrong, we have all learned to be a little wary of. Cited and wrong is worse, because the citation is the signal a careful person looks for to decide an answer can be trusted. A reviewer who checks "is there a source, and is it the right document" finds both, and moves on. The citation did persuasive work the answer had not earned. An experienced oncology data specialist here would have caught the 9800 against 9801; a newer one, or a downstream analyst trusting a clean and cited response, may not. In regulated work, where a wrong code propagates into counts that feed real decisions, a confident answer wearing a correct citation is the failure most likely to get waved through.

This is not a cancer-registry quirk, either. The same failure shows up anywhere the right answer is a small value buried in a larger retrieved document: a statute number, an API parameter, a medication dose, a tax rule, a product SKU. The model cites the right page and still reports the wrong figure from it, and the citation makes nobody look twice.

What you can do about it

The cheapest lever is the prompt, and it is more powerful than its reputation. My change was one line. I went from "consult the source where possible" to "always show the exact source entry before answering, and never answer from memory when the source covers the question." Same model, same corpus, and on this question, the right answer. I want to be honest about the scope of that, though: it fixed the leukemia case only some of the time.. Tweaking the prompt is not a trick I invented. Zhou et al. (2023) showed that targeted prompting, reframing the context as an external statement the model must attend to, and letting it abstain, measurably improves how faithfully models stick to provided context, with no retraining. Those gains seem to be sensitive to exact wording, so you test the new prompt against cases, you do not assume it worked.

And forcing the model to always defer to the source cuts both ways. The same Xie et al. paper that explains our bug shows the opposite failure too: give a model coherent, convincing evidence and it will often follow it, even when the evidence is wrong. So a hard "always trust the retrieved passage" instruction trades a context-memory problem for an inter-context one. If retrieval surfaces a stale or contradictory passage, you have now told the model to repeat it. Grounding harder is not the same as grounding better, and which way you want the model to lean depends on whether you trust your corpus more than the model's prior. For a curated coding table, you probably do. For a messy document pile, maybe not.

The prompt is the lever I happened to pull first, but it is not the one I would check first. Here is a fuller menu, roughly in the order I would actually work it:

• Start upstream, with preprocessing. How a document gets chunked, embedded, and formatted decides what retrieval can even return cleanly, and it is a very common cause of exactly this kind of one-row-off error. Tables are a weak spot for RAG: a lookup table only means something when the row and column stay aligned, and most chunkers flatten it into a run of text, so adjacent entries blur and the tie between a term and its code can loosen. An Excel source adds its own failure points in how the cells get serialized and where a chunk boundary lands. I am on Microsoft Copilot, so a lot of this is handled for me, but handled is not the same as verified. This is what I suspect in my own bug, and it is where I would start.
• Then retrieval itself. Make sure the right passage is actually coming back, and coming back near the top. Most RAG systems only pass the model the top few results, so if the relevant chunk ranks low, the model never sees it. In my case the document was retrieved and cited, so this was not the bottleneck, but for a lot of systems it is the first thing that is broken.
• Lean on the model at decoding time. Methods like context-aware decoding (Shi et al., 2023) push the model to weight the retrieved context more heavily as it generates.

In the spirit of working in the open: I have not yet worked these upstream pieces for my own setup, the chunking strategy and how that Excel table is represented in particular. That is what I am digging into over the next few weeks, and if I find something worth sharing I will update this page or write a follow-up.

What strikes me looking back is that the guardrails we reached for by trial and error, mandatory citations, confidence tags, a conservative default, and an instruction to say so when the source does not cover the question, are the same inference-time levers this literature recommends. We added them for their own sake, and it turned out they were the thing doing the grounding. With one caveat the leukemia bug makes obvious: a mandatory citation makes a system feel safer to audit, but only if you audit the right thing, the value, and not the fact that a citation showed up.

Do not test for the presence of a citation. Test whether the cited value is the one in the source. A system that always cites looks trustworthy and can still be ungrounded on the part that matters: the answer itself.

That is the evaluation point that matters most here, and the one most demos skip. Do not only check whether the final answer was correct, and do not let a citation reassure you. Concretely: build a small gold set where each question has both the known correct code and the known source span that contains it, then score two things separately, whether the answer is right, and whether the cited span actually says what the answer claims. The presence of a source is not evidence of grounding; agreement between the answer and the source is. Scoring them apart is what surfaces the cited-but-wrong cases, because an accuracy-only metric will happily pass a lucky guess that is right without being grounded.

The takeaway

RAG gives you the machinery for grounding. It does not give you grounding. A retrieved passage, a vector store, even a correct citation, are the apparatus. Grounding is the property that the answer's content is determined by the source, and that is a behavior you specify, test, and monitor, not something you get for free by wiring up retrieval. Same lesson as the threshold in the last piece: the model gives you capability, and the judgment about how to use it stays with you.

Two threads I will pick up from here. First, how you actually know a system like this is any good without fooling yourself, which is its own discipline. Second, the question sitting underneath all of it: whether reaching for a generative model is even the right call for a given job, when a smaller and more bounded approach is often the better one. More on both soon.

References and further reading

• Xu, R., Qi, Z., Guo, Z., Wang, C., Wang, H., Zhang, Y., & Xu, W. (2024). Knowledge Conflicts for LLMs: A Survey. EMNLP 2024. arXiv:2403.08319
• Longpre, S., Perisetla, K., Chen, A., Ramesh, N., DuBois, C., & Singh, S. (2021). Entity-Based Knowledge Conflicts in Question Answering. EMNLP 2021. arXiv:2109.05052
• Xie, J., Zhang, K., Chen, J., Lou, R., & Su, Y. (2023). Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflicts. ICLR 2024. arXiv:2305.13300
• Zhou, W., Zhang, S., Poon, H., & Chen, M. (2023). Context-faithful Prompting for Large Language Models. Findings of EMNLP 2023. arXiv:2303.11315
• Shi, W., Han, X., Lewis, M., Tsvetkov, Y., Zettlemoyer, L., & Yih, S. (2023). Trusting Your Evidence: Hallucinate Less with Context-aware Decoding. NAACL 2024. arXiv:2305.14739