Retrieval can provide useful context, but a retrieved passage does not guarantee a correct answer. Source quality, access boundaries, retrieval behavior and the way an answer uses evidence all need evaluation. There is no universal chunk size or similarity threshold that makes every business assistant factual.
Find where the answer failed
Separate retrieval from generation when investigating a problem. Did the relevant approved source exist? Was it retrieved for the question? Did the response accurately reflect its conditions and exceptions? Each failure suggests different work. Rewriting a prompt will not create information absent from the knowledge collection.
Microsoft's RAG overview describes grounding with proprietary content and the challenges involved. For a particular implementation, inspect the evidence supplied to the answer stage rather than treating a fluent response as proof of successful retrieval.
Choose document boundaries using tasks
A small fragment may lose important conditions. A large fragment may mix unrelated instructions. Prepare a sample collection with useful headings, source identifiers and revision information. Compare how different boundaries affect representative questions rather than selecting a fixed token count as a universal rule.
Include tables, short policies and multi-step procedures. Ask a document owner which parts must remain together to preserve meaning. The right preparation depends on both the source structure and the job the assistant supports.
Evaluate thresholds in context
A score depends on the retrieval approach, content and configuration. A threshold copied from another demonstration may reject useful evidence or admit irrelevant passages. Use a labelled test set and examine the practical trade-off between answering, asking for clarification and abstaining.
Change one important factor at a time where possible and record the configuration. Review unfamiliar examples as well as those used for tuning. A change that improves a small set of rehearsed questions may not improve the broader experience.
Require evidence for meaningful statements
Design an answer that exposes useful source references and important conditions. Ask reviewers whether the cited material actually supports the statement. A link to an entire manual is less helpful if nobody can locate the supporting procedure. Where sources disagree or have expired, show that limitation and an appropriate escalation path.
Make abstention useful
Agree when the assistant should say it cannot support an answer. Explain whether the problem is missing information, an ambiguous question, insufficient access or a decision outside scope. Give the user a practical next action rather than an invented answer.
Include these cases in evaluation. A good no-answer response is a successful outcome when the available evidence cannot support the requested conclusion. Evaluate that behavior explicitly instead of rewarding every completed paragraph.
Keep access and tool permissions separate
A document can also contain instructions attempting to redirect the assistant. OWASP identifies prompt injection as a risk; retrieval does not eliminate it. Source text should not gain authority to change permissions or trigger unrelated actions.
For a pilot, narrow the audience, sources and permitted operations. Test restricted documents, stale sources and hostile instructions. Expand only when the evaluation and operating process support a particular next step.
Read the complete production RAG guide ↗ · Explore RAG services ↗

