Why a single score misleads
A retrieval-augmented system can fail because relevant passages were not retrieved, because retrieved passages were ignored, or because the generated text goes beyond the evidence. A single accuracy number conflates all three.
Component metrics
Retrieval can be evaluated with recall and ranking metrics against labelled relevant passages. Grounding can be evaluated by checking whether each claim in an answer is supported by retrieved text. Answer quality can be rated for completeness and clarity.
Each metric needs a labelled set of questions drawn from real usage, refreshed as content and behaviour change.
Using model graders carefully
Language-model graders scale evaluation but have biases. They should be calibrated against human judgements on a sample and re-checked periodically.