To validate retrieval augmented generation and citations for ABA work, test three levels: the source corpus, the retrieval result, and each output claim. Confirm authority, product, jurisdiction, effective date, and version; check whether needed passages are retrieved and irrelevant ones excluded; verify that every material claim follows from its cited passage; and require abstention when sources are missing, stale, conflicting, or out of scope.

Treat RAG as a chain of evidence

Dario separates source acquisition, parsing, chunking, indexing, retrieval, ranking, generation, citation display, human review, and final action. A correct final sentence can hide a failed retrieval, while a real citation can fail to support the sentence attached to it. Validation records each stage so the practice can locate the defect and avoid crediting the whole pipeline for a lucky answer.

Govern the source corpus

For every document, record issuer, title, URL or repository, product and jurisdiction, publication and effective dates, revision, supersession, status, access terms, checksum, parser result, and owner. Separate statutes and binding plan documents from manuals, forms, portals, FAQs, call notes, and internal interpretations. Preserve withdrawn or prior versions when open claims, appeals, or historical decisions still depend on them.

Test retrieval independently

NIST TEVV emphasizes evaluation in context and meaningful data. Create questions with known relevant passages and difficult distractors. Measure whether at least one necessary passage appears, whether the retrieved set contains irrelevant or conflicting material, and whether ranking puts the strongest source where the generator can use it. Include tables, footnotes, scanned PDFs, repeated headings, cross-references, and documents with similar titles. A top-k result is a system setting, not proof of adequate retrieval.

Verify support one claim at a time

NIST's Generative AI Profile warns that confabulations can include false logic or citations. Dario breaks each answer into material claims and marks each as supported, contradicted, partly supported, unsupported, or outside scope. The reviewer opens the cited passage, checks its context and version, and confirms that the output did not combine two sources into a broader rule neither source states.

Handle missing and conflicting sources

The system should say what is missing and route the question to the responsible owner. When two current sources conflict, preserve both, pause automated enforcement, and obtain written clarification or qualified legal, payer, clinical, or coding review as applicable. Retrieval confidence cannot decide authority. A source that applies to Medicare, one state, one payer product, or one service route cannot silently become a universal ABA rule.

Protect retrieval from adversarial content

A document can be authoritative for facts and still contain text that must never control the model or tools. Apply the prompt-injection controls described in NIST AI 100-2e2025, restrict corpus write access, verify provenance, scan unexpected changes, and keep retrieved instructions from altering permissions or destinations. Sensitive corpus access follows the user's role and purpose; the index is not a shortcut around record access.

Work through claim-level evidence

Dario locks 40 material claims produced from a fictional payer-source corpus. Thirty-four cite the correct current passage and stay within its scope: 34 of 40, or 85%. Two cite superseded manuals, one citation supports only half the claim, one merges two products, one fabricates a page reference, and one gives an answer when no governing source is present. All six fail even though five point to real documents.

Use separate RAG measures

Corpus freshness equals current in-scope documents with verified status divided by documents due for review. Retrieval coverage equals questions with all required passages retrieved divided by locked questions. Claim support equals supported material claims divided by material claims reviewed. Citation correctness equals citations that identify the supporting passage divided by citations checked. Report abstentions, contradictions, parser failures, and unresolved source conflicts separately.

Revalidate every material change

Changes to documents, parser, chunking, embeddings, index, ranking, filters, prompt, model, citation renderer, access rules, or user interface can change results. Run the locked regression set before release and sample production outputs afterward. Keep prior versions and rollback instructions. If the source state cannot be reproduced, the practice cannot explain why an answer was generated.

Build a claim-review queue around consequence

Dario routes claims that could affect clinical care, safety, privacy, authorization, billing, employment, or legal rights to the qualified owner before action. The queue shows the exact claim, cited passage, source status and date, retrieved context, competing sources, model version, proposed use, and deadline. Reviewers can mark supported, narrowed, contradicted, unresolved, or outside scope and explain the next evidence step. An unanswered high-impact claim produces a hold, not a silent omission.

Lower-impact outputs can use risk-based sampling only after the practice has validated the population and established stop triggers. Samples include abstentions, user corrections, rare source formats, old-versus-new document conflicts, and outputs that users acted on. A rising correction rate, stale-source hit, or unsupported claim expands the review cohort and may pause the affected route.

Diagnose the stage that actually failed

When a citation is wrong, first ask whether the governing document was present and current. Then check parsing, chunk boundaries, metadata, access filtering, retrieval, ranking, answer construction, claim splitting, citation attachment, rendering, and the final human decision. Repairing the prompt will not fix an unreadable table, a missing plan version, or an index that mixed two payer products.

Preserve a small failure packet for each material defect: question, expected sources, retrieved passages, answer, claim labels, displayed citation, reviewer finding, root cause, correction, retest, and affected production cohort. That packet lets engineers fix the system while payer, clinical, privacy, or legal owners separately address any downstream decision. Close the defect only when both the technical path and affected work are reconciled.

Use a source-support matrix for release

Create rows for the high-impact question types the workflow is expected to answer and columns for authoritative source present, correct version, parser success, required passage retrieved, conflict surfaced, material claims supported, citation rendered, qualified review completed, and downstream action reconciled. A row fails when any required link fails. This makes it clear whether the practice has an answer-quality problem, a corpus-governance problem, or both.

Dario includes explicit no-answer cases in the matrix. The desired result may be an abstention that names the missing authority and routes the question to a payer, clinician, privacy officer, or counsel. He also tests sources that contain exceptions, cross-references, tables, footnotes, and effective-date transitions. These cases reveal whether a fluent summary is omitting the part that controls the real decision.

At release, record the supported scope in plain language: which issuers, products, jurisdictions, document types, date range, questions, users, and actions were evaluated. List known gaps and prohibited extrapolations. A retrieval system validated for one payer's current manuals cannot inherit authority over another product, an earlier service date, or a legal conclusion simply because the interface looks the same.

Related resources

Sources