Automation bias

Plausible output can weaken scrutiny

Automated suggestions often arrive inside the workflow, use polished language, and appear to summarize more information than one person could review quickly. That convenience can encourage acceptance before the reviewer checks the source.

AHRQ's Human-AI Interaction brief describes automation bias as overreliance on automation, especially under time pressure and high workload. It also discusses complacency, confirmation bias, functional fixedness, and possible deskilling.

Commission and omission errors differ

A commission error occurs when a person follows an incorrect automated suggestion, such as accepting a fabricated observation in a draft note. An omission error occurs when a person misses something because the tool stayed silent, such as failing to investigate a risk that no alert identified.

Both can happen in scheduling, authorization checks, claim edits, documentation, assessment summaries, treatment planning, or safety review. A high overall acceptance rate cannot show whether the accepted suggestions were correct.

Clinical decision support should support a decision

ASTP/ONC describes clinical decision support as person-specific information provided at useful times to improve care and says it should be clear, organized, and fit the workflow. The qualified clinician remains responsible for clinical judgment within scope.

Software may surface evidence, calculate, draft, or flag. It should not hide uncertainty, invent clinical facts, or make its preferred action difficult to reject. A payer, vendor, or model cannot assign clinical authorship to itself.

Design for an independent check

Before revealing a recommendation in high-stakes work, ask the reviewer to inspect key evidence or record an initial judgment when feasible. Then present the automated result with its source, version, data time, intended use, confidence or uncertainty where meaningful, and known limits.

The reviewer needs to:

• access the underlying record and competing evidence
• recognize which content or suggestion came from automation
• change, reject, or defer the output without penalty
• document the final rationale and author
• report an error and see whether it was resolved

Avoid default selections that release clinical content or claims without active review. Add useful friction at the consequence, not random clicks that people learn to dismiss.

A fictional decision-support review

Talia's practice reviews 24 automated suggestions in a locked fictional cohort. Reviewers independently agree with 15 after checking source evidence, override six, and defer three for missing information. These states total 24 of 24.

Four of the 15 agreements later require correction after a second-source check. Report agreement as 15 of 24, or 62.5%, overrides as 6 of 24, or 25%, deferrals as 3 of 24, or 12.5%, and corrected agreements as 4 of 15, or 26.7%. None of these rates alone measures accuracy, benefit, or causation.

Watch the surrounding work system

Automation bias grows from more than individual carelessness. Staffing, workload, alert volume, production targets, interface design, training, missing source data, and consequences for disagreement all affect review.

AHRQ's electronic-health-record primer discusses risks from inaccurate rules, alert fatigue, automation bias, and copied content. Review the full system rather than telling clinicians to “pay more attention.”

Make the manual path practical. Protect time for review. Sample decisions across users, shifts, sites, suggestion types, and consequences. Include cases where the system made no suggestion.

Practice disagreement before launch

Training should include correct suggestions, incorrect suggestions, missing alerts, incomplete source data, and ambiguous cases. Ask reviewers to explain which evidence they used and when they would defer or escalate. Observe whether the interface makes disagreement practical.

Use drills that resemble real workload without involving live care decisions or exposing client data. Rotate which answer is correct so staff do not learn that the exercise always rewards rejecting automation. Review disagreement respectfully and separate thoughtful override from careless clicking. If reviewers repeatedly follow the same error, examine the display, default, timing, workload, source availability, and policy. A reminder alone rarely repairs a work system that rewards speed over verification.

Monitor disagreements and silent failures

Track suggestions reviewed by date divided by suggestions due, overrides by reason, deferrals, unsupported statements, corrected agreements, missed events, incidents, and aged feedback. Separate model errors, source-data problems, interface problems, and reviewer mistakes.

The voluntary NIST AI Risk Management Framework can organize governance, context mapping, measurement, and management. CASP's AI practice parameters address selection, deployment, monitoring, change management, and auditing in ABA. Neither source replaces clinical authority or direct workflow validation.

Pause or narrow use when the source is unavailable, performance changes, workload prevents meaningful review, errors cluster, a vendor changes the model, or the manual alternative fails. Reassess after every material data, interface, policy, population, or workflow change.