Algorithmic bias

Bias can enter before a model runs

An organization first chooses the problem and desired outcome. A scheduling model built to maximize filled hours may disadvantage people who need interpreters, transportation coordination, evening visits, or longer intake conversations. The optimization target is part of the system.

Bias can also enter through historical records, missing data, measurement choices, proxy variables, labels, sample exclusions, feature design, thresholds, interface placement, staff behavior, and feedback loops. NIST SP 1270 treats bias as a broader socio-technical concern rather than a model-only defect.

Define the use and consequence

Record what the system predicts or recommends, who receives the output, and which decision may follow. Risk differs sharply between a tool that groups help-desk tickets and one that influences intake priority, staff evaluation, treatment content, or payment work.

Identify the people affected even when they never see the tool. Document excluded populations, unsupported languages, missing modalities, uncertain labels, and conditions where the system should abstain or route to manual review.

Measure errors people experience

Overall accuracy can conceal unequal errors. Select measures that match the consequence: false positive and false negative rates, calibration, abstention, override, delayed access, successful correction, or another use-specific result. Report counts with percentages and define the evaluation cohort, period, exclusions, ground truth, and decision threshold.

Comparisons need adequate sample sizes and responsible interpretation. Small subgroup counts can be unstable. Group averages can also miss disability, language, age, geography, payer, and technology interactions. Privacy and legal review should shape which attributes are collected and how they are used.

A fictional routing audit

Iris reviews 60 mature referrals from one month: 30 used the standard portal and 30 requested language assistance. The model routes 24 standard-portal referrals and 15 language-assistance referrals to the correct next queue by the target: 24 of 30, or 80%, versus 15 of 30, or 50%.

That difference triggers investigation. It does not prove unlawful discrimination or identify the cause. The groups may differ in missing fields, services, geography, or case complexity. Manual review finds that eight language-assistance records lacked a structured channel field after a form change. Five of those eight were routed late.

The practice fixes the form, restores the missed referrals, and repeats the predeclared evaluation on a later cohort. It keeps all 60 records, error types, corrections, time to resolution, and family-reported access problems visible.

Human review needs real authority

A reviewer should see relevant source evidence, understand the output’s role and limitations, change or reject it, and record the final decision. Excessive queues, automation pressure, vague explanations, or punitive override metrics can turn human review into a formality.

Provide a usable correction and appeal route. Tell affected people when an automated output materially shapes a decision when required or appropriate, explain the next step in accessible language, and protect communication support. Urgent safety and mandated processes need direct routes outside routine automation.

Test change and deployment together

NIST AI RMF is a voluntary framework that addresses validity, reliability, safety, security, transparency, privacy, and harmful bias across the lifecycle. NIST currently says version 1.0 is being revised.

Validate the deployed interface, data pipeline, threshold, users, and follow-up process. Reassess after model, vendor, prompt, population, policy, form, or workflow changes. Monitor both intended benefit and unintended burden. Define a stop condition and a safe manual path before release.

The CASP AI page describes ABA-sector parameters for oversight, selection, deployment, monitoring, and auditing. It supplies guidance rather than proof that a particular model or workflow is fair.

Ask what a fairness result leaves unanswered

Before accepting a fairness claim, ask which population, time period, task, outcome, group definitions, threshold, and comparison produced it. Confirm who chose the ground truth and whether the label itself reflects unequal access or prior decisions. A model can match one statistical measure while failing another because fairness definitions can conflict.

Examine intersectional and setting-specific effects when the sample supports responsible analysis. Retain raw counts beside rates, confidence or uncertainty information where appropriate, and records of people outside the evaluation. Document why a measure fits the consequence of this use.

Include qualitative evidence from affected people. A route may appear timely while its form blocks AAC, requires unsupported English text, or makes correction difficult. Build accessible feedback into the workflow and examine whether complaints receive a response.

Bias review continues after release. Track changes in population, missingness, thresholds, human overrides, vendor versions, and downstream outcomes. Pause the automated path when critical subgroup evidence becomes stale, an error crosses the approved threshold, or an affected person lacks a workable correction route.