To test ABA AI for fairness across languages, disabilities, and user groups, define the affected decision, population, access route, benefit, and possible harm. Build representative strata with affected-person input, test performance and usability within each group, preserve small and missing cohorts, investigate causes behind gaps, and change data, interface, workflow, or availability rules before release. One pooled score cannot establish fair access or outcomes.

Define fairness for the actual workflow

Meera begins with who receives the output, who must use the interface, what decision follows, and what benefit or burden can change. Fairness may involve error rates, abstentions, wait time, correction burden, communication access, denial of an option, or who receives human help. She writes the claim narrowly. Equal pooled accuracy does not establish equal access, and identical treatment can fail when people need different effective supports.

Involve affected people before selecting measures

Ask clients, families, staff, interpreters, accessibility specialists, and augmentative and alternative communication users which failures matter and how they show up. Compensate participation appropriately and provide usable communication routes. Record whose perspective is absent. A proxy selected by the product team can miss humiliation, added labor, unusable timing, or a workflow that silently routes one group to more holds.

Build meaningful evaluation strata

Stratify by the variables that can affect the system: language and dialect, document quality, device, assistive technology, communication form, sensory or motor access, setting, user role, payer source, and intersection of relevant conditions. Set planned counts and minimum evidence rules before results. Do not publish unstable percentages for tiny groups without counts and uncertainty. Keep missing, excluded, and unsupported groups visible as evidence gaps instead of calling them equivalent.

Test access and performance separately

DOJ Title III guidance addresses equal opportunity, effective communication, reasonable modifications, and physical access for covered public accommodations, subject to the law's standards and defenses. ASHA's AAC portal says AAC users should always have access to their tools or devices. Test keyboard, screen reader, magnification, captions, language route, response timing, device compatibility, backup communication, error correction, and human assistance in addition to output quality.

Avoid adverse shortcuts

A disability, language, AAC use, interpreter need, or accessibility request cannot serve as a proxy for poor fit, lower priority, fraud risk, noncompliance, or low value. Keep clinical competence, legal authority, payer state, operational capacity, and accommodation work as separate decisions with proper owners. When the AI predicts or ranks people, inspect whether historic service gaps or staff behavior have become labels the system repeats.

Investigate causes behind gaps

NIST's managing-AI-bias work treats bias as broader than a purely computational problem and includes human and systemic factors. Examine data coverage, label quality, model behavior, prompt language, retrieval sources, interface design, staff interpretation, queue rules, and downstream policy. A numerical adjustment may hide a workflow barrier. Document the proposed cause, repair, retest, residual uncertainty, and affected scope.

Interpret disparities cautiously

Predeclare which comparisons can support a decision and what minimum evidence each requires. Report counts, rates, uncertainty, missingness, and the actual opportunity or exposure behind every denominator. A difference can reflect model behavior, measurement error, sample composition, unequal source quality, inaccessible testing, staff response, or ordinary variation. A small observed gap does not prove fairness, and a large one does not identify its cause. Route clinical, accessibility, privacy, employment, and legal interpretations to qualified owners. When evidence is thin, narrow the supported population, collect better data through an approved route, or keep the workflow unavailable for that use.

Work through a stratified test

Meera locks 72 fictional cases across six predefined language, communication, device, and user-role strata. Fifty-four have valid source evidence, accessible test execution, expected answers, reviewer calibration, and complete results: 54 of 72, or 75%. One stratum lacks enough AAC-device cases, two language groups have unresolved labels, and several screen-reader tasks fail before model output. The practice reports each gap and withholds a broad fairness claim.

Set release and monitoring rules

Require minimum evidence for every material stratum, no unresolved critical access failure, qualified review of disparities, and a repair plan for supported limitations. Monitor volume, performance, abstention, queue age, correction burden, complaints, accommodation completion, and human override by relevant group and version. Protect privacy and avoid reidentification when groups are small. A new population, language, interface, vendor model, or workflow can trigger revalidation. Keep the supported scope visible.

Use a fairness review checklist

  • Which people can gain, lose, wait, or do extra work?
  • Which group and intersection counts are adequate?
  • Can every participant use the evaluation route?
  • Are labels based on qualified evidence?
  • What operational rule follows the AI output?
  • Which gap blocks release or requires a narrower claim?

Respond to a subgroup gap with a cause-specific plan

A disparity starts an investigation rather than a cosmetic threshold adjustment. Check sample size, missingness, labeling, source quality, language support, assistive technology, interface access, workflow differences, reviewer behavior, model performance, and downstream decision rules. Lock the affected cases and compare them with a meaningful reference group while preserving the individual failures that the aggregate may hide.

The response may require better data, a qualified interpreter, an accessible interface, a narrower use, different human review, retraining, vendor correction, or a manual route. State who owns each action and how the practice will test benefit and unintended effects. If adequate evidence or access cannot be established, hold the affected use rather than accepting poorer service for that group.

Keep lived experience and performance evidence distinct

Quantitative accuracy cannot show whether people could understand, operate, correct, or safely decline the system. Invite affected clients, families, staff, and accessibility specialists to test representative tasks through their usual communication and access methods. Record barriers, workarounds, burden, privacy concerns, and preferred alternatives without converting every report into a single score.

Participation requires accessible materials, appropriate language support, privacy, clear purpose, voluntary feedback, and a route to raise concerns without retaliation. Explain how feedback influenced the release decision and which requests remain unresolved. A listening session is not a substitute for performance testing, and a performance test is not a substitute for direct experience.

Monitor fairness after release

Track eligibility, completion, abstention, error severity, reviewer correction, delay, appeal, accessibility failure, and downstream outcome by the approved strata when lawful and appropriate. Watch for changes in who uses the tool and who is routed away from it. Small groups may need case review and qualitative evidence rather than unstable percentages. Every monitoring plan should state privacy protections and limits on interpretation.

When a stratum is too small for a stable rate, preserve the raw count, uncertainty, case characteristics, and direct experience instead of merging people into a broader group that erases the relevant barrier. The release owner can collect more cases, narrow the claim, use a manual route, or keep the feature unavailable for that use. Document the choice and the evidence needed to reconsider it.

Related resources

Sources