What Happens When ABA Staff Disagree About Data?

Disagreement can reveal several problems

Observers may use different definitions, see different parts of an event, start timers at different moments, treat prompts differently, or work in different settings. One record may also contain an entry error. The person receiving care may genuinely respond differently with each partner.

Start by identifying the date, target, observers, settings, opportunities, versions, and exact values. Avoid assuming that one worker is careless or that averaging the scores solves the problem.

Weak definitions can cause observer disagreement

Ask both observers to score the same examples independently when appropriate. Compare agreements and disagreements at the event or interval level. Review examples, nonexamples, start and stop rules, prompt coding, supports, and missing-data instructions. Retrain and recheck after clarification.

The BCBA Test Content Outline covers operational definitions, measurement systems, interobserver agreement, measurement validity, and data interpretation as examination content.

Preserve correction history

The RBT Ethics Code requires accurate documentation. The BACB Ethics Code addresses accuracy, documentation, data use, correction, supervision, and evaluation for covered behavior analysts.

If an entry is corrected, retain the original author, value, date, reason, corrected value, and reviewer under the applicable record policy. If neither observation can be resolved, mark the uncertainty.

Clinical decisions may need to wait

A disagreement on one low-risk observation may call for clarification at the next review. A pattern affecting safety, health, a restrictive procedure, a mastery decision, a discharge decision, or a major treatment change may require an immediate hold and new observation. The responsible clinician should explain the effect on the plan.

Client and family reports remain separate sources and can help identify setting differences.

A practical example

Two observers independently score 20 intervals. They agree on 15 and disagree on 5, so interval agreement is 15 of 20, or 75%. Review shows the prompt rule was ambiguous. The clinician clarifies the definition, retrains both staff, and collects a new agreement sample before using the measure for a mastery decision.

Agreement is evidence about the measure

Interobserver agreement can show how consistently two observers applied a definition during the sampled events. It does not prove that the definition measures the right outcome, that implementation was appropriate, or that every unsampled record is accurate.

The calculation should match the measure. Interval agreement, exact-count agreement, total-count agreement, trial-by-trial agreement, and duration agreement answer different questions. Useful questions include which method was used, the numerator and denominator, and why it fits the data.

Review the disagreement event by event

Start with the exact intervals, trials, or episodes that differ. Ask whether both observers had the same view, start and stop cues, device access, response window, prompt rule, and current program version. One person may have missed part of the event. The response itself may have been ambiguous.

Avoid resolving the issue by automatically choosing the supervisor's score or averaging two categorical judgments. Use contemporaneous evidence when available. If the event cannot be resolved, preserve the uncertainty and collect another observation.

Distinguish measurement disagreement from setting differences

Two staff may collect at different times and see genuinely different performance. That is not interobserver disagreement because they did not observe the same event. Compare the setting, partner, opportunity, health, communication access, and supports before attributing the difference to scoring.

These patterns can guide a useful clinical question. They should not become a personnel conclusion without appropriate evidence and authority.

Build a correction and retraining path

When the definition is clear and one entry conflicts with source evidence, use the authorized correction process. When the definition is ambiguous, change the controlled version, mark the phase, retrain everyone using it, and validate new scoring. Search for other records created under the same faulty rule.

Track the due cohort, reviewed records, affected records, corrected records, and unresolved records. A fix is incomplete if the team corrects only the example that exposed the problem.

A second example with unresolved evidence

Two observers score the duration of a transition. One starts the timer when the instruction is delivered; the other starts when the person stands. Their totals differ by three minutes. No video or synchronized record can establish the intended start for that session.

The team marks the value uncertain, clarifies the start event, and collects a new agreement sample. It avoids selecting the shorter duration because it supports a favorable trend. The clinician decides whether the remaining evidence is enough for the pending plan review.

Communicate without blaming staff or the client

A family-facing explanation can say that observers applied the response rule differently, the affected dates are under review, and the practice is clarifying the definition and validating new data. It does not need to disclose discipline or speculate about intent.

The person receiving care should not be described as “inconsistent” solely because observers disagree. Keep measurement uncertainty, implementation, and actual context as separate possibilities.

Know when to escalate quickly

Immediate review is appropriate when disagreement affects injury, medical care, mandated reporting, restrictive procedures, consent, or another safety-critical decision. Follow the relevant emergency, clinical, legal, and reporting routes while preserving the record. Routine agreement calculations should not delay necessary protective action.

Close with a defensible decision

The qualified clinician should state which observations remain usable, which are corrected or excluded, whether a graph or report changed, and what evidence is required before the next decision. Ask for the definition version, validation result, and follow-up date.

Prevent disagreement from becoming hidden consensus

Staff may stop recording uncertainty when they feel pressure to match a supervisor or favorable target. A sound process invites questions, preserves original observations, and treats disagreement as information about the measurement system.

Supervision can review examples without publicly shaming a worker. The practice should protect honest reporting while addressing competence or accuracy through the appropriate process. Families need the care-record outcome, not confidential employment details.

Recheck after the repair

Clarifying the definition and retraining staff are intermediate steps. Obtain a new agreement sample under the relevant setting and target. Report the eligible events, method, result, unresolved differences, and whether the sample meets the practice's decision rule.

If agreement remains weak, reconsider the measure itself. A difficult-to-observe response may need a clearer unit, different observation method, or another outcome that better serves the person's priorities.

Questions families can use

Ask what the observers saw, whether they used the same version, how agreement was checked, which definition changed, whether records were corrected, what uncertainty remains, who decided whether data can be used, and whether any clinical decision is on hold.