What Happens When ABA Observers Use Different Definitions?

Resolve the definition mismatch

Create a definition history with wording, examples, nonexamples, response boundaries, start date, end date, affected records, trainer, and decision owner. Mark the graph or report at the version change. Explain whether prior data remain usable, require separate display, can be rescored from preserved evidence, or must stay uncertain.

The first task is containment. Identify which observers, clients, settings, measures, and dates used each definition. Pause decisions that depend on combining the affected records until a qualified reviewer understands the scope. Routine care and immediate safety actions can continue through the applicable plan while the measurement issue is reviewed.

Find the exact difference

Definitions can diverge in several ways:

• one includes a broader response form
• observers use different start or stop points
• one requires an opportunity that the other does not
• response windows or interval lengths differ
• one observer counts prompted responses as independent
• a plan update reached some people but not others
• examples in training conflict with the written definition

Write the competing wording side by side and apply it to the same fictional examples. This often shows whether the problem is a distribution failure, ambiguous language, or a deeper disagreement about what should be measured.

Check what the result can support

Agreement cannot repair a definition mismatch after the fact. Two people applying different rules may disagree for a predictable reason. The repair begins with one approved operational definition, calibration on the same examples, and a new independent observation rather than pressuring scores to match.

Decide what happens to prior records

There are several defensible outcomes. Records collected under each definition may remain valid for separate questions. They may be displayed as separate phases. A preserved recording or detailed source record may support authorized rescoring. Some periods may remain uncertain because the original event cannot be reconstructed.

Rescoring requires the original evidence to contain the detail needed for the approved definition. A narrative total cannot usually be converted into interval-level data after the fact. Preserve the original record, add the rescored result with author, date, reason, and method, and explain which one governs the later analysis. Silent replacement hides the measurement problem.

Issue and teach the approved version

The qualified owner should approve the definition, examples, nonexamples, boundaries, and effective date. Operations can distribute it, retire the earlier material, and track affected observers. Calibration should use shared practice material followed by an independent check. The team should also decide when drift will be reviewed.

A clinical definition change can affect the graph, mastery criterion, treatment plan, family report, or payer document. Each affected artifact needs an owner. The observer-agreement repair alone does not authorize a clinical, billing, or coverage decision.

Use current clinical and measurement sources

The CASP public summary places assessment, planning, implementation, and evaluation within its autism-treatment scope. The BACB Ethics Code addresses competence, client involvement, consent and assent when applicable, documentation, and data-based evaluation for covered behavior analysts.

The BCBA Test Content Outline includes measurement, data integrity, observer agreement, and visual analysis as examination content. It does not set one calculation, sampling percentage, or action threshold for every case.

Choose the calculation for the measure

Vollmer and colleagues discuss practical consequences of data reliability and treatment-integrity monitoring. Reed and Azulay describe several agreement calculations. Method choice depends on the measurement system and the question being asked.

Keep communication and conditions visible

The ASHA AAC portal supports continuous communication-tool access. Observer agreement is meaningful only when the record also identifies communication access, ordinary supports, opportunity availability, and other conditions that affect what could be observed.

Definitions should reflect the person's actual communication. If one observer counts speech and another includes AAC, the repair should begin with the client's reliable forms and the clinical question. A measurement system should not make a valid message disappear because a staff document omitted it.

A practical example

One observer counts every pause longer than five seconds. Another counts only pauses that begin within ten seconds after an instruction. Six sessions contain a mix of the two records. The supervisor separates those sessions from the earlier graph and traces the mismatch to two training examples that conflicted with the written definition.

The clinician reviews the goal with Aria and approves a definition tied to a specific opportunity and start point. The team cannot reconstruct the six sessions because the original notes lack timestamps, so those records remain labeled uncertain. They are not converted or discarded.

Both observers calibrate on twelve fictional examples and then independently score ten new matched opportunities. They agree on nine. The graph begins a new phase at the approved version date, and the family report explains the definition history, uncertain period, and new sample. The repair preserves the evidence instead of creating a smooth but misleading trend.

Questions for the correction meeting

Ask when each definition was used, how the mismatch was discovered, which decisions and reports relied on the data, whether source evidence supports rescoring, who approved the new version, how observers demonstrated readiness, and how the graph marks the change. End with an owner and due date for every affected artifact.

Review decisions already made from the affected data

The definition mismatch may have influenced mastery, goal revision, staffing, supervision, discharge, safety planning, payer submissions, or family reports. List each downstream decision and classify whether it remains supportable, needs a qualified review, or should be held. Correcting the graph alone does not resolve every downstream use.

Notify the people responsible for affected decisions using the practice's approved process. A clinician reviews clinical conclusions. A qualified billing or payer owner reviews submitted records or claims. Privacy and legal roles review any disclosure or reporting concern within their scope. Keep the client and family informed in understandable language, including what is known, uncertain, corrected, and still under review.

After remediation, test the full chain: approved definition, current training material, observer access, data form, graph, report, and decision rule. A copied old label in one tool can restart the mismatch.

Retain the correction log with the affected dates, records, owners, approvals, and completed validation. Families can ask for a plain-language summary showing how the practice knows every active tool now uses the same definition.

Include the next drift-check date.

Questions families can use

Ask which definitions differed, when each was active, which records are affected, whether original evidence permits rescoring, how observers were recalibrated, and how the change appears in progress reports.