Respond to Treatment-Integrity Observer Disagreement

To respond when observers disagree on treatment integrity, pause broad conclusions and inspect the scored component, eligible opportunity, visibility, timing window, recording, observer training, expectancy, reactivity, and calculation. Compare records component by component. Clarify the definition, recalibrate with shared examples, and resample. Observer disagreement is measurement evidence, not proof that one observer or implementer failed.

Localize the disagreement

Compare opportunity detection, component occurrence, sequence, timing, prompts, invalid states, and calculation. Use the total percentage as one summary of those underlying records.

Align the observers' records by timestamp or opportunity before discussing the score. Identify whether they disagreed about what was due, what happened, whether an adaptation applied, or how the denominator was constructed. Preserve independent original entries and mark unresolvable items. A point-by-point review often shows that one total difference contains several distinct definition or visibility problems.

Build a disagreement table with the opportunity, active version, each independent score, evidence available to each observer, resolution status, and effect on the summary. Separate agreement on opportunity occurrence from agreement on component scoring. This prevents a high overall agreement rate from hiding systematic disagreement on a rare but critical communication or safety step.

Check what each observer could see

Review position, audio, recording frame, technology, privacy limits, and competing tasks. Some disagreements are visibility failures.

Document the sensory and technical access available to each observer, including whether AAC output, subtle partner action, or an off-camera event could be detected. Respect consent, privacy, and recording limits while improving future observation design. Do not label an unseen component incorrect. When client protection is at issue, act on available safety information while keeping broader scoring qualified.

Ask the client and implementer whether relevant events occurred outside the observer's view while preserving those reports as separate sources. Improve positioning, audio, or live notation only within valid privacy and consent boundaries. When adequate visibility cannot be created, redesign the measure or limit the intended use instead of pretending the observation can support every component.

Recalibrate without answer coaching

Use new shared examples, require an independent score, and discuss the rule afterward. Avoid training observers to match a preferred conclusion.

Select examples that represent boundary cases, valid adaptations, missing opportunities, and critical components without reusing the disputed session as the answer key. Have each observer explain the rule and score independently. Record agreement by component and opportunity, not only a total. Persistent disagreement may require clearer definitions, different observation access, additional training, or clinical revision.

Repair before judging implementation

Revise the definition or observation system, document the change, and collect a new agreement sample. Keep disputed observations qualified.

Version the scoring rule prospectively and state when it takes effect. Do not rescore historical observations under a later definition unless the report clearly labels a secondary analysis and preserves the originals. Collect a representative follow-up sample with independent observers. Delay performance comparisons that depend on unresolved data, and route any employment or certification issue through its authorized process.

Apply the observer-disagreement review prospectively

Quinn's observers disagree because the live observer hears an AAC message that the recording does not capture, while the recording shows a timing cue the live observer missed. The team marks both items unresolvable for agreement, improves camera and audio boundaries with consent, clarifies timing, and samples again.

Keep supervision, quality, and employment roles separate for Quinn

Quinn's integrity observer-disagreement review identifies the clinical decision owner, observer, supervisor, quality reviewer, operations owner, privacy contact, and employment owner. State which process is active, what notice and evidence it uses, who may decide, and where the record belongs. One integrity observation should not silently change purpose after collection.

Protect client access during Quinn's review

Quinn's process keeps AAC, interpreters, mobility, food, water, bathroom use, prescribed care, health support, rest, relationships, and emergency help available. Obtain required consent and assent when applicable and follow the governing response to withdrawal or distress. Client protection can require immediate action while broader measurement and personnel conclusions remain pending.

Use raw evidence and matched denominators for Quinn

Report Quinn's correctly implemented components divided by eligible components due, observations completed divided by observations due, agreement pairs divided by pairs due, and repairs verified divided by repairs due. Keep procedure versions, critical steps, exposure, client outcomes, direct experience, access, missingness, adaptations, drift, burden, and adverse effects in separate series.

Ask six supervision and review questions for Quinn

Use these questions in the integrity observer-disagreement review:

Which client decision, procedure version, component, and eligible opportunity apply?
Which people, settings, risks, supports, observation modes, and sample limits matter?
Which integrity, agreement, exposure, outcome, experience, access, and adverse-effect series stay separate?
Which skill, clarity, resource, workflow, safety, burden, supervision, or plan-fit barrier is supported?
Which clinical, quality, operations, privacy, employment, certification, or payer owner may act?
Which repair, stop rule, follow-up evidence, and review date close the loop?

Keep unresolved evidence visible with a state, owner, age, and next action.

A fictional supervision example for Quinn

Quinn is fictional and involved in a meal-choice routine scored live and from an approved recording. Reviewers freeze 28 component, opportunity, visibility, timing, recording, observer, training, expectancy, calculation, and repair fields and complete 19 of 28 by the checkpoint. Missing, unobservable, inapplicable, adapted, drifted, disputed, failed, and pending fields keep their defined states.

The integrity observer-disagreement review measures evidence completeness. It does not establish effectiveness, safety, acceptability, competence, employment performance, certification compliance, medical necessity, authorization, payment, generalization, maintenance, or causation.

Use current sources within their scope for Quinn

For Quinn's integrity observer-disagreement review, the CASP public summary supplies high-level individualized assessment, implementation, and evaluation scope for ABA treatment of autistic people. The BACB Ethics Code addresses competence, client involvement, consent and assent when applicable, supervision, risk, documentation, and evaluation for covered behavior analysts. The BACB supervision page routes readers to role-specific supervision and training resources; those requirements do not replace licensure, payer, employment, or case authority.

For Quinn, the BCBA Test Content Outline addresses procedural fidelity as examination content; case protocols require qualified case-specific authority. The Ferguson practitioner guide supports observable components and eligible opportunities. The Essig, Rotta, and Poling review supports caution in fidelity-report interpretation. ASHA says AAC users should always have access to their tools or devices. Set coaching triggers, comparison rules, and performance thresholds from the current case and setting.

Close Quinn's review

Ask Quinn, the implementer, and the responsible clinician to review the integrity observer-disagreement review through accessible communication. Record the evidence, perspectives, limitations, selected repair, responsible system, clinical decision, follow-up observation, client outcome and experience, and review date. Reopen the analysis when the procedure, client priority, access, health, setting, implementer, supervisor, risk, or outcome changes.

How to Respond When Observers Disagree on Treatment Integrity