How Should Observer Agreement Be Reported for Rare ABA Events?

ABA agreement rare events

Define the event, observation period, opportunity, interval length when used, event-matching tolerance, zero-event rule, and calculation before reviewing results. Report how many matched observations contained the event. A result based mostly on shared zeroes should be labeled so readers understand what drove the percentage.

Rare events are difficult because the important occurrences supply only a small amount of information. Two observers can agree during many quiet intervals and still disagree whenever the event occurs. Families should see the occurrence evidence rather than a single percentage dominated by nonoccurrence.

Build the raw agreement table

For interval recording, a simple table can show:

• intervals where both observers recorded an occurrence
• intervals where Observer A recorded an occurrence and Observer B did not
• intervals where Observer B recorded an occurrence and Observer A did not
• intervals where both recorded no occurrence

These four counts let the team calculate overall, occurrence, and nonoccurrence agreement when those methods fit. They also show whether shared zeroes drive the result. Report the observation duration, interval length, number of sessions, settings, and any event-matching tolerance.

Check what the result can support

Rare-event data may require longer or differently distributed observation, though more observation is useful only when it samples the relevant contexts ethically and feasibly. Avoid arranging dangerous or distressing situations merely to create events for an agreement check.

Define the opportunity when one exists

Some events are rare because the relevant opportunity is rare. A rate per hour may conceal that the event was possible only twice. When the measure supports it, report responses divided by eligible opportunities and define how an opportunity begins and ends.

For a medical, safety, or crisis event, ordinary clinical and safety records may be more important than an agreement calculation. The team should not delay help or create exposure so two observers can watch. Retrospective review can use available evidence while labeling its limits.

Plan ethical, representative observation

Distribute observation across times and contexts where the event might naturally occur, based on the clinical question and safety plan. Longer observation in irrelevant conditions adds shared zeroes without improving understanding. A camera or extra observer can also change the environment, so include consent, assent, privacy, and client burden in the design.

If no event occurs, report zero observed occurrences and the amount of observation. Do not call occurrence agreement 100% when there were no occurrence intervals to compare. Use “not applicable” or another defined label for a calculation with no eligible cases.

Use current clinical and measurement sources

The CASP public summary places assessment, planning, implementation, and evaluation within its autism-treatment scope. The BACB Ethics Code addresses competence, client involvement, consent and assent when applicable, documentation, and data-based evaluation for covered behavior analysts.

The BCBA Test Content Outline includes measurement, data integrity, observer agreement, and visual analysis as examination content. It does not set one calculation, sampling percentage, or action threshold for every case.

Choose the calculation for the measure

Vollmer and colleagues discuss practical consequences of data reliability and treatment-integrity monitoring. Reed and Azulay describe several agreement calculations. Method choice depends on the measurement system and the question being asked.

Keep communication and conditions visible

The ASHA AAC portal supports continuous communication-tool access. Observer agreement is meaningful only when the record also identifies communication access, ordinary supports, opportunity availability, and other conditions that affect what could be observed.

A rare communication event may be missed when observers look only for speech or when the AAC system is absent. Record whether the person's ordinary communication tools and partner supports were available. A missing support changes the conditions and can change whether the opportunity was valid.

A practical example

Across 100 intervals, both observers record no event in 96. They agree on one occurrence. Observer A records an occurrence alone in two intervals, and Observer B records one alone in one. Overall interval agreement is 97 of 100, or 97% because the 96 shared nonoccurrences and one shared occurrence agree.

The occurrence evidence tells a different story. Only one of four intervals scored as an occurrence by either observer was a shared occurrence. The clinician reports the raw table and avoids presenting 97% as strong evidence about event detection.

The team reviews the definition and learns that the event-matching tolerance was unclear. It clarifies the rule and plans matched observations during naturally relevant routines without arranging the event. The later sample remains a new phase. Immediate safety procedures continue regardless of whether an agreement percentage can be calculated.

What families can ask the report to say

Ask for occurrence count, shared zero count, observation time, opportunities, settings, calculation, zero-event rule, and uncertainty. The report should explain how the result affects care and which safety, health, or clinical decisions rely on other evidence.

Consider whether another measure is more informative

An interval agreement percentage may be a poor fit for every rare event. Depending on the clinical question, the team might use event-by-event matching within a defined tolerance, opportunities with a response, total count agreement, duration, latency, a verified incident record, device data, or another valid source. Each method has limits and should be selected by the qualified professional responsible for measurement.

For events reported by families but rarely seen by providers, keep the source and context. A caregiver report, client report, medical record, school record, and direct observation are different evidence types. The team can compare them without declaring one source wrong because another observer did not witness the event.

Keep urgent decisions separate from data collection

Low event frequency does not reduce the importance of immediate health, safety, abuse, neglect, or emergency action. Follow the applicable response and reporting process from the available facts. Observer agreement can inform later quality review, while it should never become a prerequisite for urgent help.

When a rare event carries high consequence, predefine the incident response, required documentation, people to notify, and follow-up. The measurement plan should support safety rather than create exposure for the sake of a statistic.

Families can ask who reviews every occurrence, how quickly the review happens, and which evidence can change the safety or clinical plan before a larger sample is available.

Keep overdue reviews visible.

Questions families can use

Ask how often the event occurred, whether shared zeroes dominate the result, what the opportunity was, which calculation was chosen, how observations were distributed, and what uncertainty remains.