Interobserver agreement

IOA checks whether observers recorded alike

Agreement can reveal unclear definitions, missed events, timing problems, inadequate observer training, or a recording system that is hard to use. It is a quality check on measurement.

High agreement does not prove that both observers were correct. Two people can consistently apply the same flawed definition or miss the same event. Accuracy requires a credible reference or other validation evidence.

Define the observation before calculating

Record the target response, eligible opportunity, session boundary, interval length, scoring window, observers, setting, and exclusions before collection. Both observers should watch the same events without copying, coaching, or seeing each other’s data.

Sampling also matters. Agreement collected only during calm, familiar sessions says little about complex settings where definitions are harder to apply. Sample relevant clients, staff, activities, response rates, locations, and times while protecting privacy and assent.

Before formal sampling, observers can score the same practice examples and discuss disagreements after both records are locked. Calibration should cover clear cases, near misses, overlapping responses, interruptions, and boundary events. During live collection, keep observers independent. Periodic recalibration can address drift, while the team preserves earlier scores and records each definition change with its effective date.

Total-count IOA compares session totals

For total-count IOA, divide the smaller observer count by the larger count and multiply by 100. Suppose one observer records 18 requests and another records 20 during the same session:

18 ÷ 20 × 100 = 90% total-count IOA.

This method is easy to calculate and can conceal disagreement about when events occurred. Observers might reach similar totals while recording different events.

Interval-by-interval agreement checks location in time

Divide the observation into equal intervals. Count intervals where both observers agree on occurrence or both agree on nonoccurrence. Divide those agreements by all eligible intervals and multiply by 100.

If observers agree in 7 of 10 intervals, interval-by-interval agreement is 7 ÷ 10 × 100 = 70%. The result is lower than the 90% total-count value because the interval method tests where events appeared, not just the final totals.

Occurrence agreement can be informative when a response is rare, while nonoccurrence agreement can be informative when it is frequent. Each uses a different eligible denominator. Name the exact formula rather than reporting “IOA” alone.

Duration and trial measures need matching units

For total-duration IOA, divide the shorter recorded duration by the longer duration. More detailed methods compare duration within intervals or event by event. For trial-by-trial agreement, divide trials with matching scores by trials observed by both people.

Use consistent units and rounding. A missing trial should remain visible as missing data instead of silently becoming disagreement or disappearing from the denominator.

A fictional observation shows why method matters

Two fictional clinicians independently observe Jordan using an accessible break message during a 20-minute art group. They use ten two-minute intervals. Observer A records 18 messages; Observer B records 20, giving 90% total-count IOA. Their interval records match in seven intervals, giving 70% interval-by-interval agreement.

The team reviews the raw timestamps and finds that rapid repeated messages near interval boundaries were scored inconsistently. It clarifies the response onset and the rule for repeated messages, practices with new examples, and samples another session. The first scores remain in the record.

Neither result describes whether staff honored Jordan’s message, whether the art group was accessible, or whether the message met Jordan’s goal. Those require separate measures.

Use agreement to improve the measurement system

When agreement is lower than the predeclared review criterion, examine the pattern before retraining people. The definition, examples, equipment, observation position, event rate, interval size, or competing duties may be the main problem.

Report agreement by method and condition. Show raw agreements, disagreements, and eligible units. A single average can hide one observer pair, setting, or response class that needs attention.

Set a review date after any definition or training change. The next sample should include the conditions that exposed the original problem. Report whether agreement improved, which disagreements remain, and whether the updated definition still measures the intended response.

The ABAI basic-principles page and BACB BCBA Test Content Outline, 6th edition place measurement and agreement concepts within professional education and examination content. CASP’s resources page provides organizational and practice resources. These sources support professional attention to measurement; the clinical team still selects a fit-for-purpose method.