What if ABA Observers Watched Different Time Periods?

ABA observer time periods

Record each observer's actual start and end, interruptions, setting, opportunity exposure, device time, and reason for mismatch. Determine whether the overlap contains enough valid evidence for the planned calculation. If it does not, label the comparison unavailable and collect another matched sample before a high-stakes decision.

A planned observation and an actual observation are different records. One person may arrive late, lose a connection, leave for a safety need, or use a clock that differs from the other observer's device. The agreement review should begin with actual exposure rather than the scheduled session length.

Reconstruct the observation timeline

Create one timeline with:

• each observer's actual start and end
• pauses, interruptions, and unscorable periods
• clock source and any known offset
• setting or activity changes
• eligible opportunities and events by period
• communication and ordinary supports available
• the reason the windows differed

Do not align clocks by changing source entries. Preserve the originals and record any approved analytical adjustment. A one-minute clock offset can matter for brief event matching even when both people watched the same activity.

Check what the result can support

Counts from different exposure lengths answer different questions. Converting both to rates may help describe each record, though it does not create independence or a matched observation. Agreement calculations need the same events, intervals, or opportunities under the chosen method.

Use only a valid overlap for agreement

Suppose Observer A watches from 10:00 to 10:30 and Observer B watches from 10:12 to 10:30. The potential overlap is eighteen minutes. Before calculating agreement, confirm that both saw and could score the same events throughout those eighteen minutes. A camera freeze or blocked view can reduce the valid overlap further.

The first twelve minutes remain part of Observer A's clinical or measurement record. They simply cannot contribute to interobserver agreement. Labeling unmatched time as disagreement would penalize observations the second person never had a chance to make.

For opportunity-based measures, matching may depend on the same eligible opportunities rather than continuous clock time. For interval measures, the intervals and boundaries must align. For duration or count measures, the method still needs the same exposure period. The qualified measurement owner should choose the valid calculation.

Avoid “fixing” the mismatch with rates alone

Observer A may record six events in thirty minutes and Observer B four events in eighteen minutes. Rates of 0.20 and 0.22 events per minute look similar. That similarity is a descriptive comparison. It is not evidence that they agreed on which events occurred.

Use rates when they address the clinical question, with the correct exposure denominator. Use a paired agreement method only on matched evidence. Keep both analyses labeled so readers do not confuse them.

Use current clinical and measurement sources

The CASP public summary places assessment, planning, implementation, and evaluation within its autism-treatment scope. The BACB Ethics Code addresses competence, client involvement, consent and assent when applicable, documentation, and data-based evaluation for covered behavior analysts.

The BCBA Test Content Outline includes measurement, data integrity, observer agreement, and visual analysis as examination content. It does not set one calculation, sampling percentage, or action threshold for every case.

Choose the calculation for the measure

Vollmer and colleagues discuss practical consequences of data reliability and treatment-integrity monitoring. Reed and Azulay describe several agreement calculations. Method choice depends on the measurement system and the question being asked.

Keep communication and conditions visible

The ASHA AAC portal supports continuous communication-tool access. Observer agreement is meaningful only when the record also identifies communication access, ordinary supports, opportunity availability, and other conditions that affect what could be observed.

If AAC was unavailable during one period, the opportunity conditions changed. Record when access was restored and whether both observers could see or hear the relevant response. The mismatch may reveal an access or operational problem beyond the timing issue.

A practical example

Observer A scores a 30-minute routine from 9:00 to 9:30. Observer B joins at 9:12 after a secure connection problem. Three events occur during the missed period, and five occur during the eighteen-minute overlap.

Within the overlap, Observer A records all five events and Observer B records four. The team applies its defined event-matching rule to those five possible events and reports 4 of 5 matched events, or 80%. It does not compare Observer A's eight total events with Observer B's four as an agreement calculation.

The record retains Observer A's first twelve minutes as single-observer data, labels the connection issue, and shows 18 of 30 minutes of paired coverage. Because the result will inform a major plan decision, the clinician schedules another full matched observation after technology testing. The new sample does not erase the timing limits of the first.

Prevent the same timing problem

Use a shared time source, pre-observation connection check, actual start confirmation, and a rule for interruptions. Assign someone to decide whether enough valid overlap remains. If the sample fails, report it promptly and reschedule rather than stretching an invalid comparison to meet a deadline.

Handle interruptions consistently

Observers may lose visibility for different reasons. One may answer an urgent question, the client may move behind a barrier, a remote feed may freeze, or another person may enter the space. Record the interruption start and end for each observer and whether the target event remained observable.

An interruption can create three periods: paired and scorable, paired but unscorable, and single-observer. Calculate agreement only from the valid paired period. Retain the other records for any appropriate clinical or operational use with their limits. If the interruption itself reflects a safety, access, or technology problem, route that issue separately.

Show the timing repair in the report

A family-facing summary can list each observer's actual window, valid overlap, events inside and outside it, calculation, reason for mismatch, and next check. It should identify whether the dependent decision proceeded, was qualified, or was held.

If a software export automatically labels the full scheduled session as paired, correct the report while preserving the source times. Test the export after the fix. Measurement quality depends on the workflow that creates the denominator as well as the people recording behavior.

Questions families can use

Ask whether start and end times matched, which events occurred outside the overlap, whether the method permits partial comparison, how missing time is labeled, and whether a fresh sample is needed.