To calculate trial by trial IOA, align the same eligible trials, compare the observers' scored response or outcome for each trial, divide agreements by trials compared, and multiply by 100. Define prompts, invalid trials, missing observation, and accepted response forms in advance. A high percentage can still hide shared scoring bias or weak trial definitions.

Create stable trial IDs

Match observers to the same cue, opportunity, time, response window, and trial number.

Define the scored outcome

State whether agreement concerns occurrence, independence, response category, correctness, prompt level, or another field.

Keep invalid trials separate

Exclude only under a predefined rule and report exclusions. A missing trial is not agreement.

Review the disagreement type

Definition, prompt timing, access, and observer visibility can need different repairs.

Lock trial eligibility before comparing responses

Create a trial ledger with a stable identifier, cue or opportunity time, response window, available communication mode, prompt state, scored outcome, and validity status. Both observers must refer to the same eligible trial. A row seen by only one observer remains a coverage gap and does not become a disagreement or an agreement.

Define the comparison field at the level needed for the decision. Binary occurrence agreement can hide disagreement about independence, response form, correctness, or prompt level. A multi-category decision should retain a confusion table showing which categories were confused, even if the final headline remains the proportion of exact trial matches.

Keep trial coverage beside IOA

The IOA denominator contains matched valid trials. A separate coverage denominator contains all trials scheduled for dual observation. Report both when any trial is missed or invalid. This prevents a high agreement percentage from representing only the easiest trials or the portion with the best visibility.

Observer records should remain independent until locked. Shared prompts, live comparison, or one observer copying a system-entered value can create agreement without independent measurement. Store recording mode, timestamps, corrections, and any loss of blinding.

Use the formula and units for Luca

For Luca's trial-by-trial IOA worksheet: trial-by-trial IOA = trials with observer agreement / matched trials compared × 100. Keep observer records independent until both are locked. Preserve the matched exposure, raw observations, numerator, denominator, unrounded calculation, and final rounding.

Work the arithmetic for Luca

Luca's observers independently score ten matched choice trials and agree on eight. Trial-by-trial IOA is 8 / 10 × 100 = 80%. One disagreement concerns whether a gesture met the accepted response definition, and one concerns whether a prompt began before the response window closed.

The disagreement proportion is 2 of 10, or 20%. Because the two mismatches have different causes, one combined retraining action would be poorly targeted. The gesture mismatch calls for an accessible response-definition review; the prompt-timing mismatch calls for a shared timing rule and observation practice.

Apply trial agreement to the decision

Luca's qualified clinician identifies the decision before choosing an agreement method or threshold. Compare a stricter or complementary method when prevalence, totals, or aggregation could hide relevant disagreement. Repair definitions, visibility, training, sampling, or matching before judging the implementer or client data.

Audit observer and sample conditions for Luca

Luca's record identifies observers, independence, training, recording mode, shared exposure, response definition, procedure version, observation coverage, missing pairs, reactivity, and disagreements. Consent, privacy, recording, employment, supervision, and payer rules remain separate requirements. Keep every open pair visible with an owner and next action.

Protect client access during Luca's observation

Luca's observation keeps AAC, interpreters, mobility, food, water, bathroom use, prescribed care, health support, rest, relationships, and emergency help available. Do not create extra distress or remove ordinary supports to obtain an agreement sample. Record when access, health, withdrawal, or observer presence changes the condition.

Ask five IOA questions for Luca

Use these questions in the trial-by-trial IOA worksheet:

  • Did both observers independently score the same exposure, unit, definition, and condition?
  • Which calculation, numerator, denominator, prevalence, and rounding reproduce the result?
  • Which disagreements, missing pairs, invalid units, reactivity, or matching problems remain?
  • Which validity, treatment-integrity, client-outcome, experience, and access questions stay separate?
  • Which repair, complementary method, next sample, owner, and review date follow?

Keep raw observer records and disagreement types available.

Use current sources within their scope for Luca

For Luca's trial-by-trial IOA worksheet, the BACB ethics hub points to current professional ethics materials. The CASP public summary provides high-level autism-treatment scope without licensed detailed procedures. The BACB Ethics Code addresses competence, client involvement, consent and assent when applicable, documentation, risk, supervision, and evaluation for covered behavior analysts. The BCBA Test Content Outline includes measurement, reliability, sampling, and data-based decisions as examination content rather than a case protocol.

For Luca, Vollmer, Sloman, and St. Peter Pipkin discuss practical implications of data reliability and treatment-integrity monitoring. Essig, Rotta, and Poling review IOA and fidelity reporting in an identified ABA research corpus. Reed and Azulay describe a calculation tool and agreement methods. ASHA says AAC users should always have access to their tools or devices. These sources do not create one universal IOA method or threshold.

Close Luca's agreement review

Ask Luca and the responsible clinician to review the trial-by-trial IOA worksheet through accessible communication. Record the method, arithmetic, sample, disagreements, limits, measurement repair, selected decision, next observation, and review date. Reopen the analysis when the definition, observer, system, exposure, prevalence, client access, procedure, or decision changes.

Clinician release checklist

  • Do both records point to the same ten eligible trials and response windows?
  • Is the scored field defined, including accepted speech, AAC, gesture, and prompt states?
  • Can another reviewer reproduce eight agreements, two typed disagreements, and 80%?
  • Are planned dual-observation coverage and matched-trial IOA kept separate?
  • Are observer independence, correction history, access, and the two different repair actions documented?

Related resources

Sources