Measurement artifact

Artifacts come from the measurement process

Suppose a graph shows a sudden increase in response count. The response may truly have changed. The same pattern could appear because sessions became longer, a button began submitting twice, observers broadened the definition, or missing zeros were added during export.

An artifact is not evidence of dishonesty. It is a reason to examine how observations became numbers. Keep the original record, correction history, and uncertainty visible while the review proceeds.

The recording system can shape the result

Common sources include:

• different observation lengths or opportunity counts
• timer drift, delayed entry, or incorrect time zones
• duplicate taps, cached submissions, and synchronization retries
• skipped intervals or automatic zero filling
• rounding, unit conversion, and denominator changes
• filters that hide missing or excluded sessions
• graphs with compressed axes or mixed procedure versions

A count of 12 during a two-hour session differs from 12 during 20 minutes. Convert to rate only when the observation time is valid and the response definition supports that comparison.

Observers can introduce reactivity and bias

People may respond differently when they know observation is occurring. Observers can also change their scoring when a supervisor is present. Codding and colleagues found observer-performance changes under some second-observer conditions. The result supports checking reactivity without predicting it in every setting.

Observer error also has patterns. One person may score ambiguous events as present, while another tends to miss brief responses. Lerman and colleagues used signal-detection methods to distinguish accuracy from response bias in simulated tasks.

Agreement between observers is useful and does not prove validity. Two observers can apply the same unclear definition consistently. Calibration should include positive examples, close nonexamples, difficult boundaries, and the actual devices and environments used in practice.

The display can create a persuasive illusion

A dashboard may join percentages built from different denominators, connect observations across a procedure change, or show an average without its range. An apparent drop from 80% to 50% means something different when the values are 4 of 5 and 50 of 100.

Inspect the raw numerator, denominator, unit, time window, exclusions, and procedure version. Mark missing data as missing. A blank field, zero response, absent opportunity, client withdrawal, and device failure represent different states.

Use a structured artifact review

When a pattern looks surprising:

1. preserve the original display and source records
2. define the claimed change and affected dates
3. check observation duration, opportunities, and eligibility
4. inspect device logs, timestamps, exports, and transformations
5. compare observer definitions, training, and agreement records
6. seek independent evidence from another measure or source
7. document the finding, correction, owner, and downstream decisions

Avoid silently overwriting a clinical record. A permitted correction should preserve original content, authorship, actual service time, entry time, reason, and audit history under the governing policy and requirements.

A fictional duplicate-record example

Amara's school team sees 24 displayed observations for a week. Four records have identical device event IDs and timestamps created during offline synchronization. The team verifies 20 unique observations and retains the four duplicates in an audit log rather than treating them as new events.

Displayed-record validity is 20 of 24, or 83.3%. The corrected analysis uses the 20 verified records. It does not assign the duplicates as client responses or zeros. The team reruns any summary affected by the changed denominator and tells decision-makers which version is current.

The team also checks a paper sample collected during the same period. Similarity would increase confidence; disagreement would trigger further review rather than automatic preference for the electronic record.

Downstream effects need their own inventory. The team checks whether the duplicates reached a progress graph, treatment review, caregiver report, supervision record, payer document, or research extract. Each affected artifact receives a correction or an explicit note explaining why no change was needed.

Protect the person during measurement review

Do not recreate dangerous, painful, or distressing events to determine whether a spike was real. Keep communication and AAC, food, water, bathroom access, mobility, prescribed care, breaks, and emergency help available.

Ask the person and relevant partners about changes in health, environment, access, routines, and observer presence. Their reports are identified sources of evidence rather than substitutes for direct measurement.

The BACB BCBA Test Content Outline, Sixth Edition covers measurement dimensions, reliability, validity, and representative procedure selection. A measurement-quality paper further illustrates calibration concerns. Neither source supplies a universal threshold for deleting data.