Systematic replication

The variation is deliberate

A direct replication aims to repeat methods and conditions closely. A systematic replication changes selected features to test robustness, boundary conditions, mechanisms, or generality.

The change should follow a research question. Altering many untracked features makes it hard to identify why results differ.

Replication uses new data

The National Academies definitions chapter distinguishes replication using new data from computational reproducibility using the same data and analysis. Fields use the terms differently, so every report should state the operational definition.

Reanalysis can catch coding errors. Replication tests a claim with another set of observations.

Start from an explicit claim

Name the effect, direction, magnitude or criterion, population, response, outcome, comparator, setting, and time frame that the original study supports. Decide which part the replication will test.

Avoid defining success after seeing the new data. Predefine consistency criteria, including uncertainty and clinically relevant differences.

Change one interpretable dimension

Useful variations can include age, communication mode, culture, site, implementer, target response, material, schedule, dosage, measure, or treatment component. Preserve enough of the design to retain the same scientific question.

Explain why the changed dimension matters and which other conditions stayed stable. Record implementation fidelity and contextual events.

Functional equivalence may matter more than form

In applied research, a procedure can require adaptation for accessibility, language, setting, or participant preference. The visible materials may change while the functional relation being tested remains defined.

Describe both the adaptation and the preserved mechanism. Do not treat inaccessible copying as methodological purity.

A fictional multisite replication

A team repeats a teaching procedure across six sites with a planned change in implementer. Four sites meet the predefined effect and maintenance criteria. Two show variable responding and incomplete implementation fidelity.

Report 4 of 6 sites meeting criterion, plus every site's raw result, fidelity, context, and uncertainty. Calling the study a success or failure would erase the mixed pattern.

Consistency is more than statistical significance

Compare direction, magnitude, uncertainty, time course, maintenance, adverse effects, and social validity. A significant result in one study and nonsignificant result in another can still have compatible estimates.

The National Academies replicability chapter emphasizes proximity and uncertainty rather than a universal binary rule.

A different result can be informative

Nonreplication can reflect a false original claim, chance, low precision, measurement differences, implementation, population, context, or a real boundary condition. Inspect these possibilities without assuming misconduct.

Preserve the full data and planned analyses. Report deviations and null or unfavorable findings.

Design quality still controls inference

Replication cannot repair weak measurement, biased sampling, absent comparison, ambiguous outcomes, or poor fidelity. Use a design capable of answering the specific question.

The NIH Additional Research Methods page provides general resources. Field-specific design standards and expert review remain necessary.

Clinical use requires another step

Consistent replication can strengthen confidence that a relation extends across studied conditions. It does not guarantee benefit for every person or authorize a clinical plan.

Integrate the evidence with assessment, competence, client goals, assent, risk, feasibility, culture, resources, and ongoing data.

A reporting checklist

• What original claim is being tested?
• Which features changed, and why?
• Which features and decision rules stayed constant?
• How were fidelity and outcomes measured?
• What counted as consistency before analysis?
• Are all sites, participants, and outcomes reported?
• What boundary condition does the result suggest?

Plan a replication matrix

Create one row for each original feature: participant, setting, implementer, materials, response definition, outcome, measurement system, comparison, dosage, fidelity, analysis, follow-up, and context. Mark each feature as held constant, deliberately changed, or uncontrolled.

For every planned change, state:

• the scientific reason for the variation
• the predicted effect or boundary condition
• the accessibility or feasibility need
• the measurement and fidelity check
• the consistency criterion and uncertainty rule

Lock the matrix before data collection. Record deviations and contextual events without rewriting the original prediction.

Report results for every participant, site, outcome, and planned condition. Keep excluded or incomplete cases visible with reasons. If a new procedure was added after weak results, show the phase change rather than pooling it with the planned replication.

This structure allows readers to see whether the study tested generality, mechanism, transportability, or simple repetition. It also prevents “systematic replication” from becoming a label for any study that resembles earlier work.

Archive the preregistration, protocol, materials, data dictionary, code, fidelity records, deviations, and analysis so another team can understand what was preserved and changed.

Use a go, revise, or stop decision after each planned replication wave. Review fidelity, adverse events, participant experience, effect uncertainty, context differences, and data quality before expanding. A result that differs from the original should trigger explanation and boundary testing rather than being relabeled as implementation failure.