Validate Permanent-Product ABA Data: Clinical Guide

To validate permanent-product ABA data, verify that the lasting product has a credible relation to the defined response, a known author, an eligible opportunity, a consistent scoring rubric, and a bounded review window. Check automation, shared accounts, prompts, edits, imports, and access barriers. Product evidence can omit latency, effort, distress, partner help, and the process used.

Decide whether a lasting product can answer the question

A permanent product is useful when the response leaves a durable, reviewable result and the product has a credible relation to that response. Define the response, eligible opportunity, product, review window, author or source, scoring unit, and clinical question. List alternate ways the same product could appear, such as automation, copying, partner creation, imports, shared accounts, or edits after the opportunity.

Malik's worksheet concerns submitted digital products. It separates source validity from later quality scoring. A submission must link to Malik, an eligible opportunity, the declared time window, and a permitted production route before it enters the outcome denominator.

Build a product-validation record

Use one row per expected or submitted product. Include product ID, opportunity ID, client, author evidence, account, creation and modification times, setting, device or system, assignment version, access supports, automation state, shared-work state, validity decision, reason, scorer, rubric version, item score when eligible, and correction history.

Keep the original artifact or governed reference according to privacy and record rules. A filename or login alone may not prove authorship. Restrict sensitive contents while allowing reviewers to see the validity evidence they need.

Apply source-validity gates before scoring

Check that the opportunity existed, the product falls within the review window, the product-response link is credible, authorship or attribution meets the declared rule, the product was not auto-generated outside that rule, and the artifact is complete enough for the intended rubric. Mark each product valid, held, invalid, missing, duplicate, or outside exposure with a source-based reason.

Only valid products move to outcome scoring. A held product remains unresolved and should not be counted as correct, incorrect, complete, or incomplete client performance. This ordering prevents a scoring rubric from giving false precision to an artifact whose source is uncertain.

Calculate Malik's validity coverage

Malik has 12 submitted digital products. Ten link to Malik and an eligible opportunity. One was auto-generated, and one came from a shared account without sufficient attribution.

Submitted products = 12.
Source-valid products = 10.
Held products = 2.
Product-source validity = 10 / 12 × 100 = 83.3%, rounded to one decimal place.

The two held products stay visible by reason. The 83.3% result describes source validity among submitted products. It is not Malik's task accuracy. No outcome percentage should be calculated from the example until the ten valid products are scored under the declared rubric.

Use a reproducible rubric for valid products

Define each scoring category, item, allowable evidence, partial-credit rule, scorer response to ambiguity, and final aggregation. Train scorers with examples and preserve item-level decisions. When reliability matters, select a representative sample for independent scoring and report the agreement method and denominator.

Do not revise the rubric after viewing Malik's results without versioning the change and rescoring the affected cohort. If one product can contain several items, distinguish product count from item count. Report missing items separately from incorrect items when the source evidence supports that distinction.

Keep process information alongside the product

A product may omit latency, time spent, prompts, assistance, distress, errors corrected before submission, communication method, access barriers, and who selected the task. Collect these variables separately when they matter to the clinical question. Do not infer independence from a finished artifact.

Review whether the opportunity was meaningful and accessible. Maintain AAC, interpreters, mobility supports, food, water, bathroom access, prescribed care, rest, relationships, and emergency help. ASHA supports ongoing AAC access. Automation or assistive technology may be an ordinary support rather than a reason to reject a product; the validity rule should reflect the actual response being measured.

Handle missing and ambiguous products

Reconcile expected opportunities with submitted products. A missing artifact can reflect no response, a save failure, deleted evidence, wrong account, inaccessible task, absent opportunity, or another process state. Use the prewritten rule and preserve the reason. Repeated missing products in one platform, partner, or access condition can limit representativeness.

For collaborative products, define whether the analysis unit is the group, individual contribution, or another observable unit. If attribution cannot be resolved, hold the item or use a different measure. Late edits need a cutoff and version history so the scorer knows which artifact belongs to the observation window.

Graph and report validity before outcomes

Show expected, submitted, source-valid, held, invalid, and missing products by date or opportunity. Then graph outcome scores only for the valid cohort, with the valid denominator visible. Avoid merging source validity and task performance into one percentage.

Malik submitted 12 digital products. Ten met the declared opportunity and source-attribution gates, yielding 83.3% source-valid coverage. One auto-generated product and one shared-account product remained held and were excluded from client-outcome scoring pending resolution.

Pritchard and colleagues, Wirth and colleagues, and Lobo and colleagues provide broader measurement-accuracy and single-case context. Product validity still requires a page-specific response-product analysis.

Keep professional sources in scope

The BACB ethics hub and CASP public summary provide professional context. The BCBA Test Content Outline covers product measures, validity, reliability, and representative measurement as examination content. These sources do not validate a particular platform, rubric, authorship rule, or product.

Clinician review checklist

To validate permanent-product ABA data, confirm the decision question, response-product relation, opportunity definition, alternate production routes, attribution rule, review window, expected cohort, submitted cohort, missing states, validity gates, held reasons, validity arithmetic, rubric version, scorer training, agreement sample when needed, process measures, graph, accessible client review, qualified owner, and review date.

Limits of permanent-product data

A lasting product does not reveal the complete response process, prove authorship, establish independence, show client experience, validate the scoring rubric, demonstrate treatment integrity, or identify cause. Digital systems can alter, duplicate, generate, or lose artifacts. Source-valid coverage is different from outcome performance. Keep ambiguous products out of the client-outcome denominator, preserve process and access information, and use direct observation or another method when the product cannot answer the question.

How to Validate Permanent-Product Data in ABA