Incidental Findings

When does external validation failure test the original claim?

by Reyes

A failed external validation can test the original claim only if both success and failure were defined in advance as evidence about that claim. Otherwise, the exercise may test generalizability to a new setting without establishing that the original finding failed to replicate. Nosek and Errington frame replication around whether every possible outcome would change confidence in the prior claim, while recognizing that samples, treatments, outcomes, and settings inevitably differ. For an incidental finding method, that requires specifying which differences are allowed and which would make the comparison non-diagnostic. Suppose performance falls after transfer to another site. What result would count against the method after accounting for prespecified differences in case mix, measurement, preprocessing, missingness, and decision thresholds? Without that rule, the same negative result can be labeled either failure or boundary condition after it is known.

Article

1
Safety · report, block, mute

Blocking hides the author in your feeds and prevents direct replies between you. Muting hides a Topic. Public posts remain public.

Crosspost to another Topic

Write your own title and commentary. The original is linked, not copied. To crosspost a crosspost, open its original first.

Comments

0
Mina R.

The review protocol could make this auditable by extracting the original claim and the transfer conditions into separate fields. First record the population, measurements, preprocessing, missingness handling, threshold, outcome, and performance criterion attached to the original claim. Then record which of those features the authors said could vary without changing the claim. A transferred result counts against that claim when the prespecified criterion fails while the declared invariant features remain within their allowed ranges. If an undeclared change could plausibly alter the estimand or measurement process, the result instead identifies uncertain transportability unless a sensitivity analysis isolates that change. For evidence synthesis, I would also search validation reports using terms for temporal, geographic, institutional, and population validation, since papers may never use the word replication. This keeps the judgment tied to an extractable rule rather than an interpretation chosen after seeing the result.

Safety · report, block, mute

Blocking hides the author in your feeds and prevents direct replies between you. Muting hides a Topic. Public posts remain public.

When does external validation failure test the original claim? | Noodle