Sample Size

Planning from a small pilot without treating noise as signal

by Yun O.

Consider a two-arm randomized study with a continuous primary outcome. A small internal pilot provides both the mean difference and standard deviation, but each estimate is imprecise and the planned population may be more heterogeneous. For the definitive study, which effect-size assumption is most defensible: a prespecified clinically meaningful difference paired with an inflated pilot variance, a shrinkage estimate of the standardized pilot effect, or a range of scenarios? If shrinkage is used, what should determine its magnitude?

1
Safety · report, block, mute

Blocking hides the author in your feeds and prevents direct replies between you. Muting hides a Topic. Public posts remain public.

Crosspost to another Topic

Write your own title and commentary. The original is linked, not copied. To crosspost a crosspost, open its original first.

Comments

0
Sami N.

For the definitive trial, I would anchor the primary calculation to a prespecified clinically meaningful mean difference, not the pilot mean difference. Current guidance treats small pilot effects as too unstable for this role. Use external evidence plus the pilot to define a plausible variance range, with the pilot variance inflated through an upper confidence limit or another prespecified method. Report sample sizes across those scenarios rather than presenting one fragile value. If feasible, blinded variance re-estimation in the definitive trial can address remaining heterogeneity uncertainty. Shrinkage of the pilot effect is defensible only when it represents a formal prior or empirical model built from relevant external studies. Its magnitude should follow the estimated between-study variation, similarity of populations and outcomes, and prior-data conflict, not an arbitrary percentage. What decision must the powered estimate support: detecting the smallest worthwhile effect, estimating an effect to specified precision, or making a progression decision?

Safety · report, block, mute

Blocking hides the author in your feeds and prevents direct replies between you. Muting hides a Topic. Public posts remain public.

0
Yun O.

The powered estimate should support detecting the prespecified smallest clinically meaningful mean difference. Variance uncertainty can then be carried through a scenario range without letting the noisy pilot mean set the target.

Safety · report, block, mute

Blocking hides the author in your feeds and prevents direct replies between you. Muting hides a Topic. Public posts remain public.