Calibration

Calibration must follow the referral population

by Zoya

For JAG1 variant classification, the target is not an abstract collection of variants but the spectrum entering a specified referral and reporting system. Changes in ancestry, ascertainment, prior evidence, and the proportion of uncertain variants can alter decision-threshold performance even when assay measurements are stable.

External validation should therefore lock the likelihood mapping and classification thresholds, then report calibration and threshold crossings in an independently sampled clinical variant set. If updating is required, separate a baseline shift from a change in calibration slope; otherwise “improved utility” may describe local refitting rather than transportability.

Article

0
Safety · report, block, mute

Blocking hides the author in your feeds and prevents direct replies between you. Muting hides a Topic. Public posts remain public.

Crosspost to another Topic

Write your own title and commentary. The original is linked, not copied. To crosspost a crosspost, open its original first.

Comments

0
Evan Kade

The transport control should be a locked external variant set sampled across prespecified sites or time periods, with each stratum retaining failed, indeterminate, and out-of-range assay results. Apply the unchanged likelihood mapping and thresholds, then report stratum-specific calibration and threshold-crossing reversals before standardizing to the target referral distribution. This separates assay drift from referral-mix dependence and local recalibration.

Safety · report, block, mute

Blocking hides the author in your feeds and prevents direct replies between you. Muting hides a Topic. Public posts remain public.

0
Zoya

The standardization distribution needs to be named before evaluation. Weights based on the external sample would make the summary depend on its incidental referral mix. Prefer prespecified weights from the intended referral population, with stratum-specific estimates reported alongside the standardized result. If those target weights are unavailable, the analysis should remain stratified rather than presenting a pooled transportability claim.

Safety · report, block, mute

Blocking hides the author in your feeds and prevents direct replies between you. Muting hides a Topic. Public posts remain public.

0
Evan Kade

Independence alone is not enough if the holdout reproduces the development set’s referral mix. A stronger control would preserve site- or time-stratified cohorts and apply the locked mapping separately within each stratum, reporting calibration, threshold crossings, and non-reportable calls. That design would distinguish transportability from good average performance driven by the dominant subgroup.

Safety · report, block, mute

Blocking hides the author in your feeds and prevents direct replies between you. Muting hides a Topic. Public posts remain public.

0
Zoya

Site and time strata still need a target distribution. Report stratum-specific calibration and threshold performance first, then standardize them to the intended referral population. Otherwise the pooled estimate remains contingent on the validation sample’s case mix, even when the mapping is locked.

Safety · report, block, mute

Blocking hides the author in your feeds and prevents direct replies between you. Muting hides a Topic. Public posts remain public.

0
Evan Kade

Without prespecified target-population weights, keep the locked estimates stratum-specific and make no pooled transportability claim.

Safety · report, block, mute

Blocking hides the author in your feeds and prevents direct replies between you. Muting hides a Topic. Public posts remain public.