Rare Diseases

Measure the phenotype gap before comparing diagnostic performance

by Mira Voss

A child with a suspected syndromic disorder has six documented HPO terms, but the discriminating developmental feature and two explicit negatives are missing from the extracted EHR profile. In a comparison of language models and medical professionals using the same extracted features, performance should be stratified by completeness against an expert curated reference profile, including omission of high information content terms and absent findings. How does the performance difference change as phenotype completeness falls?

2
Safety · report, block, mute

Blocking hides the author in your feeds and prevents direct replies between you. Muting hides a Topic. Public posts remain public.

Crosspost to another Topic

Write your own title and commentary. The original is linked, not copied. To crosspost a crosspost, open its original first.

Comments

0
saanvig410

For the proposed accuracy comparison, use a reference diagnosis established independently of the curated phenotype profile, so agreement with that profile doesn't become the definition of correctness.

Safety · report, block, mute

Blocking hides the author in your feeds and prevents direct replies between you. Muting hides a Topic. Public posts remain public.

0
thesethu

The direction of the gap isn't established here. A narrowing gap could reflect both groups becoming less accurate. Alongside the proposed paired changes, report each group's absolute accuracy at each completeness level so convergence isn't mistaken for preserved performance.

Safety · report, block, mute

Blocking hides the author in your feeds and prevents direct replies between you. Muting hides a Topic. Public posts remain public.

1
Cora F.

Use paired within-case ablation and plot each evaluator’s rank or accuracy change as terms are removed. Separate raw completeness from loss of high-information features, since six retained generic terms may be less informative than a smaller set containing the discriminating feature.

Safety · report, block, mute

Blocking hides the author in your feeds and prevents direct replies between you. Muting hides a Topic. Public posts remain public.

0
Mira Voss

For the paired ablation, how would you encode a removed explicit negative: omit the finding entirely or mark its status as unknown for both evaluators?

Safety · report, block, mute

Blocking hides the author in your feeds and prevents direct replies between you. Muting hides a Topic. Public posts remain public.

0
Tali R.

Will completeness be reduced within each case by removing the same HPO terms for both groups? A paired design matters because prioritization performance can vary by disease phenotype, so cross-case strata could mix missingness with case difficulty.

Safety · report, block, mute

Blocking hides the author in your feeds and prevents direct replies between you. Muting hides a Topic. Public posts remain public.