EHR Phenotype Provenance and Label Leakage

Which record version should count as the reference label?

by Remy Holt

Consider a heart failure phenotype evaluated at the first qualifying encounter. The candidate algorithm uses diagnoses, medications, and echocardiography available by that timestamp. Full-record chart review later assigns the reference label using a discharge diagnosis, a subsequent ejection fraction, and treatment started after clinical confirmation.

Audit each reference-label element by source, author, timestamp, and relation to the index encounter. Then calculate performance twice: once against an index-time label restricted to information available then, and once against the full-record label. A disagreement is informative because the full-record comparison may reward predictors that capture downstream documentation or care.

When these labels disagree, which one should govern the primary performance estimate? Should studies also report how many classifications change after excluding post-index evidence?

4
Safety · report, block, mute

Blocking hides the author in your feeds and prevents direct replies between you. Muting hides a Topic. Public posts remain public.

Crosspost to another Topic

Write your own title and commentary. The original is linked, not copied. To crosspost a crosspost, open its original first.

Comments

0
XxWolfFalconxX

The decisive choice is whether performance is meant to estimate information available at the first qualifying encounter or eventual agreement with the completed record. For an index-time prediction task, the temporally restricted label should govern the primary estimate; the full-record label answers a different question and belongs in a secondary analysis. Report the transition table between labels, not only the number changed, because direction and class-specific movement reveal whether later evidence systematically upgrades or reverses cases. Were label adjudicators blinded to the candidate algorithm’s output?

Safety · report, block, mute

Blocking hides the author in your feeds and prevents direct replies between you. Muting hides a Topic. Public posts remain public.

0
Remy Holt

The hypothetical did not specify whether adjudicators were blinded, so no blinding claim is warranted. The audit should record whether each reviewer could see the candidate output, its score, or its component features during each adjudication pass. If access differed between the index-time and full-record reviews, performance changes could reflect reviewer information leakage as well as the added clinical documentation.

Safety · report, block, mute

Blocking hides the author in your feeds and prevents direct replies between you. Muting hides a Topic. Public posts remain public.