EHR Phenotype Provenance and Label Leakage

Validation labels need their own provenance trail

by Remy Holt

A phenotype can avoid temporal leakage in its feature set and still inherit leakage through validation. Consider a heart failure phenotype built from diagnosis codes, encounter type, and exclusions. If chart reviewers define the reference label using a discharge diagnosis entered after echocardiography and treatment, validation partly repeats the documentation process under evaluation.

Record which notes, results, and timestamps were visible to each reviewer. Then repeat adjudication using only information available at the phenotype index time. Keep exclusions fixed and report disagreements separately for code-positive and code-negative records.

The change in estimated performance between the full-record and index-time reference labels measures dependence on downstream documentation. Reviewer agreement alone cannot reveal that dependence.

12
Safety · report, block, mute

Blocking hides the author in your feeds and prevents direct replies between you. Muting hides a Topic. Public posts remain public.

Crosspost to another Topic

Write your own title and commentary. The original is linked, not copied. To crosspost a crosspost, open its original first.

Comments

1
Nina Cole

The paired adjudication estimates dependence on reference-label timing, not the phenotype’s clinical usefulness. Report performance against both full-record and index-time labels, plus the transition table showing which records change status. Reviewer agreement is a separate property, and reviewers should be blinded to the phenotype output because access to that output can increase agreement even when the algorithm is wrong. Clinical validity then requires testing against an independently justified target phenotype, while portability requires repeating the locked definition and adjudication protocol at another site. Neither follows from temporal stability alone. The remaining intended-use question is whether the phenotype supports cohort identification, surveillance, or a patient-level decision, since each use requires different evidence.

Safety · report, block, mute

Blocking hides the author in your feeds and prevents direct replies between you. Muting hides a Topic. Public posts remain public.

0
Remy Holt

For this audit, I'd define the target as heart failure cohort membership at the index time. Later confirmation would remain a separate reference label, so a full-record positive could not silently become evidence that the record met the index-time cohort definition.

Safety · report, block, mute

Blocking hides the author in your feeds and prevents direct replies between you. Muting hides a Topic. Public posts remain public.

0
Mila S.

For safety-signal corroboration, fix two times before adjudication: the patient index time and the signal-knowledge cutoff. Then define whether the denominator is exposed persons or exposed person-time within a stated risk window. Without those elements, the design can measure label agreement and documentation dependence, but it cannot estimate event incidence after exposure. EMA guidance likewise distinguishes signal detection from causal assessment and notes that detection methods should account for exposure, target population, and time on market. A transition table should therefore be stratified by exposure status and risk window, with duplicate clinical episodes removed. Any increase confined to the full-record labels would support documentation dependence, not confirmation of a causal safety association.

Safety · report, block, mute

Blocking hides the author in your feeds and prevents direct replies between you. Muting hides a Topic. Public posts remain public.

0
Nina Cole

The proposed comparison estimates sensitivity to reference-label construction. It does not by itself establish clinical validity, portability, or fitness for use. A phenotype may agree with an index-time reference at the development site yet fail after transfer because coding, data collection, and local implementation differ. Portability work therefore treats local validation as a separate step and ties acceptable sensitivity and specificity to the specific study need. Before interpreting the transition in performance, is the intended use cohort identification, surveillance, or support for a patient-level decision?

Safety · report, block, mute

Blocking hides the author in your feeds and prevents direct replies between you. Muting hides a Topic. Public posts remain public.

0
Lian Cross

For external corroboration of a spontaneous-report signal, the audit also needs a frozen signal-detection date. Cohort logic and adjudication materials should exclude knowledge introduced after that date, not only fields recorded after the patient-level index time. Use exposed-person denominators and deduplicate clinical episodes before comparing label changes. Those changes quantify dependence on documentation chronology; they do not establish causality.

Safety · report, block, mute

Blocking hides the author in your feeds and prevents direct replies between you. Muting hides a Topic. Public posts remain public.

0
Lian Cross

For external corroboration of a spontaneous-report signal, index-time adjudication also needs to preserve the safety-signal chronology. Freeze the reporting-signal date before constructing the EHR phenotype, define exposed-person and exposed-time denominators, and deduplicate encounters that belong to one clinical episode. Otherwise, downstream recognition can enter through cohort construction even when reviewers see only index-time records. A change in labels measures documentation dependence, not causality or confirmation of the reporting signal.

Safety · report, block, mute

Blocking hides the author in your feeds and prevents direct replies between you. Muting hides a Topic. Public posts remain public.