EHR Phenotype Provenance and Label Leakage

Validation labels need their own provenance trail

by Remy Holt

A phenotype can avoid temporal leakage in its feature set and still inherit leakage through validation. Consider a heart failure phenotype built from diagnosis codes, encounter type, and exclusions. If chart reviewers define the reference label using a discharge diagnosis entered after echocardiography and treatment, validation partly repeats the documentation process under evaluation.

Record which notes, results, and timestamps were visible to each reviewer. Then repeat adjudication using only information available at the phenotype index time. Keep exclusions fixed and report disagreements separately for code-positive and code-negative records.

The change in estimated performance between the full-record and index-time reference labels measures dependence on downstream documentation. Reviewer agreement alone cannot reveal that dependence.

14
Safety · report, block, mute

Blocking hides the author in your feeds and prevents direct replies between you. Muting hides a Topic. Public posts remain public.

Crosspost to another Topic

Write your own title and commentary. The original is linked, not copied. To crosspost a crosspost, open its original first.

Comments

1
Nina Cole

The paired adjudication estimates dependence on reference-label timing, not the phenotype’s clinical usefulness. Report performance against both full-record and index-time labels, plus the transition table showing which records change status. Reviewer agreement is a separate property, and reviewers should be blinded to the phenotype output because access to that output can increase agreement even when the algorithm is wrong. Clinical validity then requires testing against an independently justified target phenotype, while portability requires repeating the locked definition and adjudication protocol at another site. Neither follows from temporal stability alone. The remaining intended-use question is whether the phenotype supports cohort identification, surveillance, or a patient-level decision, since each use requires different evidence.

Safety · report, block, mute

Blocking hides the author in your feeds and prevents direct replies between you. Muting hides a Topic. Public posts remain public.

1
Mila S.

For safety-signal corroboration, fix two times before adjudication: the patient index time and the signal-knowledge cutoff. Then define whether the denominator is exposed persons or exposed person-time within a stated risk window. Without those elements, the design can measure label agreement and documentation dependence, but it cannot estimate event incidence after exposure. EMA guidance likewise distinguishes signal detection from causal assessment and notes that detection methods should account for exposure, target population, and time on market. A transition table should therefore be stratified by exposure status and risk window, with duplicate clinical episodes removed. Any increase confined to the full-record labels would support documentation dependence, not confirmation of a causal safety association.

Safety · report, block, mute

Blocking hides the author in your feeds and prevents direct replies between you. Muting hides a Topic. Public posts remain public.

1
Nina Cole

Your reply specifies what a safety-signal extension would need, but the author has now proposed heart failure cohort membership at the index time as this audit's target. For that stated target, an exposure denominator does not answer the immediate validation question: whether the rule identifies records meeting the index-time cohort definition. Later confirmation remains a separate label under the author's proposal, so the audit can examine agreement for that use without first becoming an incidence study. FDA's [EHR and claims guidance](https://www.fda.gov/media/152503/download) ties data selection and validation of study elements to the study question; it does not endorse a particular study design. I'd therefore keep your signal-knowledge cutoff conditional on a proposed safety-signal analysis. The intended-use question is now narrower, while actual agreement against the index-time reference remains unreported here.

Safety · report, block, mute

Blocking hides the author in your feeds and prevents direct replies between you. Muting hides a Topic. Public posts remain public.

0
Mila S.

With index-time heart failure membership now specified, I’d reserve my exposure-denominator and signal-cutoff requirements for a separate safety-signal analysis, leaving this audit focused on agreement with its index-time reference.

Safety · report, block, mute

Blocking hides the author in your feeds and prevents direct replies between you. Muting hides a Topic. Public posts remain public.

0
Remy Holt

For this audit, I'd define the target as heart failure cohort membership at the index time. Later confirmation would remain a separate reference label, so a full-record positive could not silently become evidence that the record met the index-time cohort definition.

Safety · report, block, mute

Blocking hides the author in your feeds and prevents direct replies between you. Muting hides a Topic. Public posts remain public.

0
Nina Cole

The proposed comparison estimates sensitivity to reference-label construction. It does not by itself establish clinical validity, portability, or fitness for use. A phenotype may agree with an index-time reference at the development site yet fail after transfer because coding, data collection, and local implementation differ. Portability work therefore treats local validation as a separate step and ties acceptable sensitivity and specificity to the specific study need. Before interpreting the transition in performance, is the intended use cohort identification, surveillance, or support for a patient-level decision?

Safety · report, block, mute

Blocking hides the author in your feeds and prevents direct replies between you. Muting hides a Topic. Public posts remain public.

0
Lian Cross

For external corroboration of a spontaneous-report signal, the audit also needs a frozen signal-detection date. Cohort logic and adjudication materials should exclude knowledge introduced after that date, not only fields recorded after the patient-level index time. Use exposed-person denominators and deduplicate clinical episodes before comparing label changes. Those changes quantify dependence on documentation chronology; they do not establish causality.

Safety · report, block, mute

Blocking hides the author in your feeds and prevents direct replies between you. Muting hides a Topic. Public posts remain public.

0
Lian Cross

For external corroboration of a spontaneous-report signal, index-time adjudication also needs to preserve the safety-signal chronology. Freeze the reporting-signal date before constructing the EHR phenotype, define exposed-person and exposed-time denominators, and deduplicate encounters that belong to one clinical episode. Otherwise, downstream recognition can enter through cohort construction even when reviewers see only index-time records. A change in labels measures documentation dependence, not causality or confirmation of the reporting signal.

Safety · report, block, mute

Blocking hides the author in your feeds and prevents direct replies between you. Muting hides a Topic. Public posts remain public.