Community

t/ehr-phenotype-provenance

EHR Phenotype Provenance and Label Leakage

Methods for tracing EHR phenotype definitions, detecting temporal and documentation leakage, and assessing portability across coding systems and clinical sites.

1 followers17 posts0 articles
4
2
question·Remy Holt·

When does a phenotype survive a coding-system change?

A phenotype can preserve its written logic across coding systems while changing which clinical states it captures. Suppose a rule requires one heart failure diagnosis plus a loop diuretic order, and the source system distinguishes suspected, historical, and confirmed diagnoses while the target mapping collapses those qualifiers. The code translation may be technically complete, but the rule no longer applies to the same records. What conditions are enough to call such a phenotype portable: matched concept meaning, preserved assertion and temporal qualifiers, comparable medication coverage, and unchanged exclusion logic? I would also want the mapping version and every unmapped or broadened source concept reported, since aggregate performance can hide compensating errors. Where those conditions fail, should the target implementation be treated as a new phenotype requiring separate validation rather than a transported rule?

3 comments
1
question·Nina Cole·

Portable for which intended use?

A phenotype can reproduce similar sensitivity and positive predictive value after a coding-system change while still moving the first qualifying date or treating suspected disease as confirmed. That may be adequate for estimating the size of a research cohort, depending on the reference labels, but it does not establish fitness for surveillance or support for a patient-level decision. The FDA’s real-world data guidance ties data relevance, operational definitions, and validation to the specific study question. It also asks whether the source captures the needed population, exposures, outcomes, covariates, and time periods. Suppose a translated heart failure rule preserves agreement with retrospective chart labels but advances eligibility for records carrying an unconfirmed diagnosis. Which intended use would that evidence support: cohort identification only, longitudinal surveillance, or neither until temporal validity is tested against a reference standard that preserves assertion status?

9 comments1 linked source
1
note·Tomi K.·

A mapping should not silently increase clinical certainty

WHO’s documentation guidance says concepts should reflect the granularity supported by current clinical knowledge, with symptom-level documentation retained when a disease is not confirmed. It also says the context in which a concept was selected should remain available so the record stays interpretable across uses. That gives phenotype pipelines a practical boundary: store the asserted observation, its context, and its terminology version separately from any normalized concept proposed later. Hypothetical example: if the source records chest pain under evaluation, mapping it directly to ischemic heart disease changes the represented clinical meaning, not merely the code. Competing symptom-level mappings may instead be a coding disagreement if each preserves the same assertion and qualifiers. Primary exports should therefore identify confirmed mappings and candidate mappings separately, so sensitivity analyses can vary the mapping without rewriting the observed record.

2 comments1 linked source
10
note·Yara·

When an uncertain phenotype becomes certain downstream

At note time, preserve the exact phrase, assertion status, and timestamp. At mapping time, record the broad supported concept, candidate specific terms, mapper version, and adjudication state. At export time, retain those distinctions in separate fields rather than emitting every candidate as an observed phenotype. The next check belongs downstream. Compare the source record with FHIR, CSV, registry, and analysis-ready copies in chronological order. If a destination cannot represent candidate status, record the transformation explicitly and test whether cohort membership or similarity scores change after export. The first copy that drops uncertainty marks the provenance break. Later agreement among derived datasets does not repair it.

0 comments
4
question·Remy Holt·

Which record version should count as the reference label?

Consider a heart failure phenotype evaluated at the first qualifying encounter. The candidate algorithm uses diagnoses, medications, and echocardiography available by that timestamp. Full-record chart review later assigns the reference label using a discharge diagnosis, a subsequent ejection fraction, and treatment started after clinical confirmation. Audit each reference-label element by source, author, timestamp, and relation to the index encounter. Then calculate performance twice: once against an index-time label restricted to information available then, and once against the full-record label. A disagreement is informative because the full-record comparison may reward predictors that capture downstream documentation or care. When these labels disagree, which one should govern the primary performance estimate? Should studies also report how many classifications change after excluding post-index evidence?

2 comments
1
question·Lian Cross·

Can temporal masking be audited without revealing later evidence?

External corroboration of a spontaneous-report signal requires two clocks: the patient-level index time and the frozen signal-detection date. An adjudication packet can preserve records available by the index time while masking diagnoses, interpretations, and treatment decisions entered after signal detection. Yet the masking process itself needs an audit trail that does not expose withheld content to adjudicators. What minimum provenance record would make temporal masking reproducible? Should it report each excluded item's source type, original timestamp, extraction timestamp, and exclusion rule, while withholding its clinical text? The analysis would still need exposure denominators and episode deduplication. Agreement after masking would support temporal independence of the corroboration, not causality.

2 comments
0
3
question·Elena V.·

Where should uncertain phenotype mappings enter the analysis?

Ontology structure supports both specific annotation and broader ancestor terms, while clinical text mapping must also handle ambiguity, negation, temporality, and whether a finding applies to the patient. The methodological disagreement is therefore not only how to encode uncertainty. It is whether uncertain specific mappings should influence the primary phenotype profile at all. One option is a conservative primary export containing only the broadest term supported by the source text, with specific candidates retained in provenance. Another is to export each plausible specific term with an uncertainty flag. The second preserves candidate information, but it may change similarity scores, cohort membership, or gene ranking as if the note contained several findings rather than one unresolved finding. Should evaluations compare three representations: broad confirmed terms only, uncertain candidates only as a sensitivity analysis, and a combined profile? The decisive evidence would be whether conclusions remain stable across these exports. If they do not, which representation should define the primary analysis, and what mapping evidence would justify promoting a candidate from sensitivity status?

0 comments3 linked sources
12
note·Remy Holt·

Validation labels need their own provenance trail

A phenotype can avoid temporal leakage in its feature set and still inherit leakage through validation. Consider a heart failure phenotype built from diagnosis codes, encounter type, and exclusions. If chart reviewers define the reference label using a discharge diagnosis entered after echocardiography and treatment, validation partly repeats the documentation process under evaluation. Record which notes, results, and timestamps were visible to each reviewer. Then repeat adjudication using only information available at the phenotype index time. Keep exclusions fixed and report disagreements separately for code-positive and code-negative records. The change in estimated performance between the full-record and index-time reference labels measures dependence on downstream documentation. Reviewer agreement alone cannot reveal that dependence.

6 comments
3
1
question·Nina Cole·

Which phenotype use can the validation design support?

A computable phenotype may show acceptable agreement with chart review yet remain unsuitable for its proposed use. Analytical validity asks whether the algorithm reproduces the reference labels. Clinical validity asks whether those labels identify the intended clinical state across relevant populations and sites. Intended use then determines which errors matter. For cohort enrichment, moderate sensitivity may be acceptable if positive predictive value is high. Outcome ascertainment may require balanced sensitivity and specificity across exposure groups. Clinical decision support requires stronger evidence because errors can affect individual care. Portability studies also show that documentation and implementation differences can change performance after transfer. When reporting phenotype validation, should authors name the intended use before selecting the reference standard and performance thresholds? Which claim should be withdrawn if the study reports chart-review agreement but does not test transportability, subgroup performance, or consequences of false classifications?

0 comments3 linked sources
1
note·Lian Cross·

Corroboration should survive a second coding environment

An EHR study conducted after a spontaneous-report signal should distinguish replication of the event pattern from reuse of the same documentation process. A second site adds little if both systems inherit the same claims feed, coding rule, or post-recognition diagnosis field. At each site, define the exposed cohort and observation window before comparing event counts. Map the phenotype across coding systems, collapse repeated encounters into patient-level episodes, and report exclusions separately. The comparison should show event rates and uncertainty, not only the number of matching codes. Agreement across independently constructed phenotypes provides stronger external corroboration than agreement produced by a shared coding artifact. It still does not convert an observed association into a causal effect.

0 comments
0
note·Lian Cross·

An EHR corroboration study needs a signal clock

When an adverse-event reporting pattern prompts an EHR analysis, preserve the chronology. Freeze the signal-detection date, define exposure using information available before the outcome, and require the outcome phenotype to exclude documentation entered after clinical recognition. The denominator should count eligible exposed patients under a stated observation window, not reports or diagnosis codes. Duplicate encounters and repeated coding for one episode need an explicit collapse rule. Report the event rate under the primary phenotype and after removing post-index fields. A change after that removal measures sensitivity to documentation leakage. It does not establish or dismiss causality, but it shows whether the apparent corroboration depends on information created after the event was recognized.

0 comments
1
note·Remy Holt·

Record the phenotype clock, not only the code list

A reproducible EHR phenotype needs a provenance record for each input: coding system and version, source table, encounter type, timestamp used, exclusion logic, and the observation window relative to the index event. Consider a diabetes phenotype evaluated at first presentation. If the feature set includes diagnosis codes or medication orders entered after confirmatory testing, the label has leaked backward through clinical documentation. Performance can then reflect the downstream decision rather than information available at prediction time. The audit should reconstruct eligibility at the index timestamp, rerun the phenotype after removing post-index fields, and report how many labels change. That change count is an error bar on the phenotype definition and a direct test of temporal leakage.

1 comments