Quality of heart failure registration in primary care: observations from 1 million electronic health records in the Amsterdam Metropolitan Area

Study design and setting

Our study used data from a large dynamic study cohort sourced from the Academic General Practice Research Network (AGPRN) of the Amsterdam University Medical Centers. Patients registered at over 100 practices in the Amsterdam Metropolitan Area affiliated with this network have the option to refuse reuse of their records for scientific research, though the opt-out rate is low (<0.5 %).

Participants

We included all patient records pertaining to patients who were at least 18 years of age and had one or more episodes of care registered on 31/12/2021.

HF identification

Patient records in our dataset were structured using episodes of care, which bundle consultation related to the same complaint or diagnosis. These episodes contain fields for ICPC-codes and physicians’ notes, which we will capitalize on for this study. The database was searched for episodes labeled with K84.03 (cardiomyopathy), K77 (decompensation cordis), and the latter’s subcodes K77.01 (acute decompensation cordis) and K77.02 (chronic decompensation cordis).

To investigate potential underregistration, we employed two information retrieval methods that make use of episode descriptions in order to find episodes that pertain to a HF diagnosis but are not coded as such.

The first search method makes use of manually engineered regular expressions (RegEx), a type of search pattern that facilitates complex query construction using wildcards, grouping, Boolean operators and many other features. Our designed expressions, which can be found in Supplement A, allow for some alternate or incorrect spellings of the search terms and exclude some expressions of uncertainty or exclusion of diagnosis. As the query is deterministic it yields a fixed number of matches.

The second text-based method further extends the idea of capturing incorrectly spelled terms or descriptions that pertain to HF by making use of FastText [12]. This algorithm converts words – preprocessed and separated into tokens – to vectorized representations. Similar to Word2Vec [13], these representations or “embeddings” attempt to capture semantic similarity, in which words with similar meaning have similar embeddings. In addition to this, FastText uses subword information to capture syntactic similarity, where words that share similar spelling will have similar embeddings. This subword embedding technique also allows for generating embeddings for out-of-vocabulary words. A FastText model was trained on all physician’s notes in our dataset. Details regarding this training procedure can be found in Supplement B. Embeddings for episode description were generated by averaging across the vectorized representations of each word within [14]. These description embeddings were normalized for sequence length through dividing them by their L2-norm [15].

This would allow us to rank episodes by calculating the distance or similarity between their description embeddings and an embedding of a known reference representing a description for an HF episode. However, as descriptions of HF episodes may come in many forms, it doesn’t suffice to look at the distance to a single reference. Our set of episodes with an HF ICPC-code contains a wide range of descriptions that we can employ as positive references, yet we lack a set of negatives. Covering the domain of descriptions that do not pertain to HF would require intractable amounts of manual validation and labeling, eliminating the typical binary classification algorithm options. The solution lies in fitting a one-class support vector machine (OCSVM) on the embedded descriptions of all validated HF-coded episodes [16]. Subsequently, all description embeddings of other episodes were ranked based on their distance from the decision boundary of the OCSVM. To facilitate comparison to the k descriptions found by the RegEx, we evaluate the top k episode descriptions as ranked by this method.

Validation

All episodes descriptions identified through either ICPC-code or textual search were evaluated by a panel of two reviewers. This panel represents expertise in general practice and cardiovascular morbidity, in both patient-facing as well as research roles (RH), combined with a background in medical informatics and the epidemiology of HF (LDC). Our study did not assess the validity of HF diagnoses themselves, but rather focused on the agreement between registered ICPC-codes and their accompanying descriptions. As such, only episodes that unambiguously excluded HF as a diagnosis, yet were coded as such, were considered to be false positives by our panel. This includes obvious accidental miscoding, explicit exclusion or uncertainty regarding an HF diagnosis, and diagnoses pertaining relatives of the patient. Inversely, in the evaluation of the textually retrieved episodes, our panel flagged only those that explicitly feature a HF diagnosis as positive cases. This approach provides the benefit of the doubt for both over- and underregistration and prioritizes specificity over sensitivity in the identification of miscoding. HF subtype information was registered when information regarding the left ventricular ejection fraction (LVEF) was found in the episode description, making a distinction between the HF subtypes with a reduced ejection fraction (HFrEF) for those with a LVEF<40 % and those with mildly-reduced (HFmrEF, LVEF=40–49 %) or preserved ejection fraction (HFpEF, LVEF≥50 %).

Statistical analysis

Over- and underregistration of HF

A total of 19,256,707 episodes were queried for HF using both ICPC-codes and descriptions, spanning 968,433 EHRs. The results of validating these codes against their descriptions can be seen in Table 1.

At the episode-level, 24,011 carried an HF ICPC-code, the majority of which were found under the K77 code (87.4 %). Our panel identified 1,324 as false positive, an overregistration rate of 5.5 %. Using the validated episode descriptions found in our textual search, we identified an additional 4,582 HF episodes, which translates to an underregistration rate of 16.8 %. Plotting these measurements over time, we observed a significant decrease in the HF episode underregistration rate (p<0.001), going from 23.8 % (95 % CI: 21.5–26.5) in 2006 to 10.0 % (95 % CI: 8.3–11.1) in 2021, as can be seen in Figure 1. By way of contrast, the overregistration rate did not present a significant trend (p=0.096).

Figure 1:

Heart failure registration quality over time. Under- and overregistration of heart failure (HF) episodes per year (A) and expressed as rate in reference to the number of validated HF episodes (B). Stems indicate the 95 % confidence interval of registration error rates.

In terms of patient records, these values translate to 19,106 patients with an HF ICPC-coded episode, of which 909 records are eliminated once episodes incorrectly coded as HF were removed, making for an overregistration rate of 4.8 %. Conversely, an additional 2,266 records of patients with a HF diagnosis were identified using the additional HF episodes found by the textual search, translating to a patient-level underregistration rate of 11.1 %. The result of this effort was a total of 20,463 EHRs validated as having HF (2.1 %). After inclusion of the previously unidentified HF episodes, the first known date of HF diagnosis was determined to be earlier in 717 records (3.7 %), with a median shift of 3.45 years (IQR 0.97–8.11). The effects of these false and missed inclusions on population characteristics are apparent in Table 2. We observed a statistically significant shift in the patients’ age and history of hypertension, coronary artery disease, atrial fibrillation, and chronic kidney disease, as registered at the time of HF diagnosis.

Distribution of ICPC-codes in miscoded HF episodes

Table 3 presents the ICPC-codes most frequently found across all miscoded HF episodes. These rates are also expressed for the subsets of “impactful” episodes, i.e. those that yielded either new patient records with HF or those that precede the first-known HF ICPC-code in a record, as opposed to those that follow a correctly ICPC-coded HF episode. Out of all episodes which ought to have a HF ICPC-code but did not, more than a quarter (25.3 %) lacked any ICPC-code. More than half (55.8 %) were found in ICPC’s chapter K, encompassing cardiovascular afflictions, which reached 59.3 % of coverage when only considering impactful episodes. The most prevalent code among these was K78 (Atrial fibrillation), making up 11.7 % of all newly identified episodes and 13.2 % of impactful episodes. The top ten ICPC-codes were responsible for nearly half (48.7 %) of impactful episodes and all but one, R02 (Dyspnea attributed to respiratory system), fall in chapter K. The distribution across the different chapters may be found in Supplemental Table C.

HF subtype registration

Our panel was able to identify 2,532 episodes (11.2 %) describing HFrEF and 1,558 episodes (6.9 %) pertaining to HFmrEF or HFpEF. The remaining 18,597 episodes (82.2 %) did not name the subtype and contained no information regarding LVEF.

Identification of HF diagnosis descriptions

The RegEx filters identified a total of k=4,881 unique episode descriptions, of which 3,260 were verified as pertaining to HF by the panel, yielding a precision of 66.8 % for this manually defined method. Evaluation of the top k as ranked by the OCSVM yielded 2,391 HF episode descriptions validated as such, for a precision-at-k of 49.0 % for this self-supervised method. A visual comparison of these precisions can be found in Supplemental Figure D. The union of both methods yielded a total of 7,518 unique descriptions, of which 3,761 were validated as pertaining to HF, for a total free-text search precision of 50.0 %. A subset of 2,244 descriptions was identified by both retrieval methods, of which 1,890 were validated as HF, achieving a precision of 84.2 % for the intersection of both methods.

Related work

Strengths and limitations

The scale and coverage of our dataset instils confidence regarding its representativeness of the population in the Amsterdam Metropolitan Area. Whether our findings translate to the Netherlands as a whole or other countries employing ICPC-coding remains to be seen.

In comparison to most validation studies of ICPC-coding in the Netherlands, our sample size is substantially bigger. This allowed us to investigate HF underregistration, being the first to do so in the Netherlands, as well as the distribution of ICPC-codes in the additionally found HF episodes.

Our information retrieval method is partially based on FastText, which generates static word embeddings that do not take into account the contexts in which words are found in. As such, the OCSVM’s performance was hampered by its inability to take e.g. statements of negation or doubt into account. Even though this was partially alleviated by combining its results with those of the RegEx-based search, this did not reduce the number of descriptions to be validated and it is likely that our retrieval methods missed miscoded HF episodes where their shortcomings overlap. More recent language models produce contextualized embeddings and are likely to substantially improve both precision and recall of this retrieval task. These can come in the form of general purpose language models such as LLaMA [31] or GPT-4 [32], or Dutch medical domain-specific models such as MedRoBERTa.NL [33].

In the validation of retrieved episodes, data privacy regulations prevented the use of an independent panel, introducing a risk for validation bias. To compensate, our panel remained conservative in its judgments. Combined with a lack of sufficient detail in many descriptions, this likely prevented identification of certain false positives and false negatives. As such, the numerators of both the over- and underregistration rates are likely to be underestimates and ought to be interpreted as lower limits. Furthermore, the correlation between the elaborateness of an episode description and the level of scrutiny that allows for may have introduced a bias in favor of less adequate notes.

Validation efforts being hampered by a large variety in note-taking quality and quantity is not a new phenomenon. Similar conclusions were drawn by Sporaland et al. in their investigation into the congruence between ICPC2-codes and their accompanying descriptions [34]. Future efforts may seek to assess an entire patient record in the validation of a code, looking beyond the episode it’s attached to, as has been performed by panels on smaller samples in the past [6, 7, 10]. Information from subsequent consultations, referrals, and prescriptions is likely to substantially improve the detection of miscoding. However, such a validation task may be considered intractable when considering datasets of this scale. Solace may be found in the aforementioned advancements in language models, whose contextual and generative capacities may be leveraged for such an effort. In this pursuit, a smaller sample may be evaluated concurrently by a panel to assess the validation performance of such a language model before applying it to broader datasets.

Finally, the use of the designated episode start date rather than the date at which it was registered is a trade-off, as GPs may choose to antedate HF diagnoses where they deem applicable, yet registration dates were found to contain technical inaccuracies in this particular dataset. Our choice in this trade-off may have introduced bias in the temporal analysis.