Diagnostic properties of natriuretic peptides and opportunities for personalized thresholds for detecting heart failure in primary care

Study design and patient population

We conducted a retrospective study making use of routine primary care data registrations collected by the Amsterdam General Practice Network, a collaboration between the Amsterdam University Medical Center and 117 general practices in the Amsterdam metropolitan area. The dataset is considered representative of the general population, since residents in the Netherlands are obligated to register to a GPs’ office for access to healthcare, and registration rates exceed 99 % of the population [11]. The data in the registry consists of structured (e.g., age, sex, diagnostic codes, medication codes and laboratory findings) and unstructured (e.g., free text within episode notations) data (also see Supplementary Table S1). Patient records are coded according to the International Classification of Primary Care (ICPC). ICPC codes were used for registration of diseases, but also for the reason for encounter, for instance for symptoms like dyspnea, edema, fatigue. For the current study, we applied the following in- and exclusion criteria. Patients were included if they were: (1) at least 18 years of age; and (2) with a BNP or NT-proBNP measurement conducted between January 2011 and December 2021. We excluded patients if they had: (1) a diagnosis of HF at baseline; (2) a follow-up of <90 days after NP-measurement or; (3) if NP measurement results did not specify an associated measurement unit preventing us to determine the applicable diagnostic threshold. If there were records of multiple NP measurements per patient, only the first test was selected for the analyses. Supplementary Figure S2 displays the process.

Index test (assay)

The reported levels of BNP or NT-proBNP were collected and converted to the same measurement unit (ng/L). Tests had been performed in several local laboratories that used different assays e.g., Advia Centaur BNP assay (Bayer, Tarrytown, NY) and Elecsys^® NT-proBNP assay (Roche, Mannheim, Germany). For our first analysis on diagnostic accuracy of NT-proBNP and BNP for HF, the standardized cut-off values we applied were <125 ng/L for NT-proBNP and <35 ng/L for BNP, as defined in the ESC-guidelines [1, 12]. For our second analysis on whether accuracy could be increased by the use of additional clinical variables, personalized thresholds were assessed. Further elaboration on the modeling of the personalized threshold can be found in the statistical analysis paragraph.

Clinical reference standard

Verification of HF diagnoses in the Amsterdam General Practice Network was described previously [13]. In short, the database was screened for HF diagnoses (up to 90 days after the natriuretic peptide test) by two researchers on ICPC-codes linked to HF (K77, K84.03) and subjected to a free text search for words suggesting HF in GP notes. An expert panel evaluated possible HF cases and subsequently determined and verified the diagnosis based on the three pillars, namely: symptoms suggestive of HF, observed findings with physical examination, and objective proof of structural or functional abnormality of the heart.

Factors considered for personalized thresholds

We performed a literature search to identify determinants that potentially affect NP values and/or increase the risk of HF. The determinant selection process can be found in Supplementary Tables S2–S4 [13–25].

Statistical analysis

Data were presented as numbers and percentages or median and 25th and 75th percentiles. Chi-squared test was used to determine statistical significance of differences in categorical variables, and the Mann-Whitney-U test was used for continuous variables. To describe diagnostic accuracy we used specificity, sensitivity, positive and negative predictive values (PPV and NPV, respectively) and calculated the area under the receiver-operator curve (AUROC) and area under the precision-recall curve (AUPRC). The ROC curve is a well-known method to graphically show the relation between sensitivity and specificity. For the AUROC we consider a value of 0.7–0.8 as fair, 0.8–0.9 as good and above 0.9 as excellent. The lesser known area under the precision recall curve (AUPRC) shows the relation between the sensitivity (‘recall’) and PPV (‘precision’). When prevalence of disease is low the AUPRC is sometimes better suitable than AUROC to demonstrate difference in discrimination, as it more closely correlates with the PPV [26, 27]. However, AUPRC results are not as easily interpretable as they vary according to disease prevalence, with AUPRC tending to 0 with decreasing prevalence [26]. The baseline of the AUPRC is the prevalence of the disease within the analyzed population. As a consequence, thresholds for fair, good or excellent AUPRC are not determinable beforehand, and comparison of AUPRC results will be narratively described. Given that primary care research often operates in low-prevalence settings, we presented both the AUROC and AUPRC to provide additional context to our findings.

To correct for differences between the sample receiving a BNP test and the sample receiving a NT-proBNP test we implemented inverse probability of treatment weighting (IPTW) using the propensity score [28]. The propensity score consisted of age, sex and the 10 previously selected clinical predictors (Supplementary Table S2). In this analysis, the “treatment” to “no treatment” comparison, as prescribed by IPTW, was filled in respectively by NT-proBNP as compared to BNP.

The second analysis consisted of assessing whether improvements in diagnostic accuracy could be made for BNP and NT-proBNP measurements. The sample was randomly divided in a test (20 %) and training set (80 %), stratified according to outcome (HF registration). The conventional ESC-defined cut-off values were compared to subgroup-specific cut-offs, in which the population was split according to age, sex and the selected covariables [13]. In determining the optimal cut-off for each NP for each group (presence or absence of each variable) sensitivity was set at that resulting from the ESC guidelines’ cut-off for each NP, with changes to the other diagnostic accuracy parameters determined by the newly differentiated NP thresholds. Significance at the 95 % level was established using p-values with correction for multiple hypotheses testing using the Bonferroni method [29]. Bootstrapping (1,000 times) was used to estimate the 95 % confidence intervals. For statistical analyses Python 3.7.11 was used with our statistical procedures making extensive use of the Scikit-learn library [30].

Patient characteristics

We identified a total of 15,234 patients who underwent NP testing over the observed time period, of whom 8,365 with NT-proBNP and 6,873 with BNP. The baseline characteristics for both NPs stratified by HF are displayed in Table 1. Patients with HF were older, had more comorbidities, more often presented with dyspnea, and had higher NT-proBNP and BNP values compared with those without HF. When comparing the groups who underwent NT-proBNP testing vs. BNP testing, we found a higher percentage of HF at follow-up in patients who underwent NT-proBNP testing compared with those who underwent BNP-testing (5.7 vs. 4.5 %, p=0.002).

NT-proBNP and BNP test characteristics

In patients in whom an NT-proBNP measurement was conducted, we found an (IPTW-adjusted) AUROC of 89.9 (95 % CI 88.4–91.2) for detecting HF. The AUPRC was 40.4 (95 % CI 35.6–45.0) When the ESC cut-off was applied, a sensitivity of 95.3 (95 % CI 93.0–97.1), a specificity of 59.1 (95 % CI 58.0–60.2), PPV of 12.6 (95 % CI 11.5–13.7) and NPV of 99.5 (95 % CI 99.3–99.7) were found.

In patients with a BNP measurement, an AUROC of 85.9 (95 % CI 83.5–88.2) was found. The AUPRC was 29.5 (95 % CI 25.0–34.8). When the ESC cut-off was applied a sensitivity of 89.7 (95 % CI 86.1–93.2), specificity of 58.0 (95 % CI 56.6–59.2), PPV of 8.9 (95 % CI 7.9–9.9) and NPV of 99.2 (95 % CI 98.9–99.5) were found.

Figure 1 graphically displays the discriminatory properties of NT-proBNP in relation to BNP. After correcting for differences in baseline characteristics and HF prevalence, we found that NT-proBNP was associated with better discriminatory ability with a higher AUROC and AUPRC compared with BNP. Furthermore, the ESC cut-off for NT-proBNP performed better on sensitivity, specificity, PPV and NPV compared to BNP’s ESC cut-off (94.7 vs. 85.5 %, 60.6 vs. 56.3 %, 12.6 vs. 8.5 %, and 99.5 vs. 98.8 %, respectively; Supplementary Tables S5 and S6).

Figure 1:

Discrimination of the natriuretic peptides (NT-proBNP and BNP). (A) AUROC and (B) AUPRC of NT-proBNP and BNP with 95 % confidence intervals. Blue for BNP and green for NT-proBNP. ESC-cut-off indicated by black dot on curve for both NPs.

Analyses of pre-specified diagnostic test modifiers

Table 2 and detailed Supplementary Tables S5 and S6 illustrate the optimal cut-off in the presence or absence of specific diagnostic test modifiers, and a comparison with the ESC cut-offs. All personalized thresholds for BNP and NT-proBNP resulted in modest increase in specificity and PPV compared to use of the uniform ESC cut-off alone, while sensitivity remained statistically unchanged. The number of patients who would have had a positive test result (and thus need for additional diagnostic evaluation) based on the differentiated cut-offs was lower for all tested variables (Supplementary Tables S5 and S6).

For NT-proBNP, most notable improvement was seen when differentiating for age (threshold of 125 for age >75 vs. 150 ng/L for age ≤75) and COPD (threshold of 138 when present vs. 141 ng/L when absent) which improved specificity by 3.5–3.6 %. For BNP, improvement was more substantial, with specificities improving by 8–11 % for age, male sex, AF and COPD (significant difference in differentiated thresholds after Bonferroni correction only for AF and COPD), and other tested variables in the range 6–8 %.

Comparison to previous work

Strengths and limitations

A strength of our study is the large dataset of patients with a high level of data granularity, resulting in a sizeable subset of suspected HF patients in whom NP testing had occurred. For our analyses on differentiated cut-offs, this provided sufficiently large subsets in whom to perform both training and internal validation analyses. The large coverage in terms of associated general practices as well as geographical spread resulted in a dataset representative of (semi)urban, multiethnic, older primary care patients in western Europe. Another strength was our use of thorough statistical modeling techniques to determine the best fitting methods of detecting HF by use of NPs. Lastly, in determining an HF diagnosis we did not base this merely on structured data but also on free text available in the dataset. This resulted in all potential HF cases (whether flagged through structured or free text data) having been adjudicated by an expert panel.

A limitation in our statistical analyses was that the index and the reference standard were likely not independent. A registered HF diagnosis, our reference standard for diagnostic accuracy of the tested NPs, is based on a number of factors including NP measurement – the index test. Ideally, the index test and reference standard would have been independently assessed to prevent a number of biases that are a common in using retrospective datasets [40, 41]. A considerable number of cases had to be excluded due to missing unit of NP measurement, with potential for selection bias in case that missingness were to have been associated with certain local laboratories (and hence place of patients’ residence).

Further limitations were that diagnostic concordance between both NPs is not perfect, in which it is possible to have an elevated BNP while NT-proBNP is normal and vice versa [42]. A difference between BNP and NT-proBNP in the likelihood of being diagnosed is created if GPs are influenced by NP values to refer for further diagnostics. In our presentation of the AUPRC to assess diagnostic accuracy of BNP and NT-proBNP we saw a significantly increased AUPRC for NT-proBNP. To what extent these results are influenced by the higher HF prevalence among the sample tested with NT-proBNP, however, is uncertain. Finally, due to the absence of ICPC subcodes for type of HF as well as limitations on data granularity within free text data, we were unable to systematically make a distinction between HF with preserved, midrange or reduced ejection fraction among incident HF cases. For this reason insight in results on applicability could not be provided divided for each HF phenotype. Although the distinction would have given more comprehension on the contents of the cohort, this should not have limited our analyses on HF incidence of any type.

Future research

Our study indicated higher diagnostic accuracy from NT-proBNP over BNP for incident HF in routine primary care. These results were derived using IPTW analyses to adjust for differences in baseline characteristics between BNP and NT-proBNP groups. However, IPTW modeling only corrects for determinants measured in our dataset. Randomized prospective studies, aimed at balancing measured and unmeasured confounders, would be better positioned to compare the clinical relevance of diagnosing HF with either BNP or NT-proBNP in primary care. Such studies would also ideally involve uniform diagnostic work-up and endpoint adjudication, which were unable to be retrieved from the current dataset. Other future work could further assess the clinical benefit of using differentiated thresholds for NP testing based on univariate risk assessment, or multivariate through models such as TARGET-HF [13].