Using repeated antibody testing to minimize bias in estimates of prevalence and incidence of SARS-CoV-2 infection

Case definitions for OR, AND, and PSeq

All case definitions are included in Box 1. For the OR case definition we consider participants to be a positive case if they have either a positive IgM or a positive IgG response on a single test. This definition optimizes sensitivity. For the AND case definition we consider participants to be a positive case only if they have both a positive IgM and a positive IgG on the same test. This definition optimizes specificity. The OR and AND definitions use only parallel testing. We define PSeq using both parallel and sequential testing as:

IgM AND IgG on first test OR [(IgM OR IgG on first test) AND (IgM OR IgG on second test)]

The first condition uses the information contained in the first test in both IgG and IgM. The second condition deals with the case of having either one of IgG or IgM positive on the first test and a confirmatory IgG or IgM positive tests on the second. In this paper, we consider only two tests. The PSeq case definition can be extended to more than two tests.

Simulation

In this Section, we describe the design of a simulation study designed to evaluate and compare estimated prevalence based on PSeq with that of the OR and AND case definitions, and that of the OR with adjustment with respect to absolute bias and positive predictive value, across levels of true prevalence (π). We considered a sample size of n = 1,000 and 1,000 iterations. To compute prevalence based on the OR (π ^OR), AND (π ^AND) case definitions, and OR with adjustment (π ^aOR), we assumed that the vector containing the number for each combination of positive and negative IgG and IgM was sampled from a multinomial distribution with probabilities defined as in Eqs. (2)–(5) of the Supplementary Material. In other words, we considered a 2-by-2 table containing the number of negative tests for both IgG and IgM (n−−), IgG positive and IgM negative (n+−), IgG negative and IgM positive (n−+) and both positive (n++) (Table 1 in the Supplementary Material). For each iteration we then computed π ^OR and π ^AND as the proportion between the sum of n+−, n−+ and n++, and between n++ and the total number of participants n respectively. We computed π aOR = π OR + s p OR − 1 s e OR + s p OR − 1 where se ^OR and sp ^OR were defined as in Eqs. (8) and (9) in the Supplementary Material. To compute prevalence based on the PSeq case definition (π ^PSeq), we assumed the vector of positive and negative tests was sampled from a binomial distribution with probability π(se ^PSeq) + (1 − π)(1 − sp ^PSeq), where se ^PSeq and sp ^PSeq are defined in Eqs. (16)–(22) and (20) of the Supplementary Materials. Sensitivity and specificity of the first and second test for IgG and IgM were set equal to 0.8 and 0.9, respectively based roughly on actual test performance (Frederick National Laboratory for Cancer Research 2020). For our primary analysis, we considered the tests to be uncorrelated, i.e., we assumed conditional independence (Hui and Walter 1980). For each iteration, we computed π ^PSeq as the sum of positive tests divided by n. We considered 81 values of the true prevalence π, from 0.1 to 0.9 and computed absolute bias of π ^OR, π ^AND, π ^aOR, and π ^PSeq as the absolute difference between their means and the true value of prevalence. Positive and negative predictive values for OR, AND, and PSeq were calculated using formulas presented in Eqs. (10), (11), (14), (15) and (35) in the Supplementary Materials, respectively.

We further extended our simulation to represent the current gold standard in COVID diagnosis, Nucleic Acid Amplification Tests (NAATs). Using the same principles as for our antibody test simulation, we calculated the absolute bias, negative predictive value, and positive predictive value for a single test and for a repeated test approach, respectively. We used estimates of sensitivity (0.98) and specificity (0.97) for NAATs taken from the higher range of test performance in ideal testing situations (Hanson et al. 2023). Estimates for NAAT performance range widely and many in real world settings are substantially lower than those used here (Hanson et al. 2023). We do not provide a simulation of the impact of these case definitions on incidence, given that the additional assumptions would be substantial and may be depend on heterogeneity of risk in specific populations.

Real world case-study

In this Section, we describe an empirical application of our proposed case definition. We used data between April 16, 2020 and December 20, 2020 from the North Carolina COVID-19 Community Research Partnership study (NC-CRP), a health system-based longitudinal syndromic and sero-surveillance study (COVID-19 Community Research Partnership 2022; Munawar et al. 2021). Briefly, participants were initially recruited through participating healthcare systems and healthcare workers were oversampled. Inclusion required an active email address and data capable cell phone. Additional details on the NC-CRP study design can be found on the study website and in the design paper (HYPERLINK “http://www.covid19communitystudy.org/index.html” \o “http://www.covid19communitystudy.org/index.html” http://www.covid19communitystudy.org/index.html) (COVID-19 Community Research Partnership 2022).

Participants were tested for antibodies against SARS-CoV-2 using two different at-home lateral flow assays from Syntron and Innovita. Both tests use the Scanwell platform and test for IgM and IgG to both SARS-CoV-2 nucleocapsid and spike proteins. The sensitivity and specificity of the Syntron test were IgG 0.73 and 0.99, and IgM 0.93 and 0.975, respectively, based on test performance results from the National Cancer Institute (Frederick National Laboratory for Cancer Research 2021). Similarly, the sensitivity and specificity of the Innovita test were IgG 0.87 and 0.99, and IgM 0.90 and 0.99, respectively. For the primary analysis and unless otherwise specified we used the Syntron test performance characteristics. Because we only included tests through December 2020, which was before access to COVID-19 vaccinations was widespread, a positive test represents evidence of prior infection with SARS-CoV-2. From the initial cohort of 9,830 participants, 39 had an inconclusive test, and 3,859 had only one test. The final cohort was comprised of 5,932 participants with more than one test, ranging from 2 to 6, with a median of 4 tests. Sixty-one percent were female, 89 % were non-Hispanic white and 46, 41, and 13 % were of age 18–45, 46–65, and 65+, respectively. We estimated prevalence using four different approaches: π ^OR, π ^AND, π ^PSeq, and π ^aOR. The first three target the OR, AND, and PSeq case definition described above where prevalence is estimated by calculating the number of positive cases for each case definition and dividing them by 5,932 (the total cohort). The fourth approach, π ^aOR, estimates prevalence adjusting it by test characteristics. Confidence intervals were constructed by using the normal approximation (Blyth and Still 1983).

As a secondary analysis to estimate prevalence and its uncertainty while controlling for correlations between tests and uncertain sensitivity and specificity, we repeated our analysis using Bayesian models (Gelman and Carpenter 2020). Specifically, we chose a truncated 0–1 normal distribution with mean 0.73, 0.99, 0.93, 0.975 and standard deviation 0.01 as informative priors for sensitivity and specificity of IgG and IgM, respectively. We used the same priors for first and second tests. Also, we fitted models using priors based on the Innovita IgG and IgM performance, i.e., a truncated 0–1 normal distribution with mean 0.87 (sensitivity IgG), 0.99 (specificity IgG), 0.90 (sensitivity IgM), 0.99 (specificity IgM), and standard deviation 0.01. We chose a truncated 0–1 normal distribution with mean 0.50 and standard deviation 0.10 as informative prior for covariances between IgG and IgM and first and second tests. These values were chosen to reflect correlations around 0.1. Similarly as in our simulation scenarios, we assumed a multinomial model for the OR and AND case definitions and a binomial model for the PSeq case definition. The posteriors of prevalence distributions were computed using a Markov-chain Monte Carlo with 2,000 iterations and 4 chains. When we considered correlation, as opposed to π ^aOR for which we assumed conditional independence, adjusted prevalence estimation, π ^Adj, was obtained by solving Eqs. (2)–(5) in the Supplementary Material.

We also estimated cumulative incidence over calendar time and incidence rates for 2-month intervals. We excluded those who reported a COVID-19 diagnosis or previous positive test for SARS-CoV-2 at enrollment. The Kaplan–Meier estimator was used to estimate cumulative incidence for each case definition across calendar time. Follow-up was from study enrollment to date of event or date of last entry, whichever came first. Incidence rates were calculated as the ratio between number of events and person-years across calendar time. Person-years were calculated from the beginning of each 2-month interval until the date of event or interval. Intervals for incidence rates were as follows: 1st interval 4/6/2020 to 6/12/2020, 2nd interval 6/13/2020 to 8/8/2020, 3rd interval 8/9/2020 to 10/4/2020, and 4th interval 10/5/2020 to 11/30/2020 and estimates are presented per 10 person-years. R version 4.0.1 was used for all analyses (R Project for Statistical Computing, https://www.r-project.org) (R Core Team 2021).

Simulation results

As shown in Figure 1, π ^PSeq outperformed π ^OR and π ^AND with respect to absolute bias, positive predictive value, and negative predictive value across values of prevalence from 0.1 to 0.9. Absolute bias ranged from 0 to 0.3 depending on the prevalence with the highest bias for the OR definition at low prevalence and the highest bias for the AND definition at high prevalence. Absolute bias of π ^PSeq was similar to that of π ^aOR across prevalence values. Positive and negative predictive values are not calculable for the π ^aOR approach as it adjusts the group level prevalence without accounting for misclassification at the individual level. Using sensitivity and specificity of current gold standard NAATs, absolute bias ranged from 0 to 1.5 and repeated testing of lower performance tests still exhibited reduced bias compared to a single test across the range of prevalence (Figure 2). PPV and NPV for NAATs were also similarly poor at low and high prevalence respectively within the range concordant with current waves of transmission and in the context of reduced but more selective clinical testing.

Figure 1:

Absolute bias (left panel), positive predictive value (PPV – middle panel) and negative predictive value (NPV – right panel) of π ^OR, π ^AND, π ^aOR, and π ^PSeq from antibody tests (sensitivity = 0.80 and specificity = 0.90) and from a single NAAT test (sensitivity = 0.98 and specificity = 0.97) across values of SARS-CoV-2 prevalence from 0.1 to 0.9 under conditional independence from simulation model. Positive predictive values are provided only for π ^OR, π ^AND, π ^aOR, and π ^PSeq and NAAT.

Figure 2:

Kaplan–Meier estimates of cumulative incidence of SARS-CoV-2 in a subset of 5,932 participants from the NC-CRP study, North Carolina, April–December 2020, number at risk and cumulative number of events over calendar time using the OR, AND, and PSeq case definitions.

Real world case-study results

Table 1 shows that the highest and lowest estimated prevalences were obtained by using the π ^OR and π ^AND, respectively, e.g., π ̂ OR = 11.5 (95 % confidence interval: 9.2–13.7) while π ̂ AND = 3.6 (95 % confidence interval: 2.3–4.9). Estimates of π ^aOR and π ^PSeq, were between those of π ^OR and π ^AND. In addition, similar results were obtained using Bayesian models and accounting for correlation between tests (Table 2). Similarly as for prevalence estimation, Kaplan–Meier cumulative incidence based on the OR and AND case definitions was the highest and the lowest, respectively, with estimates based on the PSeq case definition falling in between (Figure 2). The overestimation of cumulative incidence based on the OR case definition at the beginning of the study period is due to the presence of false positives and that overestimation continues across the entire timeline. Incidence rates were higher for the OR definition than the PSeq definition across the entire timeline (Table 3). Similarly, the AND definition underestimated incidence rates compared to the PSeq definition during every interval.

Context

Some of the earliest estimates of SARS-CoV-2 prevalence in the US suffered from fatal methodological flaws and may have damaged the response to the pandemic by encouraging the public to embrace incorrect conclusions about its nature (Offord 2020). These attempts were followed by the elegant work of investigators who provided seroprevalence estimates in multiple countries (Bajema et al. 2021; Pollán et al. 2020; Stringhini et al. 2020). These authors make use of gold standard approaches to account for false positives and low positive predictive values, but overwhelmingly rely on a cross-sectional design with a single antibody test. Very few studies on SARS-CoV-2 with large sample sizes and broad recruitment have included repeated antibody testing in their design, and current estimates of active infection often rely on a single testing framework. Further, few studies investigating the risk factors for serious COVID outcomes or the impact of SARS-CoV-2 infection have attempted to address misclassification. Our findings speak to the importance of repeated testing and the improvement in prevalence and incidence estimates that can be achieved with a confirmatory test. Given that the positive and negative predictive values differ depending on the underlying prevalence, comparisons between groups will likely be underestimated using results from a single test. The bias is likely to be the highest when the difference between two groups is the greatest. While post hoc adjustment generally performs well for estimates of prevalence, caveats about the disconnect between the population of interest and the study sample used to estimate test characteristics have been previously noted (Accorsi et al. 2021; Takahashi, Greenhouse, and Rodríguez-Barraquer 2020). Moreover, similar approaches for measures of incidence or time-to-event analyses are lacking and may not be effective if they do not address misclassification at the individual level. This may prove particularly challenging in the context of increasing reinfection rates. In addition, test performance may vary among individuals, and thus extensions to individual-level misclassification approaches may be important. Our results and basic testing theory suggest that: 1. Additional use of single cross-sectional test samples will continue to generate biased incidence estimates and hinder prevention and control measures, and 2. The proposed approach will be just as successful at minimizing bias throughout the pandemic as it has been in the low prevalence setting of the early pandemic.

Repeated testing has already been implemented or recommended to improve accuracy in SARS-CoV-2 testing in a variety of ways: 1. To rule out false negatives even when using gold standard NAAT tests for diagnosis given the heterogeneity in test performance by timing since exposure or symptom onset, 2. For improved sensitivity of rapid antigen tests in both symptomatic and asymptomatic individuals (Soni et al. 2023), 3. To reduce false positives from screening of asymptomatic groups (Connor et al. 2022), and 4. To improve negative predictive value and rule out false negatives in high pretest probability settings when using rapid antigen tests at home for those with symptoms, exposure, or risk of infecting others (Centers for Disease Control 2022; United States Food & Drug Administration 2022). While these are just some examples, our results confirm the potential of a repeated testing approach to benefit clinical care, research, and policy across a wide range of contexts. Extending a repeated testing approach to more research studies and public health surveillance efforts would greatly benefit our understanding of this pandemic.

Limitations and strengths

The primary limitation of this work is that we do not have a gold standard test of SARS-CoV-2 infection with which to confirm our results. This issue is not unique to our study and continues to hamper the field as a whole; however, the misclassification problems we investigate here remain a serious concern for all test types, including those currently considered the gold standard. Second, not all Wake Forest NC-CRP participants had more than one antibody test and cadence between tests was highly variable. Further, those with and without more than one antibody test are demographically different based on differential timing of recruitment. The surveillance data from this cohort is likewise not informative with regards to ideal testing cadence or optimal testing protocols for the general population. Similarly, this study sample includes results obtained using two different tests as the test was changed to improve test performance. The main impact of this is on the post hoc adjustments. Given the limitations of the NC-CRP cohort as a data source, we extracted a subset of the NC-CRP as a clean cohort to best demonstrate the proposed approach. As such, the real world data we use for this analysis may not be fully generalizable to other populations, although we expect that the performance of these case definitions will not differ substantially across populations. Addressing other issues that impact prevalence and incidence estimation including sampling strategy and access to testing and care is beyond the scope of this paper. Finally, the improved estimates from our approach come at the expense of the cost and time required for participant follow-up and repeated testing.

Despite these limitations, our study used the illustrative example of SARS-CoV-2 seropositivity with antibody testing to demonstratesing the utility of a case definition that leverages repeated testing. With a combination of simulations and real world data, we demonstrate the importance of choosing a case definition that reduces bias across the range of prevalence as the COVID-19 pandemic progresses. While none of the methods used in this paper are new, the application of this approach during the COVID-19 pandemic is novel and it is expected that these results could impact our future public health response. By going back to basics and building on traditional testing theory, our proposed approach could be expanded to any type of testing to reduce the bias of SARS-CoV-2 prevalence and incidence estimates. This approach will also offer the benefit of addressing misclassification at the individual level. While post hoc prevalence adjustments that account for single test characteristics perform well, they are not available for adjusting estimates of incidence which are vulnerable to misclassification of both the numerator and denominator. Further, this approach can be applied to other types of test, and specifically may reduce the uncertainty around the ideal time frame for testing in many situations.