Comparison of efficacy and reliability of six commercial COVID-19 diagnostic PCR kits

Introduction

The coronavirus disease 2019 (COVID-19) pandemic, which emerged in Wuhan, China, in 2019, was the largest pandemic of the last century regarding its spread and the number of people infected. All countries have been severely affected by the pandemic, which has strained their health systems and impacted the economy and workforce. Therefore, various measurement methods have been implemented to control this disease and prevent its spread. One of these measures is early diagnosis of COVID-19. Probe-based real-time polymerase chain reaction is the gold standard for disease diagnosis [1]. Reverse transcriptase-polymerase chain reaction (RT-PCR) is preferred because of its advantages in early-stage disease detection, high sensitivity, high specificity, and ease of use. Since the pandemic outbreak, more than 250 known COVID-19 PCR kits have been developed. All these kits use cycle threshold (Ct) values to determine the patient’s viral load, which provides information for detecting severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2). The Ct value is a quantitative PCR measure used to calculate the initial concentration or copy number of a target nucleic acid sequence in a sample. The Ct value represents the relative amount compared with the standard reference sample. By comparing the Ct values of different samples, users can evaluate the copy numbers of the samples. The Ct value was calculated starting with a known amount of the target sequence and then monitoring the increase in fluorescence (called the relative fluorescence unit (RFU)) as the increased number of amplifications. The RFU is an important measurement used to assess the intensity of fluorescence produced when a sample is amplified and provides a valid relative reference point to determine the concentration and abundance of target sequences. The Ct value is the point at which the fluorescence increases above the background level. In general, the lower the Ct value, the higher the abundance of the target sequence in the sample and, therefore, the faster the amplification of the target sequence. Some studies have claimed that SARS-CoV-2 has no actual genome and that RT-PCR testing can be misleading [2]. Although these claims are unfounded, most newly produced SARS-CoV-2 diagnostic kits show inconsistent performance, contradicting lung tomography findings. Patients with severe infection without pulmonary tomography findings may have negative PCR results, increasing the possibility that COVID-19-positive cases are causing viral spread. The inconsistent performance of COVID-19 RT-PCR kits may be due to the pandemic’s rapid evolution and the disease’s spread. Therefore, some kits’ validation, consistency, and optimization tests were performed directly in the field. These events require a comprehensive comparison of the results between kits to select and improve the best one [3].

SARS-CoV-2 is a positive-stranded RNA (+ssRNA) virus with a single open reading frame called ORF1ab that encodes non-structural proteins involved in viral RNA transcription, replication, and immune evasion. One of these non-structural proteins is RNA-dependent RNA polymerase. Replication and transcription also occur within the ORF1ab framework. The structural proteins of SARS-CoV-2, including the membrane (M), envelope (E), nucleocapsid (N), and spike (S) proteins, are expressed through the production of subgenomic messenger RNAs. ORF1ab, E, N, and S genes are the most commonly used primer targets for detecting SARS-CoV-2 [4, 5]. The E gene target region is used to detect SARS-CoV-2 and genetically similar viruses (pan-Sarbecovirus) for general screening, whereas the ORF1ab target region and N gene specifically identify SARS-CoV-2. Some kits use more than one primer to detect multiple SARS-CoV-2 genes simultaneously. Housekeeping genes, such as RNase P (ribonuclease P) and GAPDH (glyceraldehyde 3-phosphate dehydrogenase), which are highly expressed in human epithelial tissues, were used as internal controls. In addition, unlike classical PCR methods, current SARS-CoV-2 detection kits use probe-based primers. The number of nucleic acids copied during the reaction was measured in real-time using fluorescence irradiation in each cycle. Probe-based RT-PCR can simultaneously detect multiple targets in a sample. However, this requires the design of target-specific probes in addition to primers. Several probe designs are available; however, hydrolysis probes are the most commonly used. The hydrolysis probe comprised two primers and one probe. The probes were designed to complement the target sequences. Each probe contained a fluorophore (donor) and a quencher (acceptor) at each end. During the annealing step of PCR, the probe binds to a specific target sequence. However, no fluorescent signals are detected because the fluorophores and quencher are close. In the next step, as an extension, the probe is hydrolyzed by the 5–3′ exonuclease activity of the polymerase. Probe hydrolysis separates the fluorophore from the quencher, increasing amplification-dependent fluorescence. Thus, the fluorescent signal from RT-PCR was proportional to the amount of the probe-target sequence present in the sample. Probe-based RT-PCR is more specific than dye-based RT-PCR and is widely used for diagnosis [6]. Currently, kits used to detect SARS-CoV-2 in many countries are probe-based. Depending on the developer’s and financial infrastructure’s initiatives, more than one probe may be used to detect viral genes. Specific primers and labeled probes were used to detect the virus’s conserved ORF1ab and N genes, and RNase I was used as the internal control.

In this study, we compared six commercial kits using 244 patients. Ct values and patient viral loads were compared. We also compared the compatibility of the kits based on the accuracy of the strongly positive (Ct<30), weakly positive (Ct<30), and negative results.

Materials and methods

Results

In this study, nasopharyngeal aspirate/lavage/swab specimens from 244 individuals collected at the PCR-COVID-19 laboratory of TOBB University of Economy and Technology Hospital, Ankara, were analyzed using six different SARS-CoV-2 detection kits. The KrosGen kit is routinely used to detect SARS-CoV-2 infection in patients. All patients with positive PCR results were followed up and examined by a specialist for associated symptoms. The stage of SARS-CoV-2 (whether the patients were in the early, acute, or late phases of the disease) was also monitored. Details of the primers, reaction protocols, and thermal programs used for all six commercial kits are listed in Table 1.

Table 2 shows three of the five kits (Bioexen, Coronex, and Anatolia) showed consistent results for Ct<30 compared with KrosGen. Conversely, for Ct≥30, the consistency of the kits fluctuated with positive results. For Ct≥30, all patients were asked if they had symptoms, and positive results were verified. In addition, all positive samples for Ct≥30 were diluted by 1:2 to eliminate false positives, and the reaction was repeated. The result was considered positive if the second reaction was also positive; otherwise, a new sample was requested from the patient the following day. Table 2 shows four of the five kits (Bioexen, Polgen, Coronex, and Diagen) had higher false negative results than the reference kit (KrosGen). As stated in the manufacturer’s instructions, the evaluation of the positive results of the Polgen kit with the detection of one of the targets showed that the false positive percentage decreased. Only the Coronex kit showed a high false positive rate (87.1 %). The Anatolia kit showed the highest kit concordance for detecting COVID-19 compared with the KrosGen kit.

The number and percentage of positive and negative samples, positive predicted value, negative predictive value, positive likelihood ratio (LR+), negative likelihood ratio (LR−), accuracy rate, and Youden index of the KrosGen kit are shown in Table 3. When comparing the target genes used in the six commercial kits, we found that all the kits used RNase P as an internal control. However, each kit uses a different target gene to detect SARS-CoV-2. The KrosGen and Polgen kits use the N1 and N2 genes for detection. KrosGen used both in a single channel, and the presence of either one gave a positive result. However, in the Polgen kit, N1 amplification was measured in 6-carboxyfluorescein (FAM), while N2 was measured in hexachlorocarboxyfluorescein (HEX); for a positive result, both N1 and N2 genes should be detected according to the kit data sheet. Nevertheless, as shown in Table 3, the simultaneous amplification of N1 and N2 decreased the consistency of the Polgen kit to 3.3 %. However, assuming that the results of the Polgen kit were evaluated in such a way that the amplification of either N1 or N2 was sufficient to achieve a positive result, the compatibility of the kit increased to 54.4 %. The Coronex and Anatolia kits were designed to detect ORF1ab and N (N1 and N2) genes in the same channel (FAM), indicating that detecting one of the target genes was sufficient to produce positive results. Although the Coronex and Anatolia kits used three target genes for viral detection, the accuracy rate of the Coronex kit (70.3) was lower than that of the Anatolia kit (87.2). In contrast, Diagen uses more than one primer (ORF1ab in the HEX and S genes in the FAM channels) to detect SARS-CoV-2, and detecting only these target genes is sufficient to achieve a positive result. However, 96 of 244 samples (39.3 %) could not be evaluated. Bioexen was designed to detect only ORF1ab, and the compatibility of the kit for detecting positive results was 65.0 %.

The Ct values of positive results for both strong positive (Ct<30) and low positive (Ct≥30) were calculated, and the Ct intervals, means, and medians are given in Table 4. The KrosGen kit had the lowest detection Ct compared with the other kits for Ct<30. In contrast, the Polgen kit (in the hex channel) only detected positive samples, with Ct intervals between 28.58 and 31.08. The mean and range of the Ct values for the other kits were similar.

Discussion

RT-PCR technique has been widely used worldwide to diagnose COVID-19 during the pandemic. This method detects viral RNA during the diagnosis of COVID-19. In basic life sciences, this method detects small amounts of nucleic acids [7, 8]. Currently, it is important to diagnose COVID-19 quickly and accurately. Various COVID-19 diagnostic kits have been used. However, it was observed that different COVID-19 diagnostic kits sometimes gave different results. There are several reasons for these different results; however, they remain controversial [9].

The first reason is the difference in the test methods used. Different manufacturers of COVID-19 diagnostic kits use various testing methods. Some kits are antibody-based and aim to detect the disease by detecting COVID-19 antibodies produced in the body [10]. Other kits use PCR to detect the genetic material of the virus. These methods are based on different principles and can give different results [11, 12].

The second reason is the quality and accuracy of the kits. There may be differences in the production and quality of the COVID-19 diagnostic kits. Certain kits provide accurate results by identifying sensitive and specific antibodies or genetic targets. However, some kits may give false positive or false negative results [13, 14]. This affects the accuracy of the test. Despite their widespread use in diagnosing COVID-19, there is speculation regarding their reliability. However, this technique is highly sensitive, specific, and preferred for pandemic conditions such as COVID-19. Various diagnostic kits have been developed for this technique in laboratories and R&D centers. The diversity of reagents specially designed for each diagnostic kit, the variety of protocols and analytical methods, and the fact that not every test is performed according to standardized procedures have raised doubts regarding the accuracy and reliability of RT-PCR for the diagnosis of COVID-19. Therefore, it was essential to compare the detection rates of the two kits. Many studies in the literature have compared commercial kits. For example, six different kits were compared for the first time in China [2]. This study found differences between samples with low and high viral loads. In another study, seven different kits were compared, and all showed an efficacy of over 96 %, indicating that these kits could be used for diagnosis [4]. Another study compared two locally manufactured kits and found them highly compatible, suggesting they were safe for diagnosis [15].

Each kit reported in the literature has its own Ct interval values for detecting COVID-19, which may differ between kits. Although the Centers for Disease Control and Prevention accepts the detection of Ct<40 as positive in some diagnostic kits, many kits used in our study and available on the market evaluate amplification results up to 35 as positive. Reactions with Ct values above 35 should be considered erroneous, and artifact binding and Taq polymerase may be inactivated. Failure to account for these factors significantly affected the results. Patients who tested positive in our study had a Ct value <35. However, a comparative study of the different kits revealed that the kits had different Ct values. Problems such as storage conditions, the timing of clinical specimen collection (transfer of specimens to the laboratory or improper specimen collection), and contamination may also affect the positivity of the kits. Therefore, further studies are needed to measure the reliability of different kits.

The third reason is the difference in sampling and processing methods. For COVID-19 diagnostic kits, sampling and processing must be performed correctly to obtain accurate results. For example, incorrect nasal or throat swabs may yield false positive results. In addition, improper storage or handling of the specimens can affect the results. This possibility was eliminated in our study by selecting patient samples, which was also supported by positive clinical findings.

The fourth reason is variations in the virus. Viral mutations and variants emerged during the COVID-19 pandemic. Some COVID-19 diagnostic kits better detect certain variants, whereas others are less sensitive. This could have led to different results. However, this possibility was not assessed because the studies were performed simultaneously on samples from the same patients (Table 1).

In this study, five of the six kits have reliable compatibility at Ct<30, but this decreases at Ct≥30. When the viral load is high, it is easier to detect the presence of viruses in the nasopharyngeal aspirate/lavage/swab specimens. However, detecting positive results in individuals recently infected with SARS-CoV-2 with a low viral load is critical. Therefore, it is important to evaluate the performance of these kits in reduced viral load.

One of the major differences among the kits is the target gene region used for primer design. The minor differences among the kits may be due to the primers used for the target gene region. These are the regions of the virus used to determine the genes, especially in positive samples. Each commercial kit uses primers that amplify at least one gene region. The most commonly used gene regions for these kits are ORF1ab, N, and S. Our study showed that utilizing one or more gene regions in the same kit did not affect the accuracy. For example, although a single gene region (ORF1ab) was used in the highly reliable Bioexen kit, primers for two gene regions were used in the Diagen kit. However, the results showed that the Bioexen was more consistent than the Diagen. The same conclusion applies to the Polgen and Coronex kits.

Fluorophores are used in the TaqMan probe-based kits. The most common fluorophores used in COVID-19 diagnostic kits are FAM and HEX, which contain fluorescent reporters and quenching dyes. In probe-based assays, the probe must be designed to be approximately 10 °C higher than the primer binding temperature to ensure correct binding. Therefore, long probes have been designed to detect regions with low cytosine-guanine (GC) ratios. However, if the probes are too long, unnecessary background radiation will occur [16]. This phenomenon is a possible reason for the unwanted radiation observed with Diagen and Polgen kits. We conclude that the GC ratios of primers, probes, primer annealing temperatures, and probe selection should be carefully determined when designing kits. The more accurate the combination, the more reliable the kit is. Although using more than one fluorophore is advantageous, optimization can be challenging [17]. Multiple fluorophores can lead to misdiagnosis of the disease, especially in pandemic conditions, and to a loss of confidence in kits, where most R&D optimizations are performed in the field.

Three-dimensional (3D) modeling of the proteins encoded in the viral genome was performed to determine whether the stability or unstable nature of the proteins encoded by the gene regions used in the kits affected the reliability of the preferred target regions. PHYRE2 database (http://www.sbg.bio.ic.ac.uk/∼phyre2/html/page.cgi?id=index) was used to determine whether there is a relationship between the stability of the proteins targeted by the primers used in the kits and the reliability of the kits. A visual comparison of the results, in which 3D modeling was performed using homology-based approaches, showed that the proteins with the most stable structures belonged to the ORF1ab (ORFa and ORFb) regions. Therefore, kits (Polgen, Diagen, Coronex) using primers from the S and N regions had lower reliability, and the protein models were more unstable (Figure 1). Although KrosGen and Polgen used the same gene regions for positive detection, KrosGen provided more consistent results than Polgen. When comparing these two kits, there was a 10 °C difference in primer annealing temperatures (Table 1). The fact that the Polgen kit uses two different fluorophores and has a 10 °C higher annealing temperature than the KrosGen suggests two possibilities. One is the presence of various fluorophores, which may affect the irradiation. Another possibility could be differences in the GC ratio of primer candidates belonging to the same gene region. It would be better for kit manufacturers to share their primer sequences. Another difference between the two kits is the ability to evaluate gene amplification in one or two distinct channels. As shown in Tables 2 and 3, the kits assess the presence of COVID-19 in one channel (using one or two genes) exhibited a higher compatibility and accuracy rate than those evaluating more than one gene in two different channels. Kits that used two genes in one channel to detect COVID-19 provided results with the highest accuracy rates and compatibility.

Figure 1:

SARS-CoV-2 detection kits_RT-PCR_COVID-19.

Conclusions

One reason for the uncontrolled global spread of COVID-19 is the false negative test results in patients with a low COVID-19 viral load. Therefore, a sensitive and suitable kit with high compatibility for Ct≥30 is essential to identify patients with a low viral load. This study compared six commercial COVID-19 detection kits for patients with low viral loads. We found that the KrosGen, Anatolia, and Bioexen kits were the most effective in detecting SARS-CoV-2.

This study has several limitations. One of these limitations is the lack of a gold standard, such as viral culture and/or international standards. Viral culture can only be performed in a Biosafety Level 3 laboratory requiring special conditions. There are fewer than 10 BSL3 laboratories in Turkey. Although a World Health Organization international standard exists, it is difficult to obtain with a limited budget. These kit manufacturers are also expected to use this standard to validate and verify their kits. However, several studies have compared kits that do not use this standard [4, 5].

The second limitation is related to the sample size of the Anatolia kit. The Anatolia kit did not reach the researchers until the final phase of the study, leaving insufficient time and resources for thorough examination with a representative sample size. This unexpected delay in acquiring the Anatolia kit hampered our ability to conduct a comprehensive analysis, as originally intended. Nevertheless, given that our sample size fell below the threshold set determined in our power analysis, we felt it was imperative to include these findings in our study because of their remarkable consistency compared with other kits, as they are more in line with our research findings.

In conclusion, to improve the accuracy and compatibility of kits, we suggest using primers to detect two genes with a single probe evaluated in a single channel. In addition, for those samples with Ct≥30, it is better to repeat the reactions with 1:1 and, if needed, with 1:9 dilutions. A consistently positive result was considered reliable. However, in cases where the results are inconsistent, it is advisable to repeat the reaction using a new sample from the patient. Nevertheless, it is important to keep in mind that regardless of the brand of the kit, a validation study should be performed before routine use of the kit for COVID-19 detection. Finally, it is strongly recommended that interlaboratory comparisons and external quality control measures validate the results of kit reactions.