Technical and health governance aspects of the External Quality Assessment Scheme for the SARS-CoV-2 molecular tests: institutional experience performed in all clinical laboratories of a Regional Health Service

EQA design

The Sicilian EQA system provides a mandatory participation in the SARS-CoV-2 EQA for every public and private laboratory that intends performing SARS-CoV-2 molecular testing to patients. The number of participating laboratories has increased sharply since the beginning of the pandemic, and thus has given more strength to the scientific data in our study.

Out of 654 laboratories belonging to the regional health service only 62 were enrolled for the participation in the EQA scheme “Molecular Biology SARS-COV-2 – COVID437” (340). These laboratories were the ones authorized to perform such analysis for the Regional Health System and for COVID-19 case identification. The authorization was released from the Regional Government Office after a specific and public selection based on a particular, structural, technological and professional features follow: (1) the presence of a molecular biologist; (2) the presence of a geneticist medical doctor or microbiologist; (3) the presence of DNA/RNA automatic extraction platforms; (4) the presence of validated Real-Time PCR systems.

Participants

The results of the participants were submitted through the CRQ’s EQA Platform. By the use of the CRQ Platform, the laboratory can manage the entire EQA process, from laboratory personal data registration and EQA orders, to the evaluation of EQA results, statistical analysis and reporting. As soon as a laboratory has been registered on the Platform, it can access to the CRQ EQA, which offers an on-line catalogue and submits its own EQA orders in an e-commerce manner. The catalogue is mainly structured into various categories, each of which give full details of EQA. The details include specimen type, sample quantity, analyte list, etc. Information about the scheduling of samples shipment and related quality test deadlines are provided, as well. A laboratory is updated about the status of its orders and samples shipments, so it has to confirm samples delivery on-line before it can access EQAs results submitting process. As a first step of SARS-CoV-2 EQA, a laboratory has to specify for each tested gene of the virus, instruments, reagents and methods used for every analysis phase: extraction, amplification and detection. Instruments, reagents and methods are not entered manually, but selected by accessing the Platform catalogues. The latter are constantly updated by the CRQ, and at the request of the laboratory. Before a laboratory enters analysis results, it has to specify if the conditions of the received samples are valid, so that the CRQ can take it into consideration for the evaluation phase. Subsequently, laboratories must enter the results of the analysis for each tested gene, expressed also in terms of CT values, including a medical report with the overall results of each samples (i.e. positive or negative for COVID-19). As a proof of the analysis performed, laboratories need to upload the reports produced by the instruments used, from which the entered results can be deduced.

Preparation of simulated samples

Briefly, 7 EQA samples (SARS-COV-2 Wuhan wild type) were shipped to each participant laboratory. In order to fulfill the quality criteria and to overcome health and legal issues, the 7 test samples were subministrated to the laboratories in a blind and name-inverted way, given that each scheme could reflect the real conditions of the patient. Coronavirus positive samples deployed in the Extra EQA scheme “Molecular Biology SARS-COV-2 – COVID437” derive from lysates of cells which have been infected with coronavirus (SARS-CoV-2, HCoV OC43 or HCoV 229E) and contain heat inactivated virus. In particular, for the EQA1 round, the positive samples represent different dilution series of lysate of cells infected with SARS-CoV-2, (1:1,000; 1:1,000,000; 1:10,000; 1:100,000). Negative samples derive from (1) lysates of non-infected cells MRC-5 as specificity control; (2) lysates of HCoV OC43-infected cells (1:2,500) as specificity control; and (3) lysate of HCoV 229E-infected cells (1:2,500) as specificity control. For the EQA2 round the positive samples were dilution series of lysate of SARS-CoV-2 infected cells (1:50,000,000; 1:5,000,000; 1:500,000; 1:50,000). Negative samples derive from (1) lysates of non-infected cells MRC-5 as specificity control; (2) lysates of HCoV OC43-infected cells lysates (1:1,000) as specificity control; and (3) lysate of HCoV NL63-infected cells (1:1,000) as specificity control. For the EQA3 round the positive samples were dilution series of lysate of SARS-CoV-2 infected cells (1:75,000; 1:50,000; 1:7,500; 1:5,000). Negative samples derive from (1) lysates of non-infected cells MRC-5 as specificity control; (2) lysates of HCoV OC43-infected cells lysates (1:1,000) as specificity control; and (3) lysate of HCoV 229-E-infected cells (1:1,000) as specificity control.

Each panel bearing the COVID437 code consists of 7 samples labeled A – G and consist of cell suspensions negative or positive for SARS-CoV-2 (genome complete viral) inactivated by heat. Each aliquot volume of approximately 0.8 mL is sufficient for at least two uses in normal nucleic acid extraction procedures.

Before the distribution to the participating laboratories of the EQA scheme, the target value of a given EQA sample is also confirmed in three ways: (1) technical support structure (Quality Control and Chemical and Biological Risk – CQRC) of the CRQ (Quality Control and Chemical) provider through two different PCR-RT assay systems: TaqPath COVID-19 CE-IVD RT-PCR (Applied Biosystems, Waltham, MA, USA) and Diagnostic kit for SARS-CoV-2 Nucleic Acid (Real-time PCR) (Shanghai Kehua bio-engineering Co., Ltd, Xuhui District, Shanghai, China); (2) INSTAND Expert Laboratories; (3) through the outcomes of the EQA scheme “Virus Genome Detection – SARS-CoV-2” (340) of the INSTAND, provider ISO/IEC 17043 accredited [13].

The target value includes median from Ct/Cp/Cq/CN values, lowest reported Ct/Cp/Cq/CN value and highest reported Ct/Cp/Cq/CN value for each gene analyzed (E, N, Orf1a, Orf1ab, RdRp, S, and a not specified region). Other samples are used for the detection of SARS-CoV-2 by PCR.

Data collection and statistical analysis

Statistical analysis, evaluation of the participants and reporting are performed by the CRQ’s EQA Platform engine. The evaluation takes into account the overall entered result of each sample against target results. Warnings about received samples, if any, are considered in this phase. Furthermore, uploaded instrument reports are evaluated. Participants achieve a positive evaluation if the target results are fulfilled for all the samples and if instrument reports are regular. Therefore, the evaluation report is produced, containing all information about participants, quality tests, results, target and performance evaluation.

Participants

In the end a total of 62 previously authorized laboratories, 32 public and 30 private laboratories, equipped with different PCR systems validated, participated in three distribution rounds (EQA1/EQA2/EQA3) of the INSTAND Extra EQA scheme “Virus Genome Detection – SARS-CoV-2” (340). For SARS-CoV-2 molecular detection testing, the number of participating laboratories increased steadily: 41 registered laboratories in the first EQA round (EQA1), 54 in the second EQA round (EQA2) and 62 in the third EQA round (EQA3) (Figure 1).

Figure 1:

Positive performance evaluation: the flow diagram summarizes the performance evaluation results of sixty-two participating laboratories.

Overall SARS-CoV-2 molecular test performance

The blind work setting was necessary to avoid false laboratory results and so false reports by the participating authorized laboratories. The false certifications would have a devastating effect in the management of the pandemic, as the diagnosis for COVID-19, unlike what happens for many other diseases, is exclusively linked to the outcome of the laboratory investigation in molecular biology. The incorrect diagnosis for COVID-19 would also have important implications on vaccination cycles and the issuance of the green pass.

In the first EQA round conducted between April, 8th 2020 and April, 20th 2020, 30 laboratories out of 41 (73%) received a Positive Performance evaluation (Figure 2A) for the CRQ’s EQA scheme COVID437 (252 samples out of 287 processed (88%) received an acceptable evaluation).

Figure 2:

Overall performance of laboratories: the graphs show the average percentage of positive/negative evaluation of participating laboratories for each EQA rounds: (A) EQA1; (B) EQA2; (C) EQA3.

In the second EQA round conducted between July, 1st 2020 and July, 17th 2020, 35 laboratories out of 54 (65%) received a Positive Performance evaluation (Figure 2B) for the CRQ’s EQA scheme COVID437 (310 samples out of 416 processed (81%) received an acceptable evaluation). In the third EQA round conducted between November, 18th 2020 and November, 26th 2020, 54 laboratories out of 62 (87%) received a Positive Performance evaluation (Figure 2C) for the CRQ’s EQA scheme COVID437 (433 samples out of 455 processed (97%) received an acceptable evaluation). The laboratories that received a Negative Performance evaluation, (27% for EQA1, 35% for EQA2 and 13% for EQA3) (Figure 2A–C) have incorrectly interpreted the EQA exercise samples, either mistaking the interpretation of the instrumental data or the results compared to the expected outcomes (negative samples given as positive or positive samples given for negative).

The simulation test based on a real patient, carried out with the EQA program considered (7 samples, one of which was educational) and covered a total of 1,453 tests with 87.8, 74.5 and 95.2% agreement with the expected outcomes for the EQA1, EQA2 and EQA3 respectively (Figure 3A–C).

Figure 3:

Performance on single sample: the graphs show the percentage of correct result (agreement to target value) of the final interpretations for each EQA rounds: (A) EQA1; (B) EQA2; (C) EQA3.

The selected test genes were: E, N, N1, N2, S, Orf1ab and RdRp. A successful participation means the achievement of 100% correct results in all the 7 samples according to the target values. In the EQA1 round the success rate for each sample test was respectively 92.7, 82.9, 82.9, 90.2, 85.4, 87.8, and 92.7% (Supplementary Table S1). In the EQA2 round the success rate for each sample test was respectively 61.1, 92.6, 92.6, 57.4, 87.0, 85.2 and 88.9% (Supplementary Table S2) and In the EQA3 round the success rate for each sample test was respectively 100, 97, 100, 89, 97, 95 and 89% (Supplementary Table S3). Interestingly, the 75% performed the SARS-CoV-2 detection with success, while the 25% failed either in one or all the 7 samples test.

The failure is mainly due to errors during the analysis (58.6%) probably due to contaminations, not-specific detection, or an operator error. Other reasons (41.4%) including inhibition of internal control, lower results due to the limit of detection (LOD), and incomplete results such as less of 7 EQA samples analyzed.

Nucleic acid extraction methods test performance

A total of 55 different nucleic acid extraction KIT (Table 1) were used. The most frequently used systems were QIAamp DSP Virus Spin Kit (QIAGEN Pty Ltd, Clayton, Australia), Magrev Viral DNA/RNA Extraction Kit (Anatolia GeneWorks, Sultanbeyli/İstanbul, Turkey) and STARMag 96x4 Universal Cartridge Kit (Seegene Inc, Walnut Creek, CA, USA), Xpert Xpress SARS-CoV- 2 (Cepheid, Sunnyvale, CA, USA).

Detection assays

A total of 55 different amplification SARS-CoV-2 kit platforms (Table 2) were used The most frequently used system were Allplex 2019 n-CoV Assay (Seegene Inc. Walnut Creek, CA, USA), TaqPath COVID-19 (Thermo Fisher Scientific, Waltham, MA, USA), Novel Coronavirus (2019-nCoV) Real Time RT-PCR Kit (Shanghai ZJ Bio-Tech Co, Shanghai, China), and COVID-19 HT Screen (CLONIT srl, Siziano PV, Italy). It is important to underline that, among the different PCR systems used by the participating laboratories, there was one PCR assay, VitaPCRTM SARS-CoV-2 from Menarini Diagnostics (Ripoli, FI, Italy), with the absence of an RNA extraction separated step, has demonstrated to be less sensitive than the other tests. Indeed, the laboratories that used this particular test have completely failed the EQA1 and EQA2 rounds, giving an indeterminate result. Only two participants among 62 used this direct assay, but considering that the direct test was used also in emergency areas and islands, in order to avoid report inhomogeneity, subsequently, a specific EQA scheme was implemented for this type of assay.