National External Quality Assessment follow-up: 2010–2017 Turkish experience

Introduction

Medical laboratories play a central role in the improvement of the healthcare system with regard to patient safety and a healthy society. They influence approximately 70% of the healthcare system both in risk analysis and in the diagnosis, treatment and follow-up process [1], [2], [3].

Important decisions have been made in recent years regarding international healthcare practices. The United Nations has determined it suitable for changes to be made that actively address healthcare needs of nations (http://www.who.int/mdg/publications/mdg_report/en/, http://www.who.int/mdg/publications/MDG_Report_08_2005.pdf). It is important in all of these objectives to ensure the active role of laboratories in this system. Non-profit organizations, such as International Federation of Clinical Chemistry and Laboratory Medicine (IFCC), International Society of Haematology (ISH), European Federation of Clinical Chemistry and Laboratory Medicine (EFLM), Clinical and Laboratory Standards Institute (CLSI), International Laboratory Accreditation Cooperation (ILAC) or national bodies; College of American Pathologists (CAP), German Federal Medical Council, set effective performance characteristics for diagnostic tests to ensure reliable, traceable and comparable laboratory test results. An improvement in the quality of laboratory procedures is essential for the improvement of the quality of the healthcare system. Modern laboratory quality management is a process [4], [5]. Among the prominent analytical tools of this process are quality control practices.

Briefly, quality control (QC) applications are error detection procedures that assess precision and accuracy studies of the analytical period, as well as systemic problems, environmental conditions, and personnel performance. They are the methods used to monitor and determine whether the service carries predefined features and how reliable it is.

QC applications made during the analytical period are an important and indispensable part of the control processes in Total Quality Management. Nevertheless, it should be taken into account that, this data alone will not be a sufficient indicator in the evaluation of quality.

Although there are different methods of implementation in recent years, there are basically two types of quality control applications. These are;

1. Internal Quality Control (IQC): Essentially precision work done within the laboratory with samples of known or unknown values.

2. External Quality Assessment (EQA): Work done externally to the laboratory by a competent independent organization with more focus on accuracy and educational aspects. When the EQA is administered by governments, it is also referred to as the “Proficiency Testing”.

IQC and EQA are regarded as one of the indicators of the laboratory operation procedure in the analytical period, interlocked with the other factors. Significant studies have been conducted and published, especially IFCC, on the quality indicators belonging to the analytical period. In addition to that, IFCC strongly recommends the use of EQA practices [6].

It is aimed to compare the analytical performances of medical laboratories with the performances of other medical laboratories at national and international scale within the scope of EQA program. EQA is often confused with qualification tests.

EQA provides important contributions to medical laboratories [7], [8], [9], [10], [11], [12], [13], [14], [15], [16], [17], [18], [19], [20], [21].

1. EQA can evaluate traceability to reference systems and harmonization between test procedures,

2. Bias is one of the most appropriate approaches to systematic error assessment,

3. Provides information on reagent, calibrator, method, device and personnel and helps to identify and evaluate errors,

4. Provides important information on method and device differences and evaluations,

5. It is an important laboratory training tool,

6. It is necessary in cases where accreditation and national requirements exist.

However, there are some limitations to it;

1. EQA results alone don’t show the quality of laboratory.

2. It doesn’t provide information about the quality of the pre- and post-analysis period. There are also limitations for the analytical period (e.g. not including sample preparation section).

3. It is difficult to apply corrective action to the past if a problem is identified.

4. In very few EQA programs, the target value for a limited number of analytes is determined by the “Reference Method”. Assessment by peer group outcome may lead to limitations in standardization studies (especially for analytes that are traceable, reference method and material).

5. Random errors are also important in EQA programs. Some control programs use between 2 and 5 different level control materials during the same period. In this case, systematic or random errors are partially understood. In this case, however, the control intervals are extended or the cost may increase.

Medical laboratories in Turkey provide their services in the scope of Law No. 992 and Implementation Regulation which entered into force in 1927 (Seriri Taharriyat Ve Tahlilat Yapılan Ve Masli Teamüller Aranılan Umuma Mahsus Bakteriyoloji ve Kimya Laboratuvarları Kanunu; Kanun Numarası: 992, Kabul Tarihi: 19/03/1927, Yayımlandığı Resmi Gazete Tarihi: 30/03/1927, Yayımlandığı Resmi Gazete Sayı: 580).

Under the Medical Laboratories Regulation, all medical biochemistry laboratories in our country were obliged to participate in the EQA Program conducted by an independent organization for a limited number of tests (24 tests in total) in order to increase the reliability of the test results and to use the monitoring system in 2010 to record EQA results. With the help of EQA data from the laboratories, monitored by the Ministry of Health, their performance is assessed, and through these outputs, national standardization and harmonization studies are planned.

Currently, over 20 commercial EQA programs are used in Turkey, two of which are conducted domestically. These programs show significant differences in terms of number of participating laboratories, outcome evaluation criteria, type of control material, frequency and outcome formats.

The main purpose of this study is to evaluate the outputs of the country’s EQA monitoring system between 2010 and 2017, compare the international data and determine the new targets by processing the big data.

Materials and methods

Within the scope of this study, 801,028 data items belonging to 1941 laboratories between 2010 and 2017 were studied. In this scope, while EQA data for alanin aminotransferase (ALT), albumin, alkaline phosphatase (ALP), aspartate aminotransferase (AST), glucose, HDL cholesterol, inorganic phosphorus, chloride (Cl), cholesterol, creatinine, lactate dehydrogenase (LDH), potassium (K), sodium (Na), total protein, triglyceride, and urea were entered into the system; in the year 2013 HbA1c and in 2014, amylase, direct bilirubin, gamma glutamyl transferase (GGT), calcium, creatine kinase, total bilirubin, uric acid tests were included within these parameters.

The results of the 18 commercial EQA companies were entered into the Ministry of Health EQA monitoring system. Association of Clinical Biochemistry Experts External Quality Control Program (KBUDEK, Turkey), LabPT Quality Control Program (Turkey), RANDOX International Quality Assessment Scheme (RIQAS, England), BIO RAD External Quality Assurance Services (EQAS, CA, USA), Bio-Development (BIO DEV, Milano, Italy), Bio Group Medical System (Italy), Medical Laboratory Evaluation (MLE, USA), American Academy of Family Physicians – Proficiency Testing, (AAFP-PT, Kansas, USA), College of American Pathologists Proficiency Testing (CAP, USA), Labquality EQAS (Finland), INSTAND (Germany), Reference Institute for Bioanalytics (RfB, Germany), AccuTest Proficiency Testing Services (USA), Digital PT-Oneworld Accuracy (Canada), United Kingdom National External Quality Assessment Service (UK-NEQAS, England), Wales External Quality Assurance Scheme (WEQAS, England), The European Society for External Quality Assessment (ESfEQA, Germany), NOBIS Quality System (Romania) EQA program results were evaluated. Only two of these programs are EQA programs in Turkey [22].

EQA result evaluations, by taking CLSI and ISO 15189 into consideration during the initial stage, are conducted according to the SDI data of the evaluation program. Each laboratory includes the EQA program, test name, test unit, test period, self result, peer group mean value, peer group SD (Standard Deviation) and Standard Deviation Index (SDI) in the system. The SDI>3 test results are defined as unsatisfactory and it was requested that these results be evaluated and recorded for nonconformities, in SDI calculation;

“SDI=Laboratory result – Peer group target value/peer group standard deviation” formula is used. Some EQA programs use the values of the total group instead of the peer group.

The coefficient of variation (CV%) is calculated as follows CV%: 100×(standard deviation)/(expected value).

Excel and STATISTICA 12 packaged programs were used for the statistical evaluation. The first evaluations are carried out by using descriptive statistics (mean, SD and 95% CI). ANOVA test was used for the group comparisons and Bonferroni correction was performed. The significance value of p was set at 0.01. As a result of the big data size for the sake of SD changes, more than 10% change rate was considered clinically significant (Table 1).

Results and discussion

Between 2010 and 2017, a total of 801,028 data entrance was conducted for 24 separate tests, then unsatisfactory performance was detected on 27,074 samples (3.5%) (Table 2).

Depending on the years, the number of participating medical laboratories and the number of tests have increased. While 10 tests and 48,278 outcomes were evaluated in 2010, 172,446 outcomes were evaluated for a total of 24 tests in 2017. When the total nonconformity ratios are examined, it has been found that the nonconformity ratios decrease with time. According to the initial year of nonconformity rates, the figure was 47.4% in 2016 but decreased to 41% (p<0.001, Figure 1) in 2017, respectively. When we compare these ratios with literature data, it is observed to be high [23], [24], [25]. Literature data are generally <1%. However, it should not be forgotten that these evaluations are based on our three SDI and others are evaluated using total error.

Figure 1:

Unsatisfactory performance by years during EQA process (top) and error reduction rates (bottom).

Within the frame of Republic of Turkey Ministry of Health EQA study, there are 18 different EQA programs. These programs differ from each other in terms of sample type, assessment algorithms, number of participants and features. In Table 3, you will see the nonconformity rates according to five EQA programs, which include the highest number of participants. According to the programs, it is observed that both the nonconformity ratios and the distributions of CV% values of the programs are significantly different. Particularly in the EQA programs with high distribution, the nonconformity ratios are low as expected (Program 1 and 5). This might be due to various reasons. However, that these two programs have wide dispersions (5.8% and 9.6%, respectively) is important. Since the EQA makes evaluations according to the SDI data, it is an important finding that the nonconformity ratios are low in EQA programs. In EQA programs evaluated by RILIBAK and CLIA, total error is used for assessments. With the help of an additional assessment to be conducted through the national total error limits, in addition to the SDI assessments, Turkey will minimize this difference. Besides, at least for certain analytes, a total error limit based on biological variation data might be used [26].

Within the framework of the EQA monitoring system, the reasons for unconformity between the years 2015 and 2017 have begun to be determined. The data for the causes of samples that are inappropriate according to the results of EQA application are presented in Table 4.

As can be seen here, a significant part of the reasons for unconformity are observed as the inability to identify the problem with data entry errors. However, users state that these conditions do not reflect on patient outcomes.

In general, it is quite difficult to create a retrospective action plan in EQA programs. This is a major problem in all programs. However, the “inability to identify the problem” in our country is higher than international data. In the literature, it is observed that the main problems in EQA studies are dilution problems, device coding errors, device errors and calibration problems. The result rate of unidentified EQA is 19–24% [23], [24], [25], [27]. The reason for this high rate may be the inability to carry out the cause and effect relation on the persons participating in the program. This situation should be considered as an important outcome.

Another important problem is data entries for cause and effect relation that are not recorded into the system by the laboratories. Generally, it has been determined that about 40% of the participants do not record “causes of unconformity” into the system. This is one of the most important outputs of the study. Therefore, efforts have been initiated to make it obligatory to record causes of unconformity into the system.

Although the collection of data began in 2010, the evaluation of system data has been improved between 2014 and 2017, depending on the software updates made over the years in the EQA monitoring system. The distribution data obtained for each of these years is presented in Table 1.

In particular, it has been determined that the range of distribution between programs is decreasing each year and the results are closer to each other. Only GGT, direct and total bilirubin values were not decreased, but all other tests showed a significant decrease. On average, a decrease of 15.7% was detected. This improvement is presented in detail in the literature [28].

Study outputs

1. EQA monitoring system organized by the Republic of Turkey Ministry of Health has made a contribution to increasing the awareness regarding analytical period in laboratories. It is identified that there has been a decrease in discordance and variations regarding analytical period. It was evaluated that it would be appropriate to continue the study by expanding it.

2. Having too many different programs may cause problems in evaluating.

3. Problems have been detected by evaluating the discordances with SDI. It has been evaluated that adding the total error and evaluation to the system will contribute to harmonization. The EQA participant results should be evaluated against agreed limits. These limits have been agreed by either professional organizations, authorities or suggested by the EQA organizers [29]. In addition, the total of the allowable error and precision values were determined for 16 test in Turkey [30].

4. Inadequacies have been identified in assessing EQA discordances and in forming preventive activities. It has been decided to plan training activities related to the subject.

Limitations of study

1. Analytical systems, control materials and EQA programs frequently change because of the frequent repetition of procurement processes in our country (1–3 years). For this reason, the study was not performed as an in-vitro evaluation.

2. For similar reasons, the lot numbers of the control materials change very frequently, which is a factor that increases the variability.

3. In 2013, limited amount of data was reached due to the software change.

4. It is important that almost half of the discordances consist of random errors that are undeterminable. This problem was mainly attributed to the high SDI values of some of the programs due to the low SD values and it was considered acceptable. It has been evaluated that in the following years this ratio can be reduced because total error will be used as well.

5. There is significant heterogeneity among the EQA programs. There are differences in the form of evaluation (formulations, target values, peer groups, participant numbers etc.), cycle intervals (between 2 weeks and 4 months), transfer types, material type differences (liquid and lyophilized), commutability. These factors might increase the variations. However, evaluation has made important contributions in this form.

6. No information, reasoning or justification was provided for almost 40% of the cases where the laboratory results were discordant with EQC expectations. The necessary training programs have been started. Assessment studies will continue.