Extra-analytical quality indicators – where to now?

Introduction

According to the famous aphorism by Galileo Galilei, “measure what is measurable and make measurable what is not so”, measurement is widely considered the key tool for reducing medical errors and enhancing patient safety [1], and the use of quality indicators (QIs) in laboratory medicine, in particular, is the starting point of programs designed to improve the quality of laboratory services in the total testing process (TTP). The value of adequately managed and identified healthcare QIs in assessing and monitoring laboratory performances has, in fact, already been demonstrated [2]. In the last few years, awareness has been raised about the use of QIs in laboratory medicine as a valuable tool in ensuring reliable decision making and providing appropriate patient care. This has prompted several programs on QIs, organized and implemented over time in several countries, including Spain (working group of the Catalonian Health Institute), Brazil (Brazilian Society of Clinical Pathology/Laboratory Medicine) and Australia (Royal College of Pathologists of Australasia) [3], [4], [5], [6], [7]. However, these programs define different QIs and criteria for setting targets, thus complicating, or even precluding, both data comparability and a universal definition of a reliable state of the art. The use of QIs in clinical laboratories worldwide has highlighted the compelling need to adopt universal QIs, common terminology and to harmonize the management of their use [8]. Moreover, it has been proven that the effective use of QIs in laboratory medicine is strongly connected to

• laboratory staff awareness of the rationale and the goals of each indicator, in order to guarantee full understanding of the reasons for their use and, consequently, a full involvement of the staff;

• standardized data collection, in order to achieve comparability of data over time regardless of the operator;

• structured data analysis and implementation of effective improvement actions, in order to reduce errors [9].

Thanks to the increasing concern expressed by Scientific Societies, International Federations and laboratory professionals and in numerous articles [10], [11], [12], [13], [14], [15], progress has been made in the harmonization of QIs’ definition and adoption. In particular, wide consensus has been reached concerning the identification of reliable QIs (number, type, terminology, rationale, purpose, collection method and target setting) covering the TTP. A special focus on the extra-analytical phases, as a result of the evidence accumulated on the vulnerability to errors of this phase, has been promoted by the International Federation of Clinical Chemistry and Laboratory Medicine (IFCC) Working Group on “Laboratory Errors and Patient Safety” (WG-LEPS) [16].

Quality indicators and total testing process

Since QIs are designed to focus on the most critical aspects of TTP, the extra-analytical phases that are more error-prone should be monitored by a higher number of indicators [17], [18], [19]. In fact, the initial steps of the testing process are characterized by complexity, different process owners (both laboratory professionals and nurses/clinicians) and limited automation. Laboratory professionals have long used quality assurance tools to control the steps in the intra-analytical phase. In fact, the availability of internal quality control procedures and external quality assessment programs, and of approved guidelines and recommendations developed by professionals for their effective use, has promoted the improvement of the intra-analytical performance, as shown by the dramatic decrease achieved in analytical errors [20], [21], [22], [23], [24]. Moreover, the impact of these activities on quality depends exclusively on laboratory staff with a sound knowledge of how to assess and monitor the performance characteristics. Vice versa, the quality performance in extra-analytical phases hinges on the variety of process owners involved (often non-laboratory personnel) and mutual responsibilities at the interfaces of several steps. For example, blood sample collection or the interpretation and/or utilization of laboratory information performed by offsite personnel strongly affect the ultimate quality of laboratory information.

There is therefore an urgent need to implement and use QIs in laboratory medicine, as a quality assurance tool, in order to control the critical steps of the extra-analytical phases. The role of QIs is well recognized by the International Standard for laboratory accreditation, the ISO 15189 [25], which calls for their use in all steps of the TTP and a monitoring process including the establishment of, for each indicator, the objective, data collection method, criteria for data interpretation, measurement limit, approach to plan the improvement actions and measurement frequency. However, although the use of QIs should be considered “a must” for accrediting medical laboratories, the standard does not specify

• the appropriate number and typology of QIs to be implemented;

• the metrics to use;

• the targets to assess the performances.

The project of WG-LEPS, on the implementation of a Model of Quality Indicators (MQI), which complies with harmonization criteria, aims to meet these needs and guarantee the comparability of data from laboratories worldwide. According to consensually accepted harmonization criteria, the QIs included in the MQI are

• patient centered;

• consistent with the requirements of the International Standard for medical laboratory accreditation (ISO 15189: 2012);

• addressed to all stages of the TTP [26].

In order to facilitate the use of the MQI, the WG-LEPS has arranged a benchmarking program; available to all laboratories, it allows the use of common QIs, standardized data collection and the diffusion of statistical data through a report. All information on the project is available in a dedicated website (www.ifcc-mqi.com).

Three different MQIs have been experimented since 2008, and now in use is the MQI discussed and approved in 2016, at the recent Consensus Conference held in Padova, “Harmonization of quality indicators in Laboratory Medicine: two years later” [15], [16], [26], [27]. The latest MQI include 27 QIs and 53 measurements (21 indicators and 43 measurements concerning the key processes; 3 and 5 concerning the Support Processes and the Outcome Measures). An order of priority has been assigned to each indicator, from 1 to 4 (1, mandatory; 2, important; 3, suggested; 4, valued), in order to facilitate the introduction of QIs into practice. In fact, participating laboratories are not obliged to collect data for the entire list, and they, at least at the beginning, can select the most appropriate QIs (priority 1) and collect and report the results on those; then eventually they can introduce and use further QIs. The greater number of priority 1 measurements concern the preanalytical [19] and postanalytical phases [9], demonstrating the need to control activities incurring a higher percentage of errors.

The list of QIs to be used cannot be fixed because, as required by the ISO 15189, they must be periodically reviewed in order to assure their continued appropriateness. Whenever the activity under control improves, the frequency of monitoring can be reduced or may no longer be justified. Efforts can therefore be focused on other activities.

Experimentation with subsequent MQIs, made since 2008, has enabled the understanding of updating needs in relation to the appropriateness of the QIs. In particular, it has highlighted the need for indicators that

• have user-friendly wording;

• identify events to keep under control or to consider undesirable;

• reflect the truly critical TTP situations.

Moreover, the experimentation has raised awareness of differences in terminology among measurement/measure, metrics and indicator, revealing the need to correct or update the indicators. Knowledge of the correct terminology, in fact, allows the appropriate formulation of the indicator and assures its effective use.

From measures, through metrics, to quality indicators

A commonly made mistake is to define all undesirable events that occur during work activities as QIs. Not all undesirable events measured are, in fact, QIs:

• all QIs are measures but not all measures are QIs;

• measure reliability, which should be defined as the ability to provide an accurate information, is strictly linked to the choice of metrics;

• several measures may be required in order to put an event, process or information under investigation.

QIs are thus objective measures that use metrics as a tool to quantify quality in relation to predefined goals.

In order to objectify an individual observation, it is essential to quantify it through a measure defined as “the dimensions, capacity, or amount of something ascertained by measuring” [28]. The National Quality Forum stressed the importance of measurement in medicine and stated, “. . . measure is defined as a fully developed metric that includes detailed specifications and may have undergone scientific testing. A fully developed measure identifies what should happen (what is being measured), who should be measured (population), where measurement should happen (setting), when it should happen (time), and how it should occur” [29]. QIs, based an appropriate metrics, are an essential tool for measuring the quality of care and, in particular, of clinical laboratory services. The terms measure and measurement are synonymous. The term metrics, often confused with measure, is “a system or standard of measurement” [28].

The concept of measure is easily comprehensible, while the term metrics is related to its influence on the interpretation of data. For example, in the case of hemolyzed samples, it is possible to measure the total number of hemolyzed samples or only hemolyzed samples requesting tests affected by hemolysis. The former indicates how many samples are hemolyzed and, therefore, the samples that affect the laboratory workflow and have to follow a different treatment, but it does not measure the impact of hemolysis on laboratory results, whereas the latter stresses the impact on laboratory results.

However, in order to make data comparable and guarantee the correct interpretation of data collected from different contexts, it is appropriate to combine individual metrics into an aggregate (composite metrics), and in the case of the above examples of hemolyzed samples, the appropriate definition of the denominator is of fundamental importance. In the former example, the number of hemolyzed samples should be related to the total number of samples, whereas in the latter, the denominator is the total number of sample requesting tests known to be affected by hemolysis.

Another issue to consider for hemolyzed samples is, for example, the detection method used by laboratories, whether visual inspection or automated serum index. Data provided might vary depending on the detection method used, thus highlighting the way in which the correct identification of state of the art, and the definition of performance targets can be compromised by subjectivity that can affect results (Figure 1) [30].

Figure 1:

QIs data collected in the 2017 (January–August) concerning hemolyzed samples detected by visual inspection (Pre-HemV) and automated haemolytic index (Pre-HemI) and the samples rejected due to hemolysis (Pre-HemR).

Moreover, the expression of the measures differs on the basis of the metrics, the most commonly used being the percentage of defects (% outside specified requirements/failure rate) and percentage of yields (% within specified requirements/success rate) that monitor the failure and success rate, respectively, or the Six Sigma metric that highlights the ability of processes [31].

Quality indicators: why is harmonization a key word?

A QI, as defined by the ISO 15189:2012, is the “measure of the degree to which a set of inherent characteristics fulfills requirements” [25]. However, the meaning of QIs is well explained in the UNI 11097: 2003, which describes them as “The information, qualitative or quantitative, associated to an event (or process or result) put under observation, that is able to evaluate its changes during the time and to verify achievements of the defined quality goals, in order to take the correct decisions and choices” [32]. This definition effectively elucidates the difference between measure and metrics and explains that the information inferred from the QI is always related to a goal. For example, if the goal is to guarantee correct patient identification, a possible QI is the evaluation of misidentified errors, which can concern errors in the personal details of patients reported on the requests or on samples, or samples without identification (unlabelled) or with lack of identification (with fewer than two identifiers). Therefore, it is necessary to measure all possible undesirable events (measure) and standardize them (metrics) in order to guarantee a structured and standardized detection of errors. However, because the absolute measures do not highlight the real level of performance if they are not referred to the total number of events kept under observation, the number of misidentified requests must be related to all requests, unlike misidentified samples, which must be related to all samples. Likewise, if the goal is to guarantee the quality of a sample to avoid affecting the results, a potential QI is the evaluation of all unsuitable samples. Therefore, it is necessary to identify and monitor the number of incorrect sample types (e.g. serum instead of plasma or incorrect containers), the number of hemolyzed, clotted and contaminated samples and with incorrect fill level, compared with the total number of samples involved in the specific evaluation. In particular, the hemolyzed samples must be related to all samples where the hemolysis is checked, the clotted samples to samples where the clots are checked, the samples contamined or with incorrect fill volume with all samples. A QI is chosen by the laboratory to gain information on specific goals that are generally in compliance with the strategic objectives. The WG-LEPS proposes an MQI that includes a list of QIs for which specific measures and metrics are defined in order to allow a harmonized approach to their adoption by clinical laboratories. In fact, although an individual laboratory should identify some QIs which enable the documentation and improvement of specific procedures and processes at a higher risk of errors, only the adoption of harmonized QIs and a common reporting system should enable the comparison of between-laboratory performance and the development of an external quality assurance program dealing with extra-analytical performances. A laboratory participating in the WG-LEPS benchmarking program has the advantage of using QIs in which the measures and metrics have been defined through consensus after experimentation in several laboratories worldwide. This harmonized system allows guaranteeing the achievement of predefined goals and the performance improvement.

Another issue discussed by International Federations, in particular IFCC and European Federation for Clinical Chemistry and Laboratory Medicine (EFLM), concerns the criterion for setting performance specifications. The definition of performance specifications makes it possible to understand the level of quality performance, thus helping professionals evaluate their QIs data, indicating the extent to which the error rate can be considered tolerable, given that a goal of “zero defects” is not always achievable.

During the most recent Consensus Conference [16], a criterion for the identification of performance specifications was defined and approved. As proposed by Fraser et al. [33], [34] for intra-analytical criteria, it is based on the identification of three performance goals. Because biological variability is not applicable, performance specifications are defined on the basis of the distribution of laboratory results, and the highest performance is assigned to results within the 25th percentile, the low performance to the results that are above the 75th percentile, and the medium performance is between the 25th and 75th percentiles [35]. Although the criterion is based on the state of the art, the clinical outcome model should be better, but no data on this issue are yet available in the literature, and few data on clinicians’ opinions have been reported. The proposed criterion has the advantage of encouraging participating laboratories to improve their performance by revealing that other laboratories have achieved better performances. The performance limits (25th and 75th percentile) based on the analyzed QI data from participating laboratories are updated at the end of each year in order to guarantee that they are always adequate with the current state of the art [16].

From quality indicators to performance improvement

The QI data collected through the benchmarking program of WG-LEPS have demonstrated that the participation of laboratories is not constant over time [16], [35], [36], [37], [38]. Although many laboratories have requested involvement in the project, few of them have systematically collected data in compliance with the planning and deadlines defined by the program. This is due, at least in part, to the lack of an automated system for data collection that does not increase the staff work and guarantees collection uniformity. One of the future goals of the WG-LEPS is to implement software, made available to all clinical laboratories, for the automated collection of the most QI data.

However, the collection of QIs data per se does not improve performance quality [39]. In order to reduce errors and improve laboratory performances, the laboratory must analyze data, identify the causes of error and undertake preventive and corrective action. The role of laboratory professionals is strategic, and any improvement will only be possible with their long-term commitment. The management of QIs must be included in the quality improvement strategy in order to be effective. An example of the use of QIs data as a starting point in implementing a risk management procedure reported in literature involves the application of the failure mode and effect analysis (FMEA) on preanalytical processes by using data of 22 QIs of the MQI. The results reported by the authors, demonstrate that “detection, identification, and monitoring of the preanalytical errors (using QIs) and implementing risk management, reduces the error rates and thereby increases the quality and improves patient safety and health system outcomes” [40]. Likewise, improvement has been achieved in the postanalytical phase in the critical values notification and report delivery processes [36].

The above reported experiences are testimony to the usefulness of MQI as a tool for quality improvement and risk management.

Future perspectives

The majority of healthcare quality measures are process measures informing users as to how the component of care is delivered [41]. In laboratory medicine, the information delivered is the result of an examination procedure affected by all preanalytical activities, which are, moreover, managed by postanalytical steps. It is therefore important to keep under control all process measures, structured as QIs, in all the extra-analytical phases. However, laboratory professionals must demonstrate, through objective measures, the effectiveness of the role of laboratory medicine in health care. Despite some efforts made to measure the impact of laboratory information on patient outcome, no structured system (in terms of measures, metrics and QIs) allowing a harmonized data collection is yet available. It is therefore still impossible to compare data from different experiences in order to achieve reliable findings and establish the state of the art.

Although outcome measures should be considered the “gold standard” in assessing and monitoring overall performances, medical laboratories encounter serious difficulties in outcome measurement, which calls for close and proactive involvement of clinicians, and depends on numerous factors, as shown in Table 1 [42]. In the current version of MQI, the majority of QIs are process measures, while a minor number concern outcome measures, three QIs and five measurements [16]. The above considerations highlight the need to define new QIs that measure the outcome of laboratory information. The road is open for engaging in this new challenge. The definition of appropriate QIs and method for data collection that evaluates the outcome should be the future mission of laboratory professionals. On the one hand, the consolidation of process measures through the systematic and timely participation in the IFCC WG-LEPS benchmarking program and, on the other hand, the implementation of new QIs focused on the assessment of outcome, should drive future efforts in the field of QIs in laboratory medicine.

Conclusions

An apposite definition for the provision of total quality in laboratory medicine might be the term “mission impossible”. The criterion for measurement should be based on the assessment of the extent to which laboratory information impacts on and improves clinical decision making, and patient management. Laboratory professionals have striven to develop an effective tool for identifying and monitoring errors and risk of errors in all steps of the TTP. Patient-centered QIs that comply with accreditation International Standards and harmonization requirements can be an effective tool in medical laboratories.

The availability of the IFCC WG-LEPS benchmarking program based on an MQI, defined and approved by a scientific consensus and addressed to monitor intra- and extra-analytical phases, has enabled the identification of the need for improvement and the importance of effective measures and metrics.

Future efforts should focus on the implementation of QIs that can highlight the impact on patient outcomes of errors, which can occur each and every phase of the TTP.