The problem with total error models in establishing performance specifications and a simple remedy
-
Jan S. Krouwer
Abstract
A recent issue in this journal revisited performance specifications since the Stockholm conference. Of the three recommended methods, two use total error models to establish performance specifications. It is shown that the most commonly used total error model – the Westgard model – is deficient, yet even more complete models fail to capture all errors that comprise total error. Moreover, total error models are often set at 95% of results, which leave 5% of results as unspecified. Glucose meter performance standards are used to illustrate these problems. The Westgard model is useful to asses assay performance but not to set performance specifications. Total error can be used to set performance specifications if the specifications include 100% of the results.
Author contributions: The author has accepted responsibility for the entire content of this submitted manuscript and approved submission.
Research funding: None declared.
Employment or leadership: None declared.
Honorarium: None declared.
Competing interests: The funding organization(s) played no role in the study design; in the collection, analysis, and interpretation of data; in the writing of the report; or in the decision to submit the report for publication.
References
1. Panteghini M, Sandberg S, editors. Special issue: 1st EFLM Strategic Conference/“Defining analytical performance goals – 15 years after the Stockholm Conference. Clin Chem Lab Med 2015;53:829–958.10.1515/cclm-2015-0303Suche in Google Scholar PubMed
2. Kenny D, Fraser CG, Hyltoft Petersen P, Kallner A. Consensus agreement: conference on strategies to set global quality specifications in laboratory medicine. Scand J Clin Lab Invest 1999;59:585.10.1080/00365519950185409Suche in Google Scholar
3. Sandberg S, Fraser CG, Horvath AR, Jansen R, Jones G, Oosterhuis W, et. al. Defining analytical performance specifications: Consensus Statement from the 1st Strategic Conference of the European Federation of Clinical Chemistry and Laboratory Medicine. Clin Chem Lab Med 2015;53:833–5.10.1515/cclm-2015-0067Suche in Google Scholar PubMed
4. Westgard JO, Carey RN, Wold S. Criteria for judging precision and accuracy in method development and evaluation. Clin Chem 1974;20:825–33.10.1093/clinchem/20.7.825Suche in Google Scholar
5. Horvath AR, Bossuyt PM, Sandberg S, St John A, Monaghan PJ, Verhagen-Kamerbeek WD, et al. Setting analytical performance specifications based on outcome studies – is it possible? Clin Chem Lab Med 2015;53:841–8.10.1515/cclm-2015-0214Suche in Google Scholar PubMed
6. Karon BS, Boyd JC, Klee GG. Glucose meter performance criteria for tight glycemic control estimated by simulation modeling. Clin Chem 2010;56:1091–7.10.1373/clinchem.2010.145367Suche in Google Scholar PubMed
7. Oosterhuis WP, Sandberg S. Proposal for the modification of the conventional model for establishing performance specifications. Clin Chem Lab Med 2015;53:925–37.10.1515/cclm-2014-1146Suche in Google Scholar PubMed
8. Lawton WH, Sylvester EA, Young-Ferraro BJ. Statistical comparison of multiple analytic procedures: application to clinical chemistry. Technometrics 1979;21:397–409.10.1080/00401706.1979.10489809Suche in Google Scholar
9. Krouwer JS. Multi-factor designs. IV. How multi-factor designs improve the estimate of total error by accounting for protocol-specific biases. Clin Chem 1991;37:26–9.10.1093/clinchem/37.1.26Suche in Google Scholar
10. Krouwer JS. Estimating total analytical error and its sources. Techniques to improve method evaluation. Arch Pathol Lab Med 1992;116:726–31.Suche in Google Scholar
11. ISO 15197. In vitro diagnostic test systems–requirements for blood glucose monitoring systems for self-testing in managing diabetes mellitus. Geneva, Switzerland: International Organization for Standardization, 2003.Suche in Google Scholar
12. CLSI. POCT12-A3 Point-of-care blood glucose testing in acute and chronic care facilities; Approved Guideline, 3rd ed. Wayne, PA: Clinical and Laboratory Standards Institute, 2012.Suche in Google Scholar
13. International Organization for Standardization. In vitro diagnostic test systems. Requirements for blood glucose monitoring system for self testing in managing diabetes mellitus. Reference number ISO 15197:2013. Geneva: International Organization for Standardization, 2013.Suche in Google Scholar
14. Draft Guidence. Blood glucose monitoring test systems for prescription point-of care use. Rockville, MD: Center for Devices and Radiological Health. [cited 2015 10th March]; Available from: http://www.fda.gov/downloads/medicaldevices/deviceregulationandguidance/guidancedocuments/ucm380325.pdf, 2015.Suche in Google Scholar
15. Krouwer JS. How to improve total error modeling by accounting for error sources beyond imprecision and bias. Clin Chem 2001;47:1329–30.10.1093/clinchem/47.7.1329Suche in Google Scholar
16. Krouwer JS. The danger of using total error models to compare glucose meter performance. J Diabetes Sci Technol 2014;8:419–21.10.1177/1932296813518673Suche in Google Scholar PubMed PubMed Central
17. Boyd JC, Bruns DE. Quality specifications for glucose meters: assessment by simulation modeling of errors in insulin dose. Clin Chem 2001;47:209–14.10.1093/clinchem/47.2.209Suche in Google Scholar
18. Brazg RL, Klaff LJ, Parkin CG. Performance variability of seven commonly used self-monitoring of blood glucose systems: clinical considerations for patients and providers. J Diabetes Sci Technol 2013;7:144–52.10.1177/193229681300700117Suche in Google Scholar PubMed PubMed Central
19. CLSI. EP21-A Estimation of total analytical error for clinical laboratory methods; Approved Guideline. Wayne, PA: Clinical and Laboratory Standards Institute, 2003.Suche in Google Scholar
20. Krouwer JS. Why specifications for allowable glucose meter errors should include 100% of the data. Clin Chem Lab Med 2013;51:1543–4.10.1515/cclm-2013-0387Suche in Google Scholar PubMed
21. CLSI. EP27-A How to construct and interpret an error grid for quantitative diagnostic assays; Approved Guideline. Wayne, PA: Clinical and Laboratory Standards Institute, 2012.Suche in Google Scholar
22. Klonoff DC, Lias C, Vigersky R, Clarke W, Parkes JL, Sacks DB, et al. The surveillance error grid. J Diabetes Sci Technol 2014;8:658–72.10.1177/1932296814539589Suche in Google Scholar PubMed PubMed Central
©2016 by De Gruyter
Artikel in diesem Heft
- Frontmatter
- Editorials
- Laboratory analytical quality – the process continues
- Measurement uncertainty – a revised understanding of its calculation and use
- Review
- Substrate-zymography: a still worthwhile method for gelatinases analysis in biological samples
- Opinion Papers
- Is the combination of trueness and precision in one expression meaningful? On the use of total error and uncertainty in clinical chemistry
- The problem with total error models in establishing performance specifications and a simple remedy
- Measurement uncertainty for clinical laboratories – a revision of the concept
- Uncertainty in measurement and total error – are they so incompatible?
- General Clinical Chemistry and Laboratory Medicine
- Using the hazard ratio to evaluate allowable total error in predictive measurands
- Performance of electrolyte measurements assessed by a trueness verification program
- Analytical interference by monoclonal immunoglobulins on the direct bilirubin AU Beckman Coulter assay: the benefit of unsuspected diagnosis from spurious results
- Preliminary probe of quality indicators and quality specification in total testing process in 5753 laboratories in China
- Analytical and clinical evaluation of the new Fujirebio Lumipulse®G non-competitive assay for 25(OH)-vitamin D and three immunoassays for 25(OH)D in healthy subjects, osteoporotic patients, third trimester pregnant women, healthy African subjects, hemodialyzed and intensive care patients
- Patient-performed extraction of faecal calprotectin
- Comparison of the clinical utility of the Elia CTD Screen to indirect immunofluorescence on Hep-2 cells
- Reference Values and Biological Variations
- Sex-related differences in the association of ghrelin levels with obesity in adolescents
- Gestation specific reference intervals for thyroid function tests in pregnancy
- Cancer Diagnostics
- SOX17 promoter methylation in plasma circulating tumor DNA of patients with non-small cell lung cancer
- Dopamine concentration in blood platelets is elevated in patients with head and neck paragangliomas
- Letters to the Editor
- Cancer dynamics and the success of cancer screening programs
- Mother’s instinct – a rare case of multiple test interferences due to heterophile antibodies
- Sigma metric or defects per million opportunities (DPMO): the performance of clinical laboratories should be evaluated by the Sigma metrics at decimal level with DPMOs
- A national survey of preanalytical handling of oral glucose tolerance tests in pregnancy
- Updating pregnancy diabetes guidelines: is (y)our laboratory ready?
- Low serum bilirubin values are associated with pulmonary embolism in a case-control study
- Effect of Hb H on HbA1c measurements as measured by IFCC reference method and affinity HPLC
- Adipocytes in venipunctures cause falsely elevated S-100B serum values
- Earlier detection of sepsis by Candida parapsilosis using three-dimensional cytographic anomalies on the Mindray BC-6800 hematological analyzer
- Theranos phenomenon – part 4: Theranos at an International Conference
Artikel in diesem Heft
- Frontmatter
- Editorials
- Laboratory analytical quality – the process continues
- Measurement uncertainty – a revised understanding of its calculation and use
- Review
- Substrate-zymography: a still worthwhile method for gelatinases analysis in biological samples
- Opinion Papers
- Is the combination of trueness and precision in one expression meaningful? On the use of total error and uncertainty in clinical chemistry
- The problem with total error models in establishing performance specifications and a simple remedy
- Measurement uncertainty for clinical laboratories – a revision of the concept
- Uncertainty in measurement and total error – are they so incompatible?
- General Clinical Chemistry and Laboratory Medicine
- Using the hazard ratio to evaluate allowable total error in predictive measurands
- Performance of electrolyte measurements assessed by a trueness verification program
- Analytical interference by monoclonal immunoglobulins on the direct bilirubin AU Beckman Coulter assay: the benefit of unsuspected diagnosis from spurious results
- Preliminary probe of quality indicators and quality specification in total testing process in 5753 laboratories in China
- Analytical and clinical evaluation of the new Fujirebio Lumipulse®G non-competitive assay for 25(OH)-vitamin D and three immunoassays for 25(OH)D in healthy subjects, osteoporotic patients, third trimester pregnant women, healthy African subjects, hemodialyzed and intensive care patients
- Patient-performed extraction of faecal calprotectin
- Comparison of the clinical utility of the Elia CTD Screen to indirect immunofluorescence on Hep-2 cells
- Reference Values and Biological Variations
- Sex-related differences in the association of ghrelin levels with obesity in adolescents
- Gestation specific reference intervals for thyroid function tests in pregnancy
- Cancer Diagnostics
- SOX17 promoter methylation in plasma circulating tumor DNA of patients with non-small cell lung cancer
- Dopamine concentration in blood platelets is elevated in patients with head and neck paragangliomas
- Letters to the Editor
- Cancer dynamics and the success of cancer screening programs
- Mother’s instinct – a rare case of multiple test interferences due to heterophile antibodies
- Sigma metric or defects per million opportunities (DPMO): the performance of clinical laboratories should be evaluated by the Sigma metrics at decimal level with DPMOs
- A national survey of preanalytical handling of oral glucose tolerance tests in pregnancy
- Updating pregnancy diabetes guidelines: is (y)our laboratory ready?
- Low serum bilirubin values are associated with pulmonary embolism in a case-control study
- Effect of Hb H on HbA1c measurements as measured by IFCC reference method and affinity HPLC
- Adipocytes in venipunctures cause falsely elevated S-100B serum values
- Earlier detection of sepsis by Candida parapsilosis using three-dimensional cytographic anomalies on the Mindray BC-6800 hematological analyzer
- Theranos phenomenon – part 4: Theranos at an International Conference
