Home Mathematics The area under the generalized receiver-operating characteristic curve
Article
Licensed
Unlicensed Requires Authentication

The area under the generalized receiver-operating characteristic curve

  • Pablo Martínez-Camblor ORCID logo EMAIL logo , Sonia Pérez-Fernández and Susana Díaz-Coto
Published/Copyright: March 24, 2021
The International Journal of Biostatistics
From the journal The International Journal of Biostatistics Volume 18 Issue 1

Abstract

The receiver operating-characteristic (ROC) curve is a well-known graphical tool routinely used for evaluating the discriminatory ability of continuous markers, referring to a binary characteristic. The area under the curve (AUC) has been proposed as a summarized accuracy index. Higher values of the marker are usually associated with higher probabilities of having the characteristic under study. However, there are other situations where both, higher and lower marker scores, are associated with a positive result. The generalized ROC (gROC) curve has been proposed as a proper extension of the ROC curve to fit these situations. Of course, the corresponding area under the gROC curve, gAUC, has also been introduced as a global measure of the classification capacity. In this paper, we study in deep the gAUC properties. The weak convergence of its empirical estimator is provided while deriving an explicit and useful expression for the asymptotic variance. We also obtain the expression for the asymptotic covariance of related gAUCs and propose a non-parametric procedure to compare them. The finite-samples behavior is studied through Monte Carlo simulations under different scenarios, presenting a real-world problem in order to illustrate its practical application. The R code functions implementing the procedures are provided as Supplementary Material.

Keywords: area under the curve; diagnostic problem; generalized receiver-operating characteristic curve; summarized indices; two-sample problem
Corresponding author: Pablo Martínez-Camblor, Department of Biomedical Data Science, Geisel School of Medicine at Dartmouth, 7 Lebanon Street, Suite 309, Hinman Box 7261, Hanover, NH 03755, USA, E-mail:

Funding source: Ministerio de Ciencia e Innovación

Award Identifier / Grant number: MTM2015-63971-P

Funding source: Gobierno de Asturies

Award Identifier / Grant number: Severo Ochoa Grant BP16118

Funding source: Ministerio de Economia y Competitividad, Spain

Award Identifier / Grant number: MTM2014-55966-P

  1. Author contribution: All the authors have actively participated in all aspects of this manuscript, all of them have read and approved the final version. All the authors have accepted responsibility for the entire content of this submitted manuscript and approved submission.

  2. Research funding: This work is supported by the Grants MTM2015-63971-P and MTM2014-55966-P from the Spanish Ministerio of Economía y Competitividad, FC-15-GRUPIN14-101 and Severo Ochoa Grant BP16118 (this one for S. Pérez-Fernández) from the Asturies Government.

  3. Conflict of interest statement: The authors have no conflicts of interest to report.

Appendix

Technical issues

We are proving here the results enunciated along the manuscript. First, we provide the proof for the Proposition 1 (Section 2).

Proof of Proposition 1

Clearly, h(η) follows a uniform distribution between 0 and 1,

G h ( η ) ( p ) = P { h ( η ) ≤ p } = P 1 − ∫ 0 1 I [ L t , U t ] ( η ) d t ≤ p = P ∫ 0 1 I [ L t , U t ] ( η ) d t > 1 − p = P η ∈ [ L 1 − p , U 1 − p ] = p .

Analogously, the distribution function of h(ξ) is

F h ( ξ ) ( p ) = P { h ( ξ ) ≤ p } = P ξ ∈ [ L 1 − p , U 1 − p ] = 1 − R g ( 1 − p ) .

And then,

R ( p ) = 1 − F h ( ξ ) { G h ( η ) − 1 ( 1 − p ) } = 1 − { 1 − R g ( 1 − [ 1 − p ] ) } = R g ( p ) .

□

Now, we will prove Theorems 1 (Section 3) and 2 (Section 4).

Proof of Theorem 1

We have that

A ̂ g − A g = ∫ F ̂ n , h ̂ m ( ξ ) ( p ) d p − ∫ F h ( ξ ) ( p ) d p = ∫ F ̂ n , h ̂ m ( ξ ) ( p ) d p − ∫ F h ̂ m ( ξ ) ( p ) d p + ∫ F h ̂ m ( ξ ) ( p ) d p − ∫ F h ( ξ ) ( p ) d p = ∫ { F ̂ n , ξ ( x ) − F ξ ( x ) } d h ̂ m ( x ) + ∫ F h ̂ m ( ξ ) ( p ) d p − ∫ F h ( ξ ) ( p ) d p = ∫ { F ̂ n , ξ ( x ) − F ξ ( x ) } d h ̂ m ( x ) + ∫ { G ̂ m , h ( η ) ( p ) − G h ( η ) ( p ) } d F h ( ξ ) ( p ) .

In the last equality, we have considered that, for each p ∈ (0, 1), there exists α p ∈ [0, 1], such that [18]

∫ F h ̂ m ( ξ ) ( p ) d p = ∫ P { h ̂ m ( ξ ) ≤ p } d p = ∫ F ξ ( G ̂ m , η − 1 [ 1 − ( 1 − p ) ⋅ α p ] ) d p − ∫ F ξ ( G ̂ m , η − 1 [ 1 − p − ( 1 − p ) ⋅ α p ] ) d p = ∫ F h ( ξ ) ( p ) d G ̂ m , h ( η ) ( p ) .

Therefore, the assumption (F) guarantees that,

n ⋅ { A ̂ g − A g } − { P n , η ( X ) − P m , ξ ( Y ) } → n 0 ( a.s . ) ,

where

P m , ξ ( Y ) = 1 m ∑ j = 1 m F h ( ξ ) ( h ( y j ) ) − E F h ( ξ ) ( h ( Y ) ) , P n , η ( X ) = 1 n ∑ i = 1 n G h ( η ) ( h ( x i ) ) − E G h ( η ) ( h ( X ) )

with X and Y two independent random variables distributed as ξ and η, respectively.

Hence, the Slutski’s lemma guarantees that the asymptotic distribution of n ⋅ { A ̂ g − A g } is equivalent to the asymptotic distribution of n ⋅ { P n , η ( X ) − P m , ξ ( Y ) } [30].

n ⋅ { P n , η ( X ) − P m , ξ ( Y ) } = n ⋅ P n , η ( X ) − m ⋅ λ n ⋅ P m , ξ ( Y ) .

The central limit theorem’s assures the asymptotic normality of the above expression. Mean is clearly zero and, from the Proposition 1, the variance is

σ 2 = ∫ t 2 d R g ( t ) − ∫ t d R g ( t ) 2 + λ 2 ⋅ ∫ R g 2 ( t ) d t − ∫ R g ( t ) d t 2 = ‖ t , R g ‖ 0,1 + λ 2 ⋅ ‖ R g , t ‖ 0,1 .

Proof of Theorem 2

Using similar notation that in the previous proof, we define

Q m , ξ ( Y , ω ) = ω 1 ⋅ P m , ξ 1 ( Y 1 ) + ω 2 ⋅ P m , ξ 2 ( Y 2 ) , Q n , η ( X , ω ) = ω 1 ⋅ P n , η 1 ( X 1 ) + ω 2 ⋅ P n , η 2 ( X 2 ) .

Arguing as in Theorem 1 proof’s we have to demonstrate the asymptotic normality of the random variable

n ⋅ { Q n , η ( X , ω ) − Q m , ξ ( Y , ω ) } = n ⋅ Q n , η ( X , ω ) − m ⋅ λ n ⋅ Q m , ξ ( Y , ω ) .

We can apply again the TCL and get the normality of the above expression. Besides,

n ⋅ E [ Q n , η ( X , ω ) 2 ] = ω 1 2 ⋅ ‖ t , R g 1 ‖ 0,1 + ω 2 2 ⋅ ‖ t , R g 2 ‖ 0,1 + 2 ω 1 ω 2 ⋅ ∬ t s d F h ( t , s ) − ∫ t d F h 1 ( t ) ∫ s d F h 2 ( s ) = ω 1 2 ⋅ ‖ t , R g 1 ‖ 0,1 + ω 2 2 ⋅ ‖ t , R g 2 ‖ 0,1 + 2 ω 1 ω 2 ⋅ < t • t , F h > 0,1 ,

where F h (⋅, ⋅) is the distribution of h ( ξ ) = (h1(ξ1), h2(ξ2)). Analogously,

m ⋅ E [ Q m , ξ ( Y , ω ) 2 ] = ω 1 2 ⋅ ‖ R g 1 , t ‖ 0,1 + ω 2 2 ⋅ ‖ R g 2 , t ‖ 0,1 + 2 ω 1 ω 2 ⋅ ⟨ R g 1 • R g 2 , G h ⟩ 0,1 ,

where G h (⋅, ⋅) is the distribution of h ( η ) = (h1(η1), h2(η2)). Since λ n 2 → n λ 2 , the proof is concluded. □

R code outlines

This is a brief example of the use of the implemented functions gROC and gAUC.test in order to estimate the gROC curve and to perform hypothesis testing for comparing gAUCs, respectively.

  1. To import the database:

  2. To compute the optimal gROC curve under restriction (C) for the markers WBC (white blood cells count) and LYM (lymphocytes percentage), respectively:

  1. To compute the approx. optimal gROC curve under restriction (C) for WBC when the starting point 1 − Sp is 1/2:

  1. To compute a 95% confidence interval for gAUC of each marker:

  1. To compute a 95% confidence interval for the gAUC difference A g LYM − A g WBC and p-value for testing the null H 0 : A g LYM − A g WBC = 0 :

References

1. Green, DM, Swets, JA. Signal detection theory and psychophysics. New York: Wiley; 1966.Search in Google Scholar

2. Hanley, JA, McNeil, BJ. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology 1982;143:29–36. https://doi.org/10.1148/radiology.143.1.7063747.Search in Google Scholar PubMed

3. Demidenko, E. The p-value you can’t buy. Am Statistician 2016;70:33–8. https://doi.org/10.1080/00031305.2015.1069760.Search in Google Scholar PubMed PubMed Central

4. Delong, ER, Delong, DM, Clarke-Pearson, DL. Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics 1988;44:837–45. https://doi.org/10.2307/2531595.Search in Google Scholar

5. Braun, TM, Alonzo, TA. A modified sign test for comparing paired ROC curves. Biostatistics 2007;9:364–72. https://doi.org/10.1093/biostatistics/kxm036.Search in Google Scholar PubMed

6. Venkatraman, ES, Begg, CB. A distribution-free procedure for comparing receiver operating characteristic curves from a paired experiment. Biometrika 1996;83:835–48. https://doi.org/10.1093/biomet/83.4.835.Search in Google Scholar

7. Venkatraman, ES. A permutation test to compare receiver operating characteristic curves. Biometrics 2000;56:1134–8. https://doi.org/10.1111/j.0006-341x.2000.01134.x.Search in Google Scholar PubMed

8. Martínez Camblor, P, Carleos, C, Corral, N. Powerful nonparametric statistics to compare k independent ROC curves. J Appl Stat 2011;38:1317–32. https://doi.org/10.1080/02664763.2010.498504.Search in Google Scholar

9. Martínez-Camblor, P, Carleos, C, Corral, N. General nonparametric ROC curve comparison. J Korean Stat Soc 2013;42:71–81. https://doi.org/10.1016/j.jkss.2012.05.002.Search in Google Scholar

10. Hilden, J. The area under the ROC curve and its competitors. Med Decis Making 1991;11:95–101. https://doi.org/10.1177/0272989x9101100204.Search in Google Scholar

11. Yousef, WA. Assessing classifiers in terms of the partial area under the ROC curve. Comput Stat Data Anal 2013;64:51–70. https://doi.org/10.1016/j.csda.2013.02.032.Search in Google Scholar

12. Pardo, MC, Franco-Pereira, AM. Non parametric ROC summary statistics. REVSTAT 2017;15:583–600 .Search in Google Scholar

13. McIntosh, MW, Pepe, MS. Combining several screening tests: optimality of the risk score. Biometrics 2002;58:657–64. https://doi.org/10.1111/j.0006-341x.2002.00657.x.Search in Google Scholar PubMed

14. Metz, CE, Pan, X. ‘Proper’ binormal ROC curves: theory and maximum-likelihood estimation. J Math Psychol 1999;43:1–33. https://doi.org/10.1006/jmps.1998.1218.Search in Google Scholar PubMed

15. Qin, J, Zhang, B. Best combination of multiple diagnostic tests for screening purposes. Stat Med 2010;29:2905–19. https://doi.org/10.1002/sim.4068.Search in Google Scholar PubMed

16. Chen, B, Li, P, Qin, J, Yu, T. Using a monotonic density ratio model to find the asymptotically optimal combination of multiple diagnostic tests. J Am Stat Assoc 2016;111:861–74. https://doi.org/10.1080/01621459.2015.1066681.Search in Google Scholar

17. Martínez-Camblor, P, Pérez-Fernández, S, Díaz-Coto, S. Optimal classification scores based on multivariate marker transformations. AStA Adv Stat Anal 2021;1–16. https://doi.org/10.1007/s10182-020-00388-z. In press.Search in Google Scholar

18. Martínez-Camblor, P, Corral, N, Rey, C, Pascual, J, Cernuda-Morollón, E. Receiver operating characteristic curve generalization for non-monotone relationships. Stat Methods Med Res 2017;26:113–23. https://doi.org/10.1177/0962280214541095.Search in Google Scholar PubMed

19. Martínez-Camblor, P, Pérez-Fernández, S, Díaz-Coto, S. Improving the biomarker diagnostic capacity via functional transformations. J Appl Stat 2019;46:1550–66. https://doi.org/10.1080/02664763.2018.1554628.Search in Google Scholar

20. Martínez-Camblor, P, Pardo-Fernández, JC. Parametric estimates for the receiver operating characteristic curve generalization for non-monotone relationships. Stat Methods Med Res 2019;28:2032–48. https://doi.org/10.1177/0962280217747009.Search in Google Scholar PubMed

21. Zhou, XH, McClish, DK, Obuchowski, NA. Statistical methods in diagnostic medicine. In: Series in probability and statistics. New York, NY: Wiley Blackwell; 2002.10.1002/9780470317082Search in Google Scholar

22. Pepe, MS. The statistical evaluation of medical tests for classification and prediction. Oxford statistical science series. Oxford: Oxford University Press; 2003.10.1093/oso/9780198509844.001.0001Search in Google Scholar

23. Krzanowski, W, Hand, D. ROC curves for continuous data. New York: Chapman and Hall/CRC; 2009.10.1201/9781439800225Search in Google Scholar

24. Pérez-Fernández, S, Martínez-Camblor, P, Filzmoser, P, Corral, N. Visualizing the decision rules behind the ROC curves: understanding the classification process. AStA Adv Stat Anal 2021;105:135–61. https://doi.org/10.1007/s10182-020-00385-2.Search in Google Scholar

25. Martínez-Camblor, P, Pardo-Fernández, JC. The Youden index in the generalized receiver operating characteristic curve context. Int J Biostat 2019;15:1–20. https://doi.org/10.1515/ijb-2018-0060.Search in Google Scholar PubMed

26. Spanos, A, Harrell, FE, Durack, DT. Differential diagnosis of acute meningitis: an analysis of the predictive value of initial observations. J Am Med Assoc 1989;262:2700–7. https://doi.org/10.1001/jama.262.19.2700.Search in Google Scholar

27. Martínez-Camblor, P. Area under the ROC curve comparison in the presence of missing data. J Korean Surg Soc 2013;42:431–42. https://doi.org/10.1016/j.jkss.2013.01.004.Search in Google Scholar

28. Mossman, D. Three-way ROCs. Med Decis Making 1999;19:78–89. https://doi.org/10.1177/0272989x9901900110.Search in Google Scholar

29. Nakas, C, Yiannoutsos, T. Ordered multiple-class roc analysis with continuous measurements. Stat Med 2004;23:3437–49. https://doi.org/10.1002/sim.1917.Search in Google Scholar PubMed

30. van der Vaart AW . Asymptotic statistics. In: Series in statistical and probabilistic mathematics. Cambridge: Cambridge University Press; 1998.10.1017/CBO9780511802256Search in Google Scholar

Supplementary Material

The online version of this article offers supplementary material (https://doi.org/10.1515/ijb-2020-0091).

Received: 2020-06-13
Revised: 2020-11-16
Accepted: 2021-03-01
Published Online: 2021-03-24

© 2021 Walter de Gruyter GmbH, Berlin/Boston

Articles in the same Issue

  1. Frontmatter
  2. Research Articles
  3. Integrating additional knowledge into the estimation of graphical models
  4. Asymptotic properties of the two one-sided t-tests – new insights and the Schuirmann-constant
  5. Bayesian optimization design for finding a maximum tolerated dose combination in phase I clinical trials
  6. A Bayesian mixture model for changepoint estimation using ordinal predictors
  7. Power prior for borrowing the real-world data in bioequivalence test with a parallel design
  8. Bayesian approaches to variable selection: a comparative study from practical perspectives
  9. Bayesian adaptive design of early-phase clinical trials for precision medicine based on cancer biomarkers
  10. More than one way: exploring the capabilities of different estimation approaches to joint models for longitudinal and time-to-event outcomes
  11. Designing efficient randomized trials: power and sample size calculation when using semiparametric efficient estimators
  12. Power formulas for mixed effects models with random slope and intercept comparing rate of change across groups
  13. The effect of data aggregation on dispersion estimates in count data models
  14. A zero-inflated non-negative matrix factorization for the deconvolution of mixed signals of biological data
  15. Multiple scaled symmetric distributions in allometric studies
  16. Estimation of semi-Markov multi-state models: a comparison of the sojourn times and transition intensities approaches
  17. Regularized bidimensional estimation of the hazard rate
  18. The effect of random-effects misspecification on classification accuracy
  19. The area under the generalized receiver-operating characteristic curve
https://doi.org/10.1515/ijb-2020-0091
Keywords for this article
area under the curve; diagnostic problem; generalized receiver-operating characteristic curve; summarized indices; two-sample problem

Articles in the same Issue

  1. Frontmatter
  2. Research Articles
  3. Integrating additional knowledge into the estimation of graphical models
  4. Asymptotic properties of the two one-sided t-tests – new insights and the Schuirmann-constant
  5. Bayesian optimization design for finding a maximum tolerated dose combination in phase I clinical trials
  6. A Bayesian mixture model for changepoint estimation using ordinal predictors
  7. Power prior for borrowing the real-world data in bioequivalence test with a parallel design
  8. Bayesian approaches to variable selection: a comparative study from practical perspectives
  9. Bayesian adaptive design of early-phase clinical trials for precision medicine based on cancer biomarkers
  10. More than one way: exploring the capabilities of different estimation approaches to joint models for longitudinal and time-to-event outcomes
  11. Designing efficient randomized trials: power and sample size calculation when using semiparametric efficient estimators
  12. Power formulas for mixed effects models with random slope and intercept comparing rate of change across groups
  13. The effect of data aggregation on dispersion estimates in count data models
  14. A zero-inflated non-negative matrix factorization for the deconvolution of mixed signals of biological data
  15. Multiple scaled symmetric distributions in allometric studies
  16. Estimation of semi-Markov multi-state models: a comparison of the sojourn times and transition intensities approaches
  17. Regularized bidimensional estimation of the hazard rate
  18. The effect of random-effects misspecification on classification accuracy
  19. The area under the generalized receiver-operating characteristic curve
Sign up now to receive a 20% welcome discount
Institutional Access
How does access work?
Have an idea on how to improve our website?
Please write us.
© 2025 De Gruyter Brill
Downloaded on 31.12.2025 from https://www.degruyterbrill.com/document/doi/10.1515/ijb-2020-0091/html
Scroll to top button