AUC-Maximizing Ensembles through Metalearning

1 Introduction

In the field of biostatistics, binary classification problems arise in many applications, for example, in diagnostic testing. There are also many problems for which the outcome is rare, or imbalanced, meaning the number of positive cases far outweighs the number of negative cases (or vice versa). In this type of prediction problem, the Area Under the ROC Curve (AUC) is frequently used to measure the performance of an estimator (i. e., model fit) due to its robustness against prior class probabilities. When AUC maximization is the goal, a classifier that aims to specifically maximize AUC can have significant advantages in these types of problems.

However, most commonly used classification algorithms work by optimizing an objective function that is unrelated to AUC − for example, accuracy (or error rate). If the training dataset has an imbalanced outcome, this can lead to classifiers where the majority class has close to 100 % accuracy, while the minority class has an accuracy of closer to 0–10 % [1]. In practice, the accuracy of the minority class is often more important than the accuracy of the majority class. Therefore, unless some type of intervention (e. g., under-sampling, over-sampling) is used to help alleviate this issue, or AUC maximization is inherent to the algorithm, class imbalance may negatively impact the performance of a binary classification algorithm.

In this paper, we introduce an ensemble approach to AUC maximization for binary classification problems. Ensemble methods are algorithms that combine the output from a group of base learning algorithms, with the goal of creating an estimator that has predictive performance over the individual algorithms that make up the ensemble. The Super Learner algorithm [2] is an ensemble algorithm which generalizes stacking [3–5], by allowing for more general loss functions and hence a broader range of estimator combinations. The Super Learner is built on the theory of cross-validation and has been proven to represent an asymptotically optimal system for learning [2].

Super Learner, described in further detail in Section 2, estimates the optimal combination of the base learning algorithms in the ensemble, with respect to a user-defined loss function. The “metalearning step” in the Super Learner algorithm is the process of data-adaptively determining the optimal combination a specific group of base learner fits via a second-level metalearning algorithm. With respect to model performance, this leads to estimators that have superior (or at worst, equal) performance to the top base algorithm in the ensemble. Even if none of the base learners specifically maximize AUC, it is possible to inject AUC-maximization directly into imbalanced data problems via the metalearning step of the Super Learner algorithm.

Any type of parametric or nonparametric algorithm (which is associated with a bounded loss function) can be used in the metalearning step, although in practice, it is common to estimate the optimal linear combination of the base learners. Since the Super Learner framework allows for any loss function (and corresponding risk function), to be used in the metalearning step, it is possible to create ensemble learners that specifically aim minimize a user-defined loss function of interest.

The loss function associated with AUC, also called “rank loss,” measures the bipartite ranking error, or disagreement between pairs of examples. The associated risk is calculated as 1.0 − AUC. In the Super Learner algorithm, minimization of the rank loss or, equivalently, maximization of the AUC, can be approached directly by using an AUC-maximining metalearning algorithm. In Section 3, we discuss how AUC maximization can be formulated as a nonlinear optimization problem. We have implemented the AUC-maximizing metalearning algorithm as an update to the SuperLearner R package and demonstrate its usage with a code example.

In Section 4, we evaluate the effectiveness of a large number of nonlinear optimization algorithms to maximize the cross-validated (CV) AUC of a Super Learner fit. We compare the cross-validated AUC of the AUC-optimized ensemble fits to the cross-validated AUC of the ensembles that do not attempt to optimize AUC. Super Learner fits using various metalearning algorithms are benchmarked using training sets with varying levels of class imbalance. The results provide evidence that “AUC-maximizing” metalearners can, and often do, out-perform non-“AUC-maximizing” metalearning methods (e. g. non-negative least squares), with respect to cross-validated AUC of the ensemble. The results also demonstrate that, on certain datasets, as the level of class imbalance increases in the response variable of the training set, the Super Learner ensemble out-performs the top base algorithm by a larger degree. This suggests that datasets with greater class imbalance in the outcome variable might benefit from Super Learner ensembles with direct AUC-maximization in the metalearning step.

2 Ensemble metalearning

The Super Learner prediction is the optimal combination of the predicted values from the base learners, which is the motivation behind the name, “Super Learner.” The optimal way of combining the base learning algorithms is precisely what is estimated in the metalearning step of the Super Learner algorithm. The output from the base learners, also called “level-one” data in the stacking literature [5], serves as input to the metalearner algorithm. Super Learner theory requires cross-validation to generate the level-one dataset, and in practice, k-fold cross-validation is often used.

The following describes how to construct the level-one dataset. Assume that the training set is comprised of n independent and identically distributed observations, {O₁,..., O_n}, where O_i=(X_i, Y_i) and X_i∈Rp is a vector of covariate or feature values and Y_i∈ R is the outcome. Consider an ensemble comprised of a set of L base learning algorithms, {ψ¹,..., ψ ^L}, each of which is indexed by an algorithm class, as well as a specific set of model parameters. Then, the process of constructing the level-one dataset will involve generating an n×L matrix, Z, of k-fold cross-validated predicted values as follows:

1. The original training set, X, is divided randomly into V roughly-equal pieces (validation folds), X₍₁₎,...,X_(V).

2. For each base learner in the ensemble, ψ^l, k-fold cross-validation is used to generate n cross-validated predicted values associated with the l^th learner. These n-dimensional vectors of cross-validated predicted values become the L columns of Z.

   The level-one design matrix, Z, along with the original outcome vector, (Y₁,...,Y_n) ∈ Rn, is then used to train the metalearning algorithm, Φ.

3 AUC maximization

Given a set of base learning algorithms, the linear combination of the base learners that maximizes the cross-validated AUC of the Super Learner ensemble can be found using nonlinear optimization.

3.2 AUC-maximizing ensembles

The SuperLearner R package was used to evaluate various metalearning methods. The five pre-existing methods, described in Table 1, were compared against a new set of AUC-maximizing metalearning methods. As part of this exercise, we implemented a new metalearning function, method.AUC, for the SuperLearner package. In the method.AUC function, the weights are initialized as αinit=1L,....,1L, where L is the number of base learners. The function will execute an optimization method that maximizes AUC, and will return the optimal weights found by the algorithm.

The method.AUC function currently supports four optim-based and sixteen nloptr-based optimization algorithms by default. Many of these algorithms support box-constraints, so additional configurations of the existing methods can also be specified using our software. An example of how to specify a custom metalearning function by wrapping the method.AUC function is shown in Figure 1. In the example, we specify the metalearner to use the unbounded, optim-based, BFGS method [22–25], with post-optimization normalization of the weights.

Table 2:

Metalearning methods evaluated.

Method	Description	min{αj}	max{αj}
NNLS	Non-negative least squares (nnls)	0	Inf
NNLS2	Non-negative least squares (quadprog)	0	Inf
CC_LS	Non-negative least squares (nloptr)	0	1
NNloglik	Negative log-likelihood (Binomial) (optim)	0	Inf
CC_nloglik	Negative log-likelihood (Binomial) (nloptr)	0	1
AUC_nlopt.1	DIRECT	0	1
AUC_nlopt.2	DIRECT	0	Inf
AUC_nlopt.3	DIRECT-L	0	1
AUC_nlopt.4	DIRECT-L	0	Inf
AUC_nlopt.5	DIRECT-L RAND	0	1
AUC_nlopt.6	DIRECT-L RAND	0	Inf
AUC_nlopt.7	DIRECT NOSCAL	0	1
AUC_nlopt.8	DIRECT NOSCAL	0	Inf
AUC_nlopt.9	DIRECT-L NOSCAL	0	1
AUC_nlopt.10	DIRECT-L NOSCAL	0	Inf
AUC_nlopt.11	DIRECT-L RAND NOSCAL	0	1
AUC_nlopt.12	DIRECT-L RAND NOSCAL	0	Inf
AUC_nlopt.13	ORIG DIRECT	0	1
AUC_nlopt.14	ORIG DIRECT	0	Inf
AUC_nlopt.15	ORIG DIRECT-L	0	1
AUC_nlopt.16	ORIG DIRECT-L	0	Inf
AUC_nlopt.17	Controlled Random Search with Local Mutation	0	1
AUC_nlopt.18	Controlled Random Search with Local Mutation	0	Inf
AUC_nlopt.19	Improved Stochastic Ranking Evolution Strategy	0	1
AUC_nlopt.20	Improved Stochastic Ranking Evolution Strategy	0	Inf
AUC_nlopt.21	Principal Axis (PRAXIS)	0	1
AUC_nlopt.22	Principal Axis (PRAXIS)	0	Inf
AUC_nlopt.23	Constrained Opt. by Linear Approximations	0	1
AUC_nlopt.24	Constrained Opt. by Linear Approximations	0	Inf
AUC_nlopt.25	Bounded NEWUOA	0	1
AUC_nlopt.26	Bounded NEWUOA	0	Inf
AUC_nlopt.27	Nelder-Mead	0	1
AUC_nlopt.28	Nelder-Mead	0	Inf
AUC_nlopt.29	Sbplex	0	1
AUC_nlopt.30	Sbplex	0	Inf
AUC_nlopt.31	BOBYQA	0	1
AUC_nlopt.32	BOBYQA	0	Inf
AUC_optim.1	L-BFGS-B	0	1
AUC_optim.2	L-BFGS-B	0	Inf
AUC_optim.3	Nelder-Mead	−Inf	Inf
AUC_optim.4	BFGS	−Inf	Inf
AUC_optim.5	Conjugate Gradient (CG)	−Inf	Inf

Figure 1:

Example of how to use the “method.AUC” function to create customized AUC-maximizing metalearning functions.

Figure 2:

Example of how to use the custom “method.AUC_optim.4” metalearning function with the SuperLearner function..

A complete list of the default SuperLearner metalearning methods plus the new optim and nloptr-based AUC-maximizing metalearning methods are listed in Table 2. A total of 37 new AUC-maximizing metalearning functions were evaluated. The AUC_nlopt.X (where X is an integer between 1 and 20) functions implement global optimization routines and the remainder (21–32) are local, derivative-free, methods. The AUC_optim.X (where X is an integer between 1 and 5) functions implement the variations of the optim-based methods.

To reduce the computational burden of evaluating a large number of metalearners, we implemented two functions for the SuperLearner package which simplify the processes of re-combining the base learner fits using a new metalearning algorithm. These functions, recombineSL and recombineCVSL, take as input an existing “SuperLearner” or “CV.SuperLearner” fit, and a new metalearning function. They re-use the existing level-one dataset and base learner fits stored in the model object, re-run the metalearning step and return the updated fit. A simple example of how to use the recombineSL function is shown in Figure 3.

Figure 3:

Example of how to update an existing “SuperLearner” fit by re-training the metalearner with a new method.

4 Benchmark results

This section contains the results from the benchmarks of various metalearning methods in a binary classification problem. The purpose of these benchmarks is twofold. We evaluate whether AUC-maximizing metalearners, compared to other methods such as NNLS, actually maximize the cross-validated AUC of the Super Learner ensemble. We also investigate the training set characteristics that lead to the greatest performance gain of the ensemble (over the best individual model), as measured by cross-validated AUC. In particular, we measure the effect that class imbalance, as well as training size, has on the performance of Super Learner, under various metalearning methods.

After computing the performance of Super Learner using the 42 metalearners under consideration on a dataset, we identify which metalearning algorithm yields the best cross-validated ensemble AUC. The remaining metalearners are compared by calculating the offset, in terms of cross-validated AUC, between the Super Learner utilizing the top metalearing method and the other methods. For example, if the top method produced a model with a CV AUC of 0.75, the offset between the top method and a model with 0.73 CV AUC would be 0.02. The AUC offsets for each dataset-metalearner combination are displayed in a grid-format using heatmaps.

For each training set, we contrast the performance of the best individual model, as determined by a cross-validated grid-search, with the cross-validated AUC of the Super Learner. In the grid search method, the estimator with the best cross-validated performance (as measured by a given loss criterion), is selected as the winning algorithm, and the other algorithms are discarded. In the Super Learning, or stacking, context, the process of generating the level-one dataset using crossvalidation is equivalent to a cross-validated grid-search of the base learners. Moreover, in the Super Learner literature, the grid-search technique is referred to as the Discrete Super Learner algorithm [26].

It is quite common for machine learning practitioners to use a grid-search to evaluate the performance of a set of unique candidate learning algorithms, or unique sets of model parameters within a single algorithm class, as a means to select the best model from those under consideration. Since the metalearning step requires a only small amount of computation as compared to the computation involved in generating level-one data, executing a grid-search and training the Super Learner algorithm are computationally commensurate tasks. Although grid-search and Super Learning require a similar amount of work, the Super Learner ensemble, by optimally combining the set of candidate estimators, can provide a boost in performance over the top base model. For context, we also provide the cross-validated AUC for each of the base models in the ensemble.

In the benchmarks, we use a single, diverse base learner library which contains the following five algorithms: Lasso Regression, Generalized Additive Models, Random Forest (with 1,000 trees), Polynomial Spline Regression and K-Nearest Neighbor (k = 10). The R packages that were used for each these methods are listed in Table 3.

4.2 Ensemble performance by metalearner

For the HIGGS datasets, the results shown in Tables 4 and 5 suggest that AUC-maximizing metalearners and the negative log-likelihood method typically perform best among all reviewed metalearning algorithms, as measured by cross-validated AUC. Although the AUC-based metalearners often perform better than the non-AUC methods for the HIGGS datasets, there are some examples where using the loss function associated with the negative log-likelihood (of the binomial distribution) yields the top Super Learner ensemble. Unlike the AUC-based methods, the log-likelihood-based metalearning methods in the SuperLearner package, such as method.NNloglik and method.CC_nloglik, use radient information in the optimization process, which can be helpful.

4.2.1 HIGGS results

Table 4:

Top Metalearner performance for HIGGS Datasets, as measured by cross-validated AUC (n=10,000; CV=10×10).

No.	P(Y=1)	Metalearner	SL AUC Gain	SL AUC	GS AUC	GS Model
1	0.01	CC_nloglik	0.003	0.721	0.718	gam
2	0.02	AUC_optim.4	0.016	0.739	0.723	gam
3	0.03	NNloglik	0.023	0.764	0.741	gam
4	0.04	AUC_optim.4	0.038	0.770	0.732	gam
5	0.05	AUC_optim.4	0.032	0.759	0.727	randomForest
6	0.10	NNloglik	0.018	0.768	0.750	polymars
7	0.20	AUC_nlopt.11	0.014	0.778	0.764	randomForest
8	0.30	AUC_optim.4	0.007	0.783	0.775	randomForest
9	0.40	AUC_nlopt.23	0.005	0.788	0.783	randomForest
10	0.50	NNloglik	0.003	0.787	0.783	randomForest

^[1]

Table 5:

Top Metalearner performance for HIGGS datasets, as measured by cross-validated AUC (n=100,000; CV=2×2).

No.	P(Y=1)	Metalearner	SL AUC Gain	SL AUC	GS AUC	GS Model
1	0.01	NNloglik	0.027	0.754	0.727	gam
2	0.02	AUC_nlopt.19	0.021	0.766	0.745	polymars
3	0.03	AUC_optim.5	0.024	0.772	0.748	polymars
4	0.04	AUC_optim.4	0.019	0.777	0.758	polymars
5	0.05	AUC_optim.3	0.020	0.779	0.759	randomForest
6	0.10	AUC_nlopt.13	0.014	0.786	0.772	randomForest
7	0.20	AUC_nlopt.5	0.007	0.793	0.786	randomForest
8	0.30	AUC_nlopt.13	0.003	0.795	0.792	randomForest
9	0.40	AUC_nlopt.21	0.002	0.798	0.796	randomForest
10	0.50	AUC_nlopt.11	0.001	0.800	0.798	randomForest

^[2]

4.2.2 Diabetes results

The Diabetes 10,000-row dataset had slightly different results than the HIGGS datasets, however the 50,000-row Diabetes datasets had results consistent with the HIGGS datasets.

In the smaller (n=10,000) set, four of the ten datasets (with different levels of imbalance) that were evaluated achieved top performance among all metalearners using a least-squares based metalearner. This is an interesting and perhaps unexpected result, considering the assumption that AUC-maximization leads to the best results. As shown in Table 6, the dataset with highest level of class imbalance (1 %), the CC_LS and NNLS metalearners performed the best and the log-likihood based metalearner came in next, still offering benefit to using an ensemble over the top single model. As for the remaining AUC-maxiziming metalearning methods, the ensemble did not perform well on any of these − the ensemble performance was actually slightly lower than the performance of GAM. This ensmeble may not represent a typical scenario in terms of performance of the base learners − the range of CV AUCs in the base learner library was 0.5 (Polynomial Spline Regression) – 0.594 (GAM), which are very low AUC values. The rest of the ensembles in the small Diabetes datasets were found to be maximized by the log-likelihood based metalearner. Overall, the cross-validated AUC values for the base learners and the ensemble are relatively low, the highest among all ten datasets being 0.651. As there is a s very low signal in the data, that may be the cause of the contrary results.

In the larger (n=50,000) datasets, the AUC-maximization metalearners dominate among other methods. As shown in Table 7, the Super Learner trained on the dataset with the highest imbalance (1 %) benefitted most from the log-likelihood based metalearning method, however all other datasets benefitted most from AUC-maximizing metalearners.

Table 6:

Top Metalearner performance for diabetes datasets, as measured by cross-validated AUC (n=10,000; CV=10×10).

No.	P(Y=1)	Metalearner	SL AUC Gain	SL AUC	GS AUC	GS Model
1	0.01	CC_LS	0.003	0.597	0.594	gam
2	0.02	AUC_optim.3	0.001	0.608	0.606	gam
3	0.03	NNloglik	0.012	0.615	0.603	gam
4	0.04	NNloglik	0.017	0.615	0.599	gam
5	0.05	NNloglik	0.017	0.620	0.603	gam
6	0.10	NNLS	0.011	0.630	0.619	gam
7	0.20	NNLS	0.007	0.632	0.625	gam
8	0.30	NNLS	0.004	0.629	0.625	gam
9	0.40	NNloglik	0.004	0.637	0.633	gam
10	0.50	NNloglik	0.002	0.651	0.649	gam

^[3]

Table 7:

Top Metalearner performance for diabetes datasets, as measured by cross-validated AUC (n=50,000; CV=10×10).

No.	P(Y=1)	Metalearner	SL AUC Gain	SL AUC	GS AUC	GS Model
1	0.01	NNloglik	0.006	0.679	0.673	gam
2	0.02	AUC_optim.5	0.002	0.686	0.684	gam
3	0.03	AUC_optim.5	0.003	0.699	0.696	gam
4	0.04	AUC_optim.3	0.004	0.685	0.681	gam
5	0.05	AUC_optim.3	0.008	0.680	0.672	gam
6	0.10	AUC_optim.4	0.019	0.677	0.658	gam
7	0.20	AUC_optim.4	0.023	0.678	0.655	gam
8	0.30	AUC_optim.4	0.009	0.658	0.649	gam
9	0.40	AUC_optim.4	0.002	0.646	0.644	gam
10	0.50	AUC_optim.4	0.008	0.668	0.660	gam

^[4]

There is not one single metalearning method that performs best across all datasets, so it’s recommended to evaluate as many metalearners as possible. Since the metalearning step is rather fast in comparison to any of the L base learning steps, evaluating a large number of metalearners is a reasonable approach to creating the highest performing Super Learner using the available training dataset. Two utility functions were created to facilitate easy re-training of the metalearning step, without having to re-train all the base learners. The functions, now available in the SuperLearner package are called recombineSL and recombineCVSL (Figure 4).

Figure 4:

Model CV AUC offset from the best Super Learner model for different metalearning methods. (No color equals top metalearner).

Figure 5:

Model CV AUC offset from the best Super Learner model for a subset of the metalearning methods. The partially unconstrained NLopt methods are removed in order to more clearly demonstrate the differences between the box-constrained methods. (No color equals top metalearner).

4.3 Effect of class imbalance

By creating a sequence of training sets of increasing class imbalance, we evaluated the ability of the Super Learner to outperform individual models. Specifically, we measured the gain, in terms of cross-validated AUC of the ensemble, that the Super Learner provides over the top base model (i. e., the grid-search winner).

What the HIGGS results demonstrate in Table 4, Table 5 and Figure 6 is that the AUC gain achieved by using a Super Learner (SL) ensemble versus choosing the best base algorithm, as selected by grid-search (GS), is highest in the P(Y = 1) ≤ 10 % range. In particular, for the P(Y=1)=4 % dataset in Table 4, there is nearly a 0.04 gain in cross-validated AUC achieved from using an AUC-maximizing Super Learner, as opposed to selecting the top base learning algorithm, which in this case is a Generalized Additive Model (GAM). For completeness, the cross-validated grid search performance of each of the base algorithms are provided in Table 8 and Table 9 in the Appendix.

Figure 6 is a plot of the Super Learner AUC gain versus the value for P(Y=1) for both HIGGS datasets, along with a Loess fit for each group of points. With both datasets, the general trend is that the Super Learner AUC gain increases as the value of P(Y=1) decreases. However, with the n = 10,000 training set, there is a sharp decrease in AUC gain near P(Y=1)=1 %. This trend is not present in the n=100,000 dataset, so it may be a an artifact of the n=10,000 dataset, due to the global rarity of minority class examples in the training set (there are only 100 minority class observations in total). Future investigation could include examining the level of imbalance with higher granularity, and in particular, in the P(Y=1) ≤ 1 % range, using larger n.

The Diabetes results in Table 6, Table 7 and Figure 7 show a similar, though slightly less pronounced, trend. For simple comparison, Figures 6 and 7 are shown on the same scale.

Figure 6:

CV AUC gain by Super Learner (with top metalearner) over grid-search winning model on HIGGS datasets. Loess fit overlays actual points.

4.4 Effect of regularization on the base learner weights

The odd-numbered NLopt-based AUC-maximizing metalearning methods enforce box constraints − weights are bounded below by 0 and above by 1. As shown in Table 2, the even-numbered NLopt-based methods are the partially unconstrained counterparts of the odd methods, where the upper bound is ∞. The top metalearner offset heatmap in Figure 4 shows that all of partially unconstrained AUC-maximizing metalearning methods do poorly in comparison to their fully constrained counterparts.

Figure 7:

CV AUC gain by Super Learner (with top metalearner) over grid-search winning model in diabetes datasets. Loess fit overlays actual points.

5 Concluding remarks

In this article, we describe a novel implementation an AUC-maximizing Super Learner algorithm. The new AUC-maximizing metalearning functionality, as well as utility functions for efficient re-estimation of the metalearning algorithm in an existing Super Learner fit, have been contributed to the SuperLearner R package.

We benchmarked the AUC-maximizing metalearning algorithms against various existing metalearners that do not target AUC. The results suggest that the use of an AUC-maximizing metalearner in the Super Learner algorithm can, and often does, lead to higher performance, with respect to cross-validated AUC of the ensemble. The benchmarks indicate that Super Learning (which includes the AUC-maximizing metalearners) is most effective when the outcome is imbalanced − in particular, in the P(Y = 1) ≤ 15 % range. The benchmarks also show that NNLS, which has been widely used as a metalearning method in stacking algorithms since it was first suggested by Breiman, often performs worse than AUC-maximizing andlog-likelihood based metalearners.

In the current implementation of method.AUC, the weights are initialized as αinit=1L,....,1L, where L is the number of base learners. In future work, we may investigate the effect of initializing the weights using existing information about the problem, for example, let α_init = (0,...,0,1,0,...,0), where the 1 is located at the index of the top base learner, as determined by cross-validation. Further, it may increase performance to use the global optimum as input for a local optimization algorithm, although this technique for chaining algorithms together was not evaluated in our study.