Validation of machine learning ridge regression models using Monte Carlo, bootstrap, and variations in cross-validation

1 Introduction

In machine learning, linear regression is one of the most widely used techniques for building a model that predicts, or estimates, a quantitative outcome. Numerous textbooks and articles have been written on the subject. At its core, linear regression involves fitting a linear model that minimizes the sum of squared error (SSE), and then uses the linear model to make predictions on unseen data. Given a linear regression model of the form

where Y is the target or outcome variable, and X ₁, X ₂,…, X _p are the independent, or feature variables, ordinary least squares (OLS) regression will find the β coefficients that minimize the SSE.

where y i is the actual outcome and y i ˆ is the predicted outcome fitted by using the regression model for i = 1 to n observations.

However, linear regression can be prone to model overfitting. This can especially be problematic when fitting a complex model having many parameters. Model overfitting means that a regression model does not generalize well beyond the dataset on which it was trained, or as Provost and Fawcett [1, p. 113] define it, “a tendency of data mining procedures to tailor models to the training data, at the expense of generalization to previously unseen data points.” A visualization of model overfit involving linear regression is provided by Figure 1. A set of x values (n = 50) is randomly generated from a normal distribution (μ = 50, σ = 10). The y values are calculated from the quadratic function: y = x ² – 5 x + ϵ , with the error term ϵ also randomly generated from a normal distribution (μ = 0, σ = 10). The data are then fit with four different regression models: (1) a straight line (underfit model), (2) a quadratic curve (correct fit), (3) a polynomial of degree 10 (overfit model), and (4) a polynomial of degree 15 (overfit model). Figure 1 shows how models (3) and (4) overfit the data. Both models chase after noise in the data, and result in more erratic curves. When a polynomial of degree 15 is fit to the data, the curve becomes extremely erratic. Even though R ² continues to improve with higher order polynomials, the overfit models would not generalize well to unseen data.

Figure 1

Four regression models are fitted on the same set of data. On the upper left panel, a line underfits the data (R ² = 0.269). On the upper right panel, a correct quadratic model is fitted (R ² = 0.938). On the bottom left, a polynomial of degree 10 is fitted (R ² = 0.945) and on the bottom right, a polynomial of degree 15 is fitted (R ² = 0.954). Even though the R ² continues to improve, the higher degree polynomials are overfit. Source: Nakatsu [4].

One technique that can be used to deal with model overfitting is known as ridge regression. Hoerl and Kennard [2], Marquardt [3], and others originally proposed the technique as a way for a regression model to achieve better predictive accuracy in the presence of multicollinearity. The technique involves fitting a model of all predictors, like in OLS regression, but the estimated coefficients are shrunken towards zero. To accomplish this, ridge regression penalizes the β parameter estimates by adding a penalty term to the SSE in equation (2).

where i is the observation from 1 to n and β j is the coefficient of the predictor variable from j = 1 to p.

The result is that ridge regression shrinks the β estimates toward 0 as λ becomes large. Conversely, the β estimates of ridge regression are the same as the β estimates of OLS regression when λ = 0. The effect of shrinking the coefficients (a process known as regularization) is that the regression model is less prone to model overfitting. Selecting and fine-tuning the λ parameter can be accomplished by finding the ridge regression model that minimizes prediction error – in regression modeling, mean squared error (MSE) is commonly used. MSE is the average of the squared residuals or equation (2) divided by n.

A typical method is to randomly split a dataset into two sets, a training set and a validation set. A ridge regression model is fit on the training set, and then validated on the validation set. The validation process calculates the MSE over a range of λ values. The λ value that results in the lowest MSE on the validation set is chosen as the best-fitting λ. Because the validation dataset is “held-out,” this method is sometimes referred to as the holdout method.

Unfortunately, this holdout technique provides only a single estimate of a model’s validation error. The split between training and validation sets could be a particularly biased choice – even if randomized – and could either underestimate or overestimate MSE. This could especially pose a problem when dealing with smaller datasets. A way to circumvent this problem is to use resampling, which means repeatedly drawing randomized samples from a dataset and refitting the model on each sample [5]. By resampling, the average error rate of multiple runs can provide a better estimate of error rate than a single-point estimate. This is the approach that is investigated in this study. To that end, four of the most common resampling methods are compared: (1) Monte Carlo resampling, (2) bootstrap, (3) k-fold cross-validation (k-fold CV), and (4) repeated k-fold CV.

Among resampling methods, the most straightforward is Monte Carlo resampling. This method is also known as repeated learning-testing [6], repeated holdout [7], or random subsampling [8]. The method involves randomly generating training/validation splits on the same dataset multiple times. In this study, a 75%/25% (training/validation) split is used. Monte Carlo resampling can be implemented first by shuffling the dataset (i.e., randomizing the rows) and then designating the first 75% of the rows as the training set, and the remaining 25% as the held-out validation set. For each repetition, a ridge regression model is fit on the training set and then validated on the validation set. The average validation error rate is then calculated over the repetitions.

Bootstrap also randomly generates training and validation sets but, unlike Monte Carlo resampling, the rows of the training set are randomly selected from the dataset with replacement [9]. Because sampling is performed with replacement, a bootstrap sample will contain rows that are duplicated – on average 63.2% of the rows are selected; by the same token, there will be some rows that are not chosen – on average the remaining 36.8% rows are not selected. The chosen rows are designated as the training set, while the unchosen rows become the validation set. Like Monte Carlo resampling, bootstrap is intended to be run multiple times so that the average validation error across the repetitions can be used to estimate error rates more accurately.

k -fold CV is a method that was introduced in 1974 [10,11,12] and over the years has become a popular resampling method. Its popularity is evidenced in the many practitioner guides and textbooks, published within the last 10 years, that advocate its use in model validation and selection [1,5,13,14,15,16,17,18,19]. The method begins by randomly splitting a dataset into k partitions called folds (5 or 10 folds are most commonly used). Subsequently, the technique iterates k times: for each iteration, one fold is set aside as the validation set, and the remaining k−1 folds are used to train the model. The model thus built is validated on the validation set. After iterating k times this way, an average of the k validation errors is calculated so that a more accurate estimate of error rate can be obtained.

The most extreme case of k-fold CV is known as leave-one-out cross-validation (LOOCV). In this approach, a single observation is held out as the validation set and the remaining observations are used as the training set (i.e., the fold size is 1). The procedure is repeated n times, where n represents the number of samples in the dataset. The average of the n validation errors is used as the estimate for the error rate. Because LOOCV is run n times, it can become computationally prohibitive for larger datasets. In addition, prior research indicates that LOOCV results in estimates that have high variance, leading to unreliable estimates [20,21]. For these reasons, LOOCV is not further investigated in this study.

Another variation in cross-validation is known as repeated k -fold CV. Under this method, k-fold CV is repeated multiple times and the average of the multiple repetitions is used to estimate the error rate. Given n repetitions, there will be n × k validation errors; hence, the average validation error is calculated over the n × k repetitions. The most common way of running k-fold CV is only once; thus, the repeated method has been suggested by others as a way of obtaining more accurate and reliable estimates of error rates [17].

This research investigation seeks to understand the resampling methods in greater detail and offer some specific guidelines on its usage. Indeed, in recent years, there have been several calls for more practical guidelines and more transparency. In a recent paper, King et al. [22] wrote: “ML is now a key technology in modern science. However, its techniques need to be better understood. We therefore call for a dialogue between ML and domain scientists in which ML methods, such as cross-validation, can be explained to domain scientists so that they can trust and benefit from them.” We could not agree more with this statement. The current literature on resampling methods such as cross-validation is confusing, and sometimes contradictory; worse, there is sometimes no practical guidance at all. Here are some current issues.

Sometimes no guidance at all is offered in the most popular practitioner guides and textbooks. While some books do an excellent job in explaining and illustrating the resampling methods themselves [5,15,16] they offer little to no guidance on how to choose among the approaches. The practitioner is left to adopt a trial-and-error approach when choosing among the methods.

Many of the practitioner guides and studies that look at resampling arrive at conflicting recommendations. For example, some recommend LOOCV in some cases, specifically when computationally feasible [17,23,24], while others suggest that LOOCV should be avoided altogether [14,20]. The fold size k is also not well understood, and the recommendations are conflicting. The most common recommendation for fold size k is 5 or 10 [8], while others claim that other values can be used [25]. In one research study that examined the selection of SVM hyperparameters, the recommendation was to use k = 2 [26].

Finally, in a review of the research literature, it appears that single-run k-fold CV is the most popular validation method to use when tuning hyperparameters on a machine learning method. Most research studies do not even consider repeated k-fold CV, even if it could potentially benefit its results [27,28,29,30]. On the other hand, other studies have suggested that repeated k-fold CV could be beneficial [7,31], while one study recommends against its use [32].

To address the confusion surrounding resampling methods, a simulation study will look at and evaluate these different resampling methods on five different datasets, all involving a regression task in which a quantitative outcome is to be predicted from a set of features. A ridge regression model is fitted on each dataset and the regularization parameter λ is tuned using resampling. Of particular importance is how well the four resampling methods perform in selecting a good λ value. The following questions are investigated:

1. Which of the four resampling methods is most effective in selecting a suitable regularization parameter λ?

2. Does increasing the number of repetitions – from 10 to 50 – improve the performance of the resampling method?

3. Keeping the number of repetitions constant, which approach, single-run cross-validation or repeated cross-validation, performs better?

4. For k-fold CV, what is an appropriate fold size k?

5. Which randomization approach is more effective, Monte Carlo (sampling without replacement) or bootstrap (sampling with replacement)?

2 Related work

Over the last decades, several researchers have investigated and compared the performance of different resampling methods. Many of the earlier studies, especially, involve experimental studies on artificial and smaller datasets. Later studies look more systematically at variations in the four resampling methods and take on more computationally intensive approaches.

Some of the earlier studies report positive results on bootstrap’s performance. Efron [21], for example, reports on sampling experiments comparing LOOCV to bootstrap. He found that LOOCV gives nearly unbiased estimates of error but often with high variance, particularly if the sample size is small. Moreover, he found that bootstrap performed best in his experiments. However, he recommended bootstrap with caution, pending further numerical and theoretical study. Delaney and Chatterjee [33, p. 261], likewise, advocate the use of bootstrap and note that its benefits include its less subjective nature, ease of implementation, and robustness: “The bootstrap choice of the ridge parameter can be justified because it is based on repeated and independent estimates of multiple predictions and is, therefore, robust.” However, they compare bootstrap only to the ridge trace method – a method that is not addressed as this study is focused on comparisons among more general-purpose resampling methods.

Later research investigations extend the evaluation of resampling methods to k-fold CV and Monte Carlo resampling. Using a simulation study, Burman [6] evaluates three methods: LOOCV (referred to as ordinary cross-validation), k-fold CV, and Monte Carlo (referred to as repeated learning-test). His recommendation is to use k-fold CV or Monte Carlo if the computational cost of LOOCV is large. Further, with respect to fold size, he advocates the use of larger fold sizes because the bias and the variance of k-fold CV estimate decreases as the value of k increases. Regarding Monte Carlo, he does not advocate its use over k-fold CV because k-fold CV has smaller variance.

Breiman and Spector [20] look at submodel selection in regression – their task is about feature selection in a regression model, which is a different task from regularization parameter tuning, but their study’s results are illuminating, nonetheless. Their study involved an extensive simulation. They discovered that non-resampling estimates such as C _p and adjusted R ² turn out to be highly biased methods for submodel selection. According to their results, the two best resampling methods to use in submodel selection are k-fold CV and bootstrap. One of their findings was that 5-fold and 10-fold CV is better at submodel selection and evaluation than LOOCV.

Kohavi [8] reports on a large-scale experiment using classification algorithms C4.5 (decision trees) and Naïve Bayes on real-world datasets. He used resampling to estimate the effects of different parameters on these classification algorithms. He compared k-fold CV to bootstrap. For k-fold CV, he varied the number of folds and whether the folds are stratified or not (by stratification he means that the folds contain approximately the same proportion of outcome labels as the original dataset). For bootstrap, he varied the number of samples. His main result was that 10-fold CV is better than bootstrap in model selection. In addition, his results show that stratification is generally a better scheme, both in terms of bias and variance when compared to regular (non-stratified) k-fold CV. (In this study, stratified k-fold sampling is always used, because it has become standard practice in machine learning model validation.)

More recent studies have looked at more computationally intensive methods of cross-validation, including repeated k-fold CV, and compared it to more traditional approaches. Molinaro et al. [31] studied classification problems using the algorithms linear discriminant analysis, diagonal discriminant classifiers, nearest neighbors (NN), and CART. Their study used microarray datasets (in genomic studies) in which there are thousands of features (i.e., gene measurements) collected on relatively few samples (i.e., patients). The goal of their analysis was to find differences among resampling methods in the estimation of generalization error. They examined the effect of repeated k-fold CV using 2, 5, and 10 folds. Each was repeated 10 and 30 times. One of their findings was that repeated k-fold CV is beneficial: when increasing repetitions from 1 to 10, they found significant improvement in classifier performance; however, there was minimal improvement when reps were increased from 10 to 30. A second finding was that Monte Carlo did not decrease bias to warrant its use over k-fold CV. A third finding was that as the sample size grows, the differences in performance among the resampling methods decrease.

Nakatsu [34] also performed an evaluation of resampling methods on four classification algorithms: support vector machines (SVM), random forests, k-NN, and decision trees. Variations in resampling methods were used to tune parameters on the classifiers. Nakatsu found significant differences in performance among the resampling methods. No one resampling method was found to be always better than the others, but repeated k-fold CV was, overall, the best performer across all four classification algorithms.

The current study continues in the tradition of these prior research investigations. Most of the prior studies involve classification tasks, but none report on fine-tuning the regularization parameter in a regression model. Because of the popularity of ridge regression with the machine learning community, this study explores whether prior research results would hold in the ridge regression context. Second, this study investigates any unique characteristics of ridge regression modeling in which the general results might not hold.

Finally, this research only considers standard resampling methods – i.e., Monte Carlo, bootstrap, and k-fold CV. It does not investigate how to modify these resampling approaches themselves to obtain better ridge regression parameter estimates. Algamal [35] proposes a modified version of the cross-validation approach, one which repeats the fold assignment and then determines a “quantile value” that is considered as the final optimal value. In a second study [36], Algamal proposes a “kidney-inspired algorithm,” which is a population-based algorithm inspired by the kidney process in the human body. Modifying the resampling methods, themselves, is not considered in this research study.

3 Methods and materials

The simulation study looked at five datasets of varying sizes, drawn from different application domains. All the following datasets are available online:

• Baseball [5]. Major League baseball data from 1986 to 1987 seasons. Target variable: salary

• Boston [37]. Housing values in the suburbs of Boston. Target variable: median home value

• Concrete [38]. Concrete mixtures and their compressive strengths. Target variable: compressive strength.

• Parkinson’s [39]. Biomedical voice measurements from 42 people with early-stage Parkinson’s disease. Target variable: Total-Unified Parkinson’s Disease Rating Scale (UPDRS).

• Superconductor [40]. Superconductors, and their properties. Target variable: critical temperature.

A ridge regression model was fit, separately, on each of the five datasets. Each modeling task involved the prediction of a numeric outcome from a feature set. The datasets range in size from n = 263 to 21,263 rows and from p = 9 to 81 features. They also vary in terms of subject matter, including a dataset from business (Baseball), socioeconomics (Boston), medicine (Parkinson’s), materials science (Concrete), and physical science (Superconductor). The intent was to develop generalized findings across a range of datasets in terms of size and problem-solving domain. Furthermore, the goal was to demonstrate these findings on real-world datasets, not simulated data.

3.1 Determining good fit, average fit, and poor fit λ values

To determine how well the four resampling methods performed, ridge regression was run multiple times to determine good fit, average fit, and poor fit values of λ on each of the five datasets. First, 10-fold CV was performed 50–100 times on each of the 5 datasets to obtain a rough sense of where the optimal value of λ lay, based on MSE. Then, a determination was made as to what an appropriate range of λ values would be to look at further. For example, on the Boston dataset, it was determined that the optimal λ value would be low, so λ values ranging from 0.01 to 1.0 (in 0.01 increments) were tested. For the Baseball dataset, a different set of λ values was used – from 0.2 to 20.0 (in 0.2 increments). To fine-tune the MSE associated with each of the λ values, 10-fold CV was repeated multiple times until we could discern what the good fit, average fit, and poor fit values were. Figure 2 shows the resulting data plots when 10-fold CV was repeated. Figure 2(a) shows the average MSE on the Boston dataset after 500 reps of 10-fold CV were run. You can see that there is still noise in the data after 500 reps. When the number of reps was increased to 10,000, we were confident where the best-fit values lay, as the MSE data converges to follow a curved line more closely, Figure 2(b).

Figure 2

Average MSE over a range of lambda (λ) values. The four plots show average MSE for a range of λ values. Average MSE is calculated over multiple reps of 10-fold CV. Plot (a) shows average MSE for 500 reps on the Boston dataset. Plot (b) shows average MSE for 10,000 reps on the Boston dataset. With more reps, the data in Plot (b) has less noise and approaches a curved line. Plot (c) shows average MSE for 75,000 reps on the Baseball dataset. Plot (d) shows average MSE for 2,000 reps on the Parkinson’s dataset.

Once an MSE curve was found, the following criteria were used to determine the fit type: (1) the top 20% λ values (i.e., associated with the lowest MSE values) were designated as a good fit; (2) the middle 60% λ values were designated average fit; and (3) the bottom 20% λ values (i.e., associated with the highest MSE values) were designated as poor fit. The good fit and poor fit intervals are indicated in Figure 2(b); all remaining λ values are designated average fit.

The same procedure and criteria were applied to the other four datasets to determine good fit, average fit, and poor fit λ values. For example, Figure 2(c) shows the MSE curve for the Baseball dataset when 75,000 reps of 10-fold CV were performed, as well as the designation of good fit and poor fit values on the graph. Good fit values lay between λ = 4.6 and λ = 8.4; poor fit values lay between λ = 0.2 and λ = 2.0 as well as between λ = 18.2 and λ = 20.0; all other values were designated average fit. Finally, Figure 2(d) shows the MSE curve for the Parkinson’s dataset, where, again, good fit and poor fit intervals are indicated. Note that the Parkinson’s dataset required only 2,000 reps to converge – this can be seen from the smooth curve in Figure 2(d) – whereas the Boston and Baseball datasets required significantly more reps, 50,000 and 75,000, respectively.

It is important to underscore that even though repeated 10-fold CV was used to determine the good fit, average fit, and poor fit λ values, the same solution could be arrived at using multiple iterations of either Monte Carlo or bootstrap. This issue is addressed in Appendix A, which illustrates how Monte Carlo resampling would arrive at the exact same solution as that found by repeated k-fold CV. The reason repeated 10-fold CV was used instead is that it arrived at the final solution much more efficiently than either Monte Carlo or bootstrap could.

4 Results

The main results of the study are presented in Table 1. The table summarizes how well each of the nine resampling approaches performed over the five datasets, in terms of how often a good fit, average fit, and poor fit λ was chosen. The methods are grouped according to whether they are 10-iteration approaches or 50-iteration approaches so that baseline comparisons can be made within each of these groups.

A first glance through Table 1 reveals significant differences in performance among the resampling approaches. For example, in looking at the Parkinson’s dataset (fourth row of Table 1), Monte Carlo and bootstrap methods performed poorly – for the 10-iteration approaches, they selected a good fit λ value only 27.6 and 27.7% of the time, respectively, whereas the 10-fold CV and 5-fold CV 2 reps approaches performed dramatically better, selecting a good fit λ value 82.3 and 79.6% of the time, respectively. Also, indicated in the table is the single best performer (denoted by **) or tied best performer (denoted by *). This is indicated, separately, for the 10-iteration approaches and the 50-iteration approaches. The difference between “single best” and “tied best” is that a resampling approach is deemed single best if the Chi-square statistic between the top performer and second-best performer is significant (p < 0.05), whereas a tied best performer means the statistic is not significant. You can quickly observe in Table 1 that the best performers are consistently k-fold CV and repeated k-fold CV for both 10-iteration and 50-iteration approaches; Monte Carlo and bootstrap are never the best performers across all five datasets.

In this section, five specific questions regarding differences in performance in the resampling approaches are explored in more detail:

1. Did performance increase when reps (iterations) were increased from 10 to 50? (Section 4.1)

2. What was the effect of different values of k in k-fold CV? (Section 4.2)

3. Keeping the number of iterations constant, which method was superior, single-run CV or repeated CV? (Section 4.3)

4. Which method of randomization was superior, Monte Carlo or bootstrap? (Section 4.4)

   To compare the performance of one resampling approach to another, Pearson’s Chi-square test is calculated. This statistic compares the frequencies observed in certain categories to the frequencies you might expect to get in those categories simply by chance. In this analysis, if the statistic is significant (p < 0.05), then there is a significant association between the resampling method and its performance (i.e., how often it chooses good fit, average fit, and poor fit). Refer Tables 2–5 in the Sections 4.1–4.4 for the Chi-square test results.

   Finally, the possibility that Monte Carlo and bootstrap require a large number of iterations to perform well was explored. The main study, as presented in Table 1, limits the number of repetitions to a maximum of 50. Section 4.5 addresses the following question:

5. Which resampling method performed best when performing 1,000 repetitions?

4.2 Effect of k (number of folds) on k-fold CV

The effect of k can be seen by comparing col 3 to col 7 in Table 1. The Chi-square tests (see col 3 of Table 2) show that increasing k from 10 to 50 folds resulted in worse performance for the Baseball and Boston datasets (these results are indicated by ↓, which signifies that MSE performance moved in the opposite, predicted direction). These results were significant for both the Baseball dataset (decreasing from 30.0% good fit to 23.7% good fit) and the Boston dataset (decreasing from 63.5 to 51.3% good fit). On the other hand, performance was significantly improved on the Concrete dataset (up from 96.5% good fit to 98.5% good fit) and on the Parkinson’s dataset (up from 82.3% good fit to 91.6% good fit). These results suggest that increasing the size of k can have mixed results. For larger datasets (in this case, datasets greater than 1,000 rows), performance was improved, but for smaller datasets increasing k beyond 10 folds resulted in poorer performance.

We also compared the performance differences between 5 folds and 10 folds, and the results are displayed in Table 3. For 10-iteration resampling, 5-fold CV 2 reps is compared to 10-fold CV. In all cases, 10-fold CV performs better (see cols 3 and 4 of Table 1), although it is only significant in one case (Boston). For 50-iteration resampling, 5-fold CV 10 reps is compared to 10-fold CV 5 reps. 10-fold CV 5 reps outperformed 5-fold CV 10 reps in all cases (see cols 8 and 9 of Table 1), and these results were significant in two. Hence, the overall recommendation is to use 10 folds rather than 5 folds because 10-fold CV does at least as well and often better than 5-fold CV in all cases studied.

To get a better sense of how the size of k affected performance, we looked more extensively at the Boston dataset, and ran the evaluation on resampling approaches (i.e., running k-fold CV 1,000 times) on a range of k sizes from 10 to 100 folds. Figure 3 graphically displays these results. First, let us focus only on the line labeled “Single-run k-fold.” You can see that performance is highest with a k size of 10 and 20, and then it gradually declines for larger values of k. Again, this shows that k = 10 is a good choice, and that increasing k beyond 10 for smaller datasets did not result in better performance (the difference in performance between k = 10 and k = 20 is insignificant). On a related note, we do not believe that LOOCV is ever a good choice for fine tuning a λ parameter in ridge regression. Much better results (with far less computational costs) can be achieved with fewer folds.

Figure 3

Comparing the performance of repeated 10-fold CV, single-run k-Fold CV, and Monte Carlo on the Boston dataset. The top line (repeated 10 fold) shows how the number of reps increased performance (% good fit) from 2-reps to 10 reps of 10-fold CV on the Boston dataset. The middle line (single-run k-fold) shows how performance decreases as fold size increases from k = 10 folds to k = 100 folds. The bottom line (Monte Carlo) shows how performance gradually increases with more reps of Monte Carlo resampling, but in all cases, Monte Carlo is the lowest performing resampling method.

5 Discussion

Based on the results of this study, three broad recommendations on resampling methods can be made when choosing the right regularization parameter λ in ridge regression modeling (the underlined portion is the broader recommendation).

Recommendation 1: For k -fold CV, the general recommendation is to use k = 10 folds; larger values of k can be used on larger datasets. LOOCV is never recommended. For smaller datasets, performance deteriorated with larger numbers of folds (greater than 10). For larger datasets greater than 5,000 samples, 50 folds performed better than 10 folds, but performance with 10 folds was respectable, and achieved at 20% the computational cost. LOOCV (where k = n) is never recommended.

Recommendation 2: Repeated k -fold CV is the best overall performer among the four resampling approaches. More specifically, we recommend 10-fold CV 5 reps as a general choice for ridge regression. For larger datasets, the differences in performance are less pronounced between repeated k-fold CV and single-run k-fold CV.

Recommendation 3: Monte Carlo and bootstrap are not recommended as general-purpose resampling methods. These two resampling methods were consistent underperformers in Study 1, across all five datasets studied. In addition, these two resampling methods did not perform better than repeated k-fold CV even when using a large number of reps (i.e., 1,000 reps).

Recommendation 1, which recommends k = 10 folds, agrees with conclusions reached by Kohavi [8] and other studies – the consensus in the literature is 5 or 10 folds. Furthermore, as has already been noted in the article, LOOCV is not recommended by many researchers because it results in estimates with high variance, leading to unreliable results. Flach [14] recommends that, as a rule of thumb, a fold should contain a minimum of 30 instances. The recommendation against LOOCV, however, is not universal and unanimous – for example, Kuhn and Johnson [17] do not write it off completely but state that LOOCV can be considered if the number of samples is small. Our recommendation is more direct and straightforward: we never recommend LOOCV, not only because it is computationally inefficient, but also because it provides inferior results compared to repeated k-fold CV. A better way to achieve more accurate estimates of validation error is not to increase the number of folds beyond 10, but to increase the number of repetitions in 10-fold CV.

Recommendations 2 and 3 are largely in line with recent empirical work by Molinaro et al. [31], and Nakatsu [34]. Repeated k-fold CV has been recommended as a way of obtaining more accurate estimates of validation error than single-run k-fold CV. However, these two studies focus exclusively on classification algorithms, not regression modeling, and we are unaware that this result has been demonstrated elsewhere on ridge regression modeling. Furthermore, even though some prior research has demonstrated the effectiveness of repeated k-fold CV, most practitioner books and textbooks do not mention the method – one exception is Kuhn and Johnson [17] who do recommend the method. However, we want to qualify Recommendation 2 by noting that it was not always necessary to increase repetitions. For example, on the larger Superconductor dataset (n = 21,263), it was unnecessary to conduct any repetitions of 10-fold CV at all. For larger datasets, or datasets that quickly converge to a solution, single-run 10-fold CV should suffice.

For Recommendation 3, we have found that Monte Carlo and bootstrap consistently underperform in ridge regression (although in other machine learning algorithms and contexts, including classification algorithms, this may not be the case). Moreover, even though we recognize that Monte Carlo and bootstrap may require a higher number of iterations to perform well, we did not find that they performed better than repeated k-fold CV when using a high number of reps.

Would these results hold and generalize when used on different datasets? To confirm and verify the results of this study, a second independent study was conducted. The results are reported and discussed in a separate appendix, which are available upon request from the authors. In this study, the exact same methodology was used, in which we evaluated nine variations of the four resampling methods. We can report that all three recommendations were verified by the results of the second study. Hence, we are more confident that the recommendations will generalize more broadly, even though we conducted Study 1 on only five datasets.

A key feature of the study’s methodology is that it controls for the number of times a ridge regression model was run, to provide baseline comparisons among (1) 10-iteration approaches and (2) 50-iteration approaches. For example, when looking at the effect of fold size on resampling method performance, 10-fold CV 5 reps is compared to 5-fold CV 10 reps (both 50-iteration approaches). Most prior studies simply compare different fold sizes without controlling for overall repetitions (e.g., [31]). Likewise, when comparing single-run CV to repeated CV, appropriate baseline comparisons are performed, for example, 10-fold CV vs 5-fold 2 reps (both 10-iteration approaches) and 50-fold CV vs 10-fold 5 reps (both 50-iteration approaches). We know of few studies that controlled for number of repetitions as systematically and consistently as we did. In the end, we believe that this approach bolsters the final recommendations and conclusions of the study.