A mixed effects multinomial logistic-normal model for forecasting baseball performance

2.1 Literature review

Most published work in sports prediction focuses on univariate, binary on-field performance metrics. One of the earliest examples comes from Lindsey (1959), who utilizes proportion tests to predict a baseball player’s batting average based on the handedness of the batter-pitcher match-up. Subsequently, in a seminal paper, Efron and Morris (1975) use Stein’s estimator via empirical Bayes to predict batting averages for baseball players for the remainder of the 1970 season. Additional works include Bennett and Flueck (1983), Albright (1993), Albert (1994), and Rosner, Mosteller, and Youtz (1996).

More recently, there has been a broader and more rich tapestry of statistical models applied to the problem of prediction in baseball and other sports, including a greater emphasis on hierarchical and non-parametric models. Berry, Reese, and Larkey (1999) use a hierarchical model to compare home run abilities across eras. They applied the same model, separately, to batting average. Chandler and Stevens (2012) use random forests to project future success of minor-league baseball players, where their metric for success was simply the binary outcome of playing in greater than 320 Major League games or not. In a non-baseball application, Wickramasinghe (2014) focuses on predicting average runs for batsmen in cricket via a hierarchical linear mixed model.

Within this realm of published research, there has been little focus on developing models that seek to predict the entire vector of plate appearance outcomes. The three most notable papers to tackle this issue are by Null (2009), Albert (2016) and Powers, Hastie, and Tibshirani (2018). Null (2009) defines 14 distinct outcomes for a plate appearance and assumes each batter has some “true” probability vector for experiencing these outcomes. Null’s work also serves as an introduction to the class of nested Dirichlet models, used as an alternative to the Dirichlet or Generalized Dirichlet for modeling multinomial data.

Albert (2016) develops a Bayesian random effects model for estimating component rates of batting outcomes in order to better predict batting average. Albert’s work treats the outcomes as multinomial in nature, but factorizes the multinomial likelihood into a product of conditional binomial likelihoods, and assigns independent prior distributions to the probabilities of each outcome. Between Null (2009) and Albert (2016), only Null attempts to include covariates, using a fixed effects regression model to capture an age effect.

Powers, Hastie, and Tibshirani (2018) seek to estimate multinomial probabilities while accounting for relative strength of the batter and pitcher. They focus on individual plate appearances, utilizing play-by-play data. The nuclear penalized multinomial regression approach they develop, similar to Null (2009) and Albert (2016), defines a nesting structure of batting outcomes which they leverage to allow for positive associations among outcome probabilities and produce better estimates, though they find the key benefit in their model is in the ease of interpretation.

Outside the realm of academic journals, there is an ample and active sector of the Internet where baseball projection systems are implemented. Most projections from these models are published each season on the Internet. Unfortunately, only one of the most common projection systems is completely open source, with all others maintaining some proprietary ownership of the methodology. Not only that, but there is a dearth of uncertainty quantification associated with the resulting projections, which would allow for comparing the “floor” and “ceiling” of different players.

One of the only systems with some level of uncertainty assessment is the Marcels (Tango 2004). The Marcels is also the only well established, completely open source projection system, with the model details available online (Tango 2004). In brief, the Marcels consists of regressing a weighted average of their last three seasons to the mean. Details, including the adjustment for age, are described in Appendix C. In terms of uncertainty, the Marcels provides a “reliability score” for each prediction; simply the proportion of the prediction which comes from player data as opposed to the regression to the mean. Players with no data, including minor league players and players coming from Japan, will be predicted at the mean and have a reliability score of 0 (Tango 2012).

Because of the lack of open source methods as well as a lack of uncertainty quantification, we think adding a model to the lexicon of published sports prediction is a useful contribution, especially considering the heretofore unexplored application.

2.2 Data description

Our population of interest in this study is the set of players to play in both NPB and MLB. The data used in this project come from two main sources. Nippon Professional Baseball player-season data were provided by Data Stadium, Inc. Major League Baseball player-season data for relevant players, as well as their corresponding covariates, were taken from the Lahman database available in R (Friendly 2015). Some data augmentation was necessary to collate the two data sources. Some artifacts of the different data sources, such as slight differences in height and weight for the same players, still remain. We also consider players switching between leagues in the middle of a season as two seasons of data.

The data consist of all players who have played in both NPB and MLB before 2015. For the batting data, this includes 605 players with 4991 player-seasons. Of the 605 players, 50 made their initial transition from NPB to MLB, while 555 made the reverse transition. We limit our data to those players making the switch for two main reasons. First, we do not have easy access to all seasons of NPB data. Second, considering players moving from NPB to MLB tend to be among the best in NPB, there is an argument to be made that the players do not belong in the same population. A similar argument could be made that MLB players transitioning to NPB do not belong in the same population as all MLB players. A potential adjustment to the model, which could allow for inclusion of all players from both leagues, given the data, is suggested in the conclusion.

While it is possible to fit a pitching outcome model using this approach (i.e., replace at bats with batters faced), we focus only on modeling hitter outcomes here, though we do include batting outcomes of pitchers and include pitcher as a positional variable.

Choosing the granularity of the count vectors depends on the nature of the data collected and the goals of the research. We consider the following 10 outcomes: base hits (Single, Double, Triple, Home Run), free passes (Base on Balls, Hit-by-Pitch), sacrifices (Sacrifice Bunt, Sacrifice Fly), and outs (Strikeout, Out-in-Play). These give enough information to describe a player’s offensive performance, while being sufficient for calculating summary statistics, such as batting average ( H A B ) or on-base plus slugging percentage (OPS) ( H + B B + H B P + T B A B ) . The individual base hit categories allow us to calculate total bases (TB) which is used in the calculation of OPS. The two free pass categories are likewise used in calculating OPS, and while the sacrifice categories are not, at-bats (AB) are made up of base hits and outs, and thus distinguishing sacrifices becomes necessary. In developing and testing the methodology, a smaller three outcome model was also used, some results from which are discussed in Appendix B.

Among the covariates available, those chosen were based on what we considered to best explain variability in performance. Height, weight, age, and batting handedness all could reasonably impact the performance of certain types of players; heavier players, for example, will be less likely to be slap hitters looking to take advantage of their speed. A quadratic age term was included, as there have been some studies which suggest the career trajectories of players’ performance are quadratic in nature (Albert 2002; Fair 2008). Berry, Reese, and Larkey (1999) use random spline curves to model individual player aging effects, though these too reflect a somewhat quadratic relationship between age and performance.

Including league as a fixed effect makes sense in that these are the only leagues we are interested in for this study. In this case the two MLB leagues American League (AL) and National League (NL), and two NPB leagues Central League (CL) and Pacific League (PL). We distinguish between these four because of the differing rules and styles of play; the AL and PL both have the Designated Hitter rule, while the NL and CL do not.

Finally, we would expect that position could explain some variability, especially given a typical batting line for a pitcher. For the analysis, since some of the player-seasons do not distinguish player’s primary position outside of infield, outfield, designated hitter or pitcher, those are the four levels of the covariate we utilize, where: Infield (C, 1B, 2B, SS, 3B), Outfield (LF, CF, RF), Designated Hitter, and Pitcher. An excerpt of the data set, showing a single player’s covariate and response values, is provided in Table 1.

3.1 Model

There have been several studies examining streaky hitting in baseball (Albright 1993; Albert 2008, 2013) with the general consensus that there is not a great deal of difference between a player’s behavior and a model that treats a player’s seasonal plate appearances as a set of independent trials. Thus, we might assume we have vectors of count data for players i = 1, …, m over seasons j = 1, …, n _i :

Each vector W _ij is length d + 1, the number of outcome categories. We denote the exposure (i.e., plate appearances) for a player-season as PA _ij , so that we can express the total number of count vectors as N = ∑ i = 1 m n i . Our desire is to predict the counts of a new player i ′ for season j = n i ′ + 1 , the first season after switching leagues. In our data, we have m = 605 players, N = 4991 player-seasons, and d + 1 = 10 categories.

Multinomial logistic regression, also known as the multinomial logit, is the most common form of analysis with a multinomial distributed response. The multinomial logit is an extension of binary logistic regression, and for a response vector of length d + 1, can in its most basic form be formulated as d “independent” logistic regressions with one category serving as a reference category.

The multinomial distribution, however, has the unfavorable property that there is a negative correlation between each pair of outcome counts. This is problematic because there are several outcomes in baseball that are known to be positively correlated. In addition, although we use covariates to allow the mean probability to differ across player-seasons, we expect there to be overdispersion. One model commonly used to account for overdispersion is the multinomial Dirichlet, but this model still imposes negative correlation. To account for overdispersion and allow for some positive correlations, we consider a compound distribution for the data, the multinomial logistic-normal distribution.

Specifically, we assume the following hierarchical structure for player i and season j:

where

and f(Π _ij ) is the logistic-normal distribution. Using Y _ij to represent the vector of log-odds:

We incorporate the covariates through the linear model:

with ψ _i  ∼ N _d (0, Φ), and ε _ij  ∼ N _d (0, Σ).

The random effect term, ψ _i , is a d-dimensional vector included to account for the longitudinal nature of the data (i.e., consecutive seasons from the same player). The inclusion of the error term ε _ij , and subsequently the covariance matrix Σ, differentiates this model from the multinomial logistic, thereby allowing positive correlations among categories. A more detailed description of the resulting covariance structure is provided in Appendix A.

We take a Bayesian inferential approach to model estimation. For prior distributions, we consider an improper flat prior on the fixed effect matrix and noninformative inverse-Wishart priors on the two covariance matrices. These prior distributions, explicitly defined in Appendix A, are as disperse as possible, allowing the data to drive the posterior distributions. We utilize a Metropolis-within-Gibbs sampler algorithm for fitting the model, due to the intractability of the multinomial logistic-normal conditional posterior likelihood.

As a baseline model, the multinomial logistic (M-Logit) model will be compared to our multinomial logistic-normal (MLN) model. The M-Logit may be formulated as the MLN above, but with the absence of the overall error term, ε _ij . In the Bayesian framework of estimation, the difference in our models amounts to the MLN adding a sampling step for the covariance matrix Σ. Mixed effects M-Logit models have been addressed previously in both frequentist (Hartzel, Agresti, and Caffo 2001; Hedeker 2003) and Bayesian (Pettitt et al. 2006) settings. To our knowledge, the Bayesian mixed effects MLN model presented here has never been used.

3.2 Inference

For the MLN model described in Section 3.1, there are five sets of parameters within four iterated Metropolis-within-Gibbs steps that are to be estimated. The details of the algorithm are provided in Appendix A, while a summary of the parameters follows:

1. Y _ij : the length d latent player-season log-odds vectors.

2. β: the p × d matrix of fixed effects of the covariates.

3. ψ _i : the length d player random effect vector.

4. Φ: the d × d random effect covariance matrix.

5. Σ: the d × d error covariance matrix.

The Metropolis-within-Gibbs sampler results in long chains of samples of the parameters. After discarding a burn-in and thinning, these chains can be considered independent samples from the posterior distributions of the parameters, and are subsequently used in prediction (see Section 3.3).

The M-Logit model parameters, which are the same as the MLN parameters aside from Σ, are also estimated via Metropolis-within-Gibbs. The details of the M-Logit algorithm follow those of the MLN, in Appendix A.

3.3 Prediction

Suppose an NPB team is in need of a catcher and is considering Kevin Plawecki, former Purdue catcher and current catcher with the Boston Red Sox. Because he has only played in the major leagues (and not NPB) we do not have information regarding his player-specific ability (ψ _i ) and therefore cannot make an accurate prediction of his performance if he were to transition to NPB. This section discusses how we approach this given that we have already generated samples from the joint posterior distribution of β, Σ, and Φ.

To accurately predict this player, we need an estimate of the ψ i ′ . We could add his output vectors to our data set and rerun the Metropolis-within-Gibbs sampler, but this is going to take time. Because we have samples from the posterior distributions of y _ij , β, ψ _i , Σ, and Φ, and we do not expect the inclusion of one player to affect these, a faster alternative is to rerun the Metropolis-within-Gibbs sampler holding the β, Σ, and Φ quantities as fixed. In other words, only estimating the latent variables and random effect for player i ′ .

By cycling through joint samples of β, Σ, and Φ, we obtain a corresponding set of posterior values of the new player’s random effect, ψ _i’ , which we may use predict his performance. This is a useful approach when we want to quickly generate predictions for new players. A comparison of the predictions based on refitting the full data set and this approach was made and showed little difference. Thus, one might consider updating the model annually and using this approach to make predictions for players of interest during the year.

In general, all predictions follow the framework of first simulating draws from the posterior distributions of the parameters β, ψ _i’ , Σ, and Φ. Then, use those samples to generate a posterior predictive distribution of new player-season log-odds y _i’j based on the parameter samples and covariate vector x i ′ j , and subsequently sample a posterior predictive distribution of counts W _i’j . When interested in only few new players, however, the above method of fixing the β, Σ, and Φ allows for faster sampling of the operative quantity, the new player’s random effect, ψ i ′ .

The main advantage of the Bayesian hierarchical model in this context is that we have generated samples from a posterior predictive distribution which will allow us to create prediction intervals for the outcome probabilities of a new player-season. These intervals allow us to not only compare the reward (the posterior mean) but also the risk (the uncertainty) for players moving from one league to another. These intervals will likely be too wide to be informative on an individual player basis. Several papers have discussed that simultaneous confidence intervals about multinomial counts tend to be quite wide (Quesenberry and Hurst 1964; Sison and Glaz 1995), and our hierarchical model accounts for further variability to allow for positive correlation. However such intervals are a quite useful tool for comparison of two or more players, at least in terms of relative floor and ceiling of player ability across categories of interest.

Based on pairs of posterior samples, t = 1, …, T, a prediction interval for a new player-season vector may be constructed via selecting the 95% most likely random effects for a new player under the posterior distribution of random effects ψ i ′ t , calculating the expected log-odds for the new player-season X i ′ ( n i ′ + 1 ) β t + ψ i ′ t , and finally transforming those into probabilities. Multiplying by the exposure represents an interval for the vector of expected counts. The same process may also be used to create prediction intervals for singular metrics, such as batting average (AVG) or on-base plus slugging percentage (OPS), which might be of interest for comparisons sake as a summary of player ability.

3.4 Assessing model fit

4.1 Model fit

The three category model serves as a test case to illustrate the benefits of the MLN model over the M-Logit, the details of which are provided in Appendix B. However, since our main interest is the full 10 outcome set, which provides a more complete picture of player ability and allows for computation of summary statistics, we report those results here.

We begin by comparing model fit of the 10 category training set with MLN and M-Logit models with DIC and then the Dunn–Smyth residual plots (Figures 1 and 2). We find that the model fit diagnostics are generally inconclusive on the most appropriate fit for the 10 category results. The DIC supports the M-Logit as the more appropriate model, with values of 141057 for M-Logit and 145797.5 for MLN. DIC favors M-Logit largely due to the sizable difference in number of effective parameters; the MLN model had over four times as many as M-Logit (10439.6–2225.3).

Figure 1:

Histogram and standard normal QQ-plot of standardized MD ² percentile (Dunn–Smyth style) residuals for MLN fit, ten categories.

Figure 2:

Histogram and standard normal QQ-plot of standardized MD ² percentile (Dunn-Smyth style) residuals for M-Logit fit, ten categories.

The standardized MD ², Dunn-Smyth type residual normal QQ-plots for the two model fits do not support one model over the other (Figures 1 and 2). Neither pointwise confidence envelope entirely includes the reference line that indicates adequate model fit. In some sense, it seems the data falls somewhere between MLN and M-Logit; in other words, that some additional variability exists, but not as much as the MLN model is recovering. This suggests the bias we see in the MLN fit residuals is due to an overestimation of Σ, which will also be reflected in an excess of variability in predictions. In the future a more informative, less sparse, prior on Σ may be more reasonable here.

Since DIC favors the M-Logit while the residual plots seem to not favor either, we continue to consider both. In a statistical sense, it is generally prudent to be more conservative when estimating variability, as the MLN model seems to, especially when there is evidence that the alternative model, the M-Logit, underestimates it. However, our main goal is prediction, and we show in Section 4.3 that the MLN model does a slightly better job in that goal than the M-Logit.

4.2 Estimation

With p = 13 (including the intercept) and d = 9, the resulting β matrix contains 117 terms for each of the model fits. In the interest of brevity, we will highlight some of the more notable results from parameter estimation of the MLN fit.

The most interesting covariates for inference’s sake are the aging curve and the league indicator variables. While the age effects are not among the largest in magnitude, they show some interesting trends, for instance that for a baseline player (average height, weight, right handed, American League, infielder) home run rate does not, on average, reach it’s peak until past the age of 30 (Figure 3).

Figure 3:

Posterior distribution (grey) and mean (black) of baseline player home run rate age trend per 500 plate appearances for MLN fit, ten categories.

In terms of the league effect, we can summarize player ability via OPS and compare how a baseline player (average height, weight, age) is expected to perform in each league by position and handedness (Tables 2 and 3).

Generally, we see that the NPB leagues (Central and Pacific) are easier to hit in than the MLB leagues (American and National), with the Pacific League producing the highest mean OPS across the board. What might be surprising is the relative weakness of the Designated Hitter position, and that the Pacific League pitchers (who do not usually bat, since the PL adopted the DH in analog with the AL) perform better on average than the Central League pitchers; as opposed to in the MLB where NL pitchers, who hit regularly, perform better than AL pitchers. It is also interesting that left-handed hitters in the CL do not appear to have quite the same advantage, when compared to the MLB leagues, as right-handed hitters.

Most of the covariance terms (26 of 36) recovered from Σ have posterior distributions which do not cover zero. Since the presence of Σ is the key distinction between MLN and M-Logit, allowing for positive correlation where M-Logit assumes negative correlation, investigating the implied correlation structure among the outcomes is telling of the MLN model’s utility. Almost all of the recovered implied correlations between outcomes for a baseline player make practical sense. The strongest positive correlations are 0.19 between (HR, SF), (1B, OIP) and (HR, BB). The strongest within season correlation is the −0.69 between (SO, OIP), followed by the −0.52 between (1B, SO). Other correlations of note are the positive correlation between (HR, SO), 0.12, and the negative correlation between (3B, HR), −0.04.

These make quite a bit of sense; striking out and putting the ball in play are exactly opposite outcomes, and players who hit a lot of singles are likely putting the ball in play often as well, instead of striking out. Meanwhile, players often attempt to hit home runs when runners are on third base, assuming that even if they don’t get all of the ball, a sacrifice fly will net the team a run. Players who hit home runs have also long been considered to also be walk and strikeout prone, while also tend to be slower and less likely to hit many triples.

Before moving to prediction of out-of-sample players, we note that there is some evidence of strong shrinkage of the recovery of random effects for those players with little information being used to estimate random effects. Comparing the estimated OPS to observed OPS for qualified batters in the initial model fit, there is a general underestimation (Figure 4). This is largely due to many players who switch from MLB to NPB with little information being predicted close to the mean due to strong shrinkage in random effect recovery, and then outperforming that prediction. However, comparing out-of-sample predictions of NPB to MLB players with an established prediction method, the Marcels, shows that even with this shrinkage, our method performs competitively (see Section 4.3.1).

Figure 4:

OPS estimation for all qualified batters (PA > 500) based on MLN fit of training set, ten categories.

4.3 Prediction

To assess prediction, we focus on the players whose careers were held out of the model fit. Figure 5 contains three plots of observed versus predicted outcomes (on the square root scale) that vary in terms of frequency. More frequent outcomes—that is, 1B, 2B, HR, BB, SO and OIP—show a fairly tight spread around the 45° line. Less frequent outcomes—that is, 3B, SH, SF, HBP—show a tendency to underpredict. This is perhaps more clear in Figure 6 where we convert each observed count into a Dunn–Smyth type residual and plot them versus the fitted values. Overall, the model performed well with some slight issues predicting the rare outcomes.

Figure 5:

Square root of predicted versus observed counts for three categories (3B, HR, SO) of test set players based on MLN fit, ten categories.

Figure 6:

Standardized MD ² percentile (Dunn–Smyth style) residual plots for three categories (3B, HR, SO) of test set players based on MLN fit, ten categories.

The MLN also predicts slightly better than the M-Logit model. We attribute this to the added flexibility of the MLN to handle positively correlated outcomes. For example, the correlation between predicted and observed OPS for the held out players was 0.526 for MLN and 0.513 for the M-Logit.

As far as players transitioning between the leagues, Figure 7 shows the prediction intervals of OPS for four players along with their observed value. These players were randomly selected for the prediction set, but are chosen here for comparisons sake. Mike Young and Rod Allen both moved from the AL to CL at the age of 30, predominantly playing OF. However, Young played nine years in MLB with 235 PA/year as opposed to Allen, who played only three years with 18 PA/year. Thus they represent very similar players in terms of covariates, but disparate career lengths and information. Tsuyuoshi Shinjo and Kosuke Fukudome both played many years in Japan before moving to MLB, with similar numbers, which on the surface may make choosing between them, should a team have to, more difficult.

Figure 7:

OPS prediction intervals with posterior means (black circle) and observed value (red cross) for four players under MLN fit, ten categories.

Allen’s prediction interval is wider than Young’s, reflecting the additional uncertainty in recovery of his random effect. While Allen’s OPS was not predicted well by the model fit, it was still within the rather wide prediction interval. Though Shinjo and Fukudome had similar careers in Japan, and a similar uncertainty about their predicted OPS, Fukudome had a higher floor, ceiling, and predicted OPS than Shinjo. Subsequently, he also performed at a higher level (according to OPS) than Shinjo in his first year in MLB.

There is quite a bit of uncertainty in forecasting a player’s seasonal performances, somewhat limiting the usefulness of these intervals. However, for our purposes, these intervals can be used to provide a comparable level of risk associated with each player.

4.3.1 Comparison to the Marcels

Pre-2015, the time frame which our data set covers, there were only 50 players to move from NPB to MLB, 23 of which were position players. For players with no previous major league experience, such as NPB players moving to MLB, the Marcels predicts those players at a weighted league average of the three previous seasons, with an adjustment for age. While the Marcels will provide slightly different predictions for each player (given different age and season) there is no prior player performance information being used in generating the predictions. Since there have been no published NPB predictions, and for this project we did not have easy access to the full NPB data set so we cannot create a “Japanese version” of the Marcels, and comparing predictions for the 555 players to make the transition from MLB to NPB becomes infeasible.

We generate posterior predictive samples under the MLN model for the 23 position players to make the NPB to MLB transition, and calculate the posterior predictive mean of those samples as the point prediction to compare to the Marcels predictions for those same players in the first season after making the transition. One player (Andy Abad for the 2001 Athletics) had only a single plate appearance in their first year in MLB, so we ignore him and focus on the other 22. Figures 8–10 plot the observed versus predicted counts (on the square root scale) for both the MLN model and Marcels in three of the outcomes. In general, the MLN model provided a tighter fit around the 45° line. Table 4 summarizes the correlation and predicted residual error sum of squares (PRESS) statistic for each of the 10 outcomes plus four other common statistics.

Figure 8:

Comparison of MLN (left panel) and Marcels (right panel) predicted triples and for 23 position players to make transition from NPB to MLB.

Figure 9:

Comparison of MLN (left panel) and Marcels (right panel) predicted home runs and for 23 position players to make transition from NPB to MLB.

Figure 10:

Comparison of MLN (left panel) and Marcels (right panel) predicted strikeouts and for 23 position players to make transition from NPB to MLB.

Table 4:

Pairwise correlations and PRESS statistics for MLN model predicted values and observed values, and between the Marcels and observed values, for 10 categories of interest and summary statistics batting average, on-base percentage, slugging percentage, and on-base plus slugging percentage.

CORR
	1B	2B	3B	HR	SH	SF	BB	HBP	SO	OIP	AVG	OBP	SLG	OPS
MLN	0.991	0.972	0.708	0.941	0.788	0.891	0.956	0.875	0.970	0.995	0.674	0.291	0.449	0.368
Marcels	0.968	0.959	0.714	0.893	0.675	0.885	0.882	0.625	0.886	0.965	0.379	0.160	0.278	0.270

PRESS
	1B	2B	3B	HR	SH	SF	BB	HBP	SO	OIP	AVG	OBP	SLG	OPS
MLN	85.670	23.420	3.956	3.965	2.983	1.088	56.351	3.246	96.235	209.540	0.005	0.012	0.023	0.053
Marcels	289.109	17.669	4.401	23.151	4.785	1.137	138.889	6.768	409.945	3521.136	0.008	0.014	0.027	0.063

Nearly across the board, the model outperforms the Marcels in terms of prediction. Most count outcomes, and all summary statistics were better predicted under the MLN model than by the Marcels. Considering the Marcels contains no prior player performance in its calculation, this is perhaps a low bar to clear, but it is an established bar. Indeed, the Marcels has generally been shown to be competitive with more complicated prediction models (Larson 2015). While first year predictions of past Japanese players under the other popular projection systems are not available, by outperforming the Marcels here it suggests that MLN may be competitive with, or outperform, these other established models.