Editor’s note: on fairness in sports analytics

For much of 2023, I served as the de facto Editor-of-Chief of the Journal of Quantitative Analysis in Sports (JQAS), following in the footsteps of my fellow Co-Editors Emily Casleton and Michael Lopez, and eventually handing the reins to the new Editor-In-Chief, Gil Fellingham. During my time, I handled what felt like an unusually large number of papers that proposed new systems of scoring (in various sports) that the authors claimed were more “fair.” This note is informed by those experiences and serves as a reflection on how we think about notions of “fairness” in sports analytics that I hope will be useful to future authors and the sports analytics community more broadly.

Three papers illustrate different paths forward when considering whether a sporting competition is “fair.” For now, let’s call them Papers A, B, and C. In each case, the authors proposed a new system of scoring or awarding prize money that they claimed was more “fair” than the existing systems, and in each case, I pressed the authors to carefully define their notion of fairness and list the criteria they use to evaluate it. Without specific criteria, I argued that any claims of superiority would be unsupported. In the case of Paper A, the authors were able to come up with a reasonable list of criteria in their revision, and showed that their scoring system satisfied the criteria in ways that were importantly better than the existing scoring system. This paper is in another round of revision and will be published in the next issue. With Paper B, the authors already had specific optimization criteria, and we eventually concluded that the designation of “fair” was not necessary, and had become a distraction from the contributions of the paper (for reasons outlined below). The authors removed the claim of “fairness” from the paper, and it too will be published in the next issue. The case of Paper C was more complicated, as the notion of “fairness” was integral to the research question, and the gender of the athletes was a central consideration. I insisted that if the authors wanted to claim that their proposed system was more “fair,” they needed to not only provide a definition of fairness, but also to inform their notions of fairness with perspectives from the extensive literature on women’s sports, and perhaps even from our colleagues in women and gender studies. To ignore those perspectives, I argued, would severely weaken the paper by abdicating our responsibility to include relevant scholarship. I knew this was asking a lot, and the authors respectfully decided to withdraw the paper from submission.

The process of reviewing these papers led me to think about what makes a sporting event “fair.” In considering this question, I am reminded of the ongoing discussion surrounding the distinction between equity and equality, aptly characterized by the cartoon in Figure 1 (which conveniently takes place at a sporting event).

Figure 1:

Equality versus Equity, courtesy of the Interaction Institute for Social Change | Artist: Angus Maguire.

In the left panel, the notion of equality centers around treating each participant the same, in this case giving each person a box of equal height regardless of each person’s height. In the right panel, the notion of equity requires that we give each person a box of the appropriate height such that they have the same opportunity (to see the game) as everyone else. The problem for sports analytics is that both systems might reasonably be described as “fair,” and we don’t agree on which notion of “fairness” we want.

I know this because the notion of fairness developed in Paper A was closer to equity, while the notions of fairness developed in Papers B and C were closer to equality.

To view this in a sporting context, consider handicapping in golf. Scoring in golf is relative to a predetermined number of strokes on each hole, which provides a baseline score called par. A handicap is an estimate of the average number of strokes that a player exceeds par over the course of 18 holes. In a professional golf tournament, handicaps are usually irrelevant, and the player with the lowest score wins. These events are “fair” because the players are treated equally, and it is a given that we as fans accept that the better players will win more often because they are better, regardless of how their superiority is derived (i.e., innate physical attributes, excellent training, mental acuity, etc.). However, in many amateur golf tournaments, players compete against their handicap. This means that on a par 72 course, a player with a 12-handicap who shoots an 80 will win against a player with a 4-handicap who shoots a 78. These events are considered “fair” because the players are treated equitably. We as fans accept that the players have more or less an equal chance of winning, and thus that the better players will not necessarily win even though they are better, because we are trying to control for their superiority. Dividing athletes into categories based on age, gender, or other attributes are related attempts to mitigate this same problem.

This distinction is not merely academic food-for-thought, nor is it as simple as amateur versus professional. In fact, the weighing of equity and equality has substantive real-world implications for professional sports. In tennis, Kovalchik and Ingram (2018) show that “longer matches favor the better player and make match outcomes more predictable.” The obvious implication is that men’s tennis (where matches are usually best-of-5 sets) is more predictable than women’s tennis (where matches are usually best-of-3 sets). Is it in the best interests of women’s tennis to extend their matches so that the better players win more often, which in a sense, would make them more equal? Or should men’s tennis shorten their matches to give the underdog a better chance of winning, and in some sense, make them more equitable? Paper B (Hall and Liu 2024) argues for something closer to equality than equity because it proposes a tournament system that raises the probability of the best two players meeting in the final (which strikes me as a thoroughly defensible outcome for players, fans, and tournament organizers alike).

At the same time, our colleagues in sports economics report that fan interest (and thus opportunities for teams and leagues to make money) is highest when the outcome of a match is more uncertain (see, for example, the discussion in Buraimo and Simmons (2009)). We considered both perspectives in Lopez et al. (2018), wherein we confirmed the finding that longer series favor the stronger team, and showed that the strongest team was least likely to win the championship in Major League Baseball (MLB), relative to the three other major North American sports. Since the publication of our paper, MLB has only expanded the playoffs with more shorter series. We can only assume that this is because the league will generate more revenue from this more equitable system, despite the frustration of players and team management. Paper A (Bartneck and Moltchanova 2024) argues for something closer to equity than equality, because it proposes a scoring system for para and master’s swimming that accounts for age and disability class.

While there are no one-size-fits-all solutions to these questions, I think that our discussions will be richer if we frame our notions of fairness using existing goalposts, as I’ve tried to do here.

As the flagship journal in sports analytics, JQAS has an obligation to publish only research of the highest quality. Apart from methodological contributions, one of the main criteria for publication in JQAS is the potential for real-world impact. Many papers are rejected from JQAS outright because they don’t address a question that people in sports actually care about. The best papers not only do this, but offer realistic solutions that are reasonably likely to be put into practice. Many have argued that athletes and sports commentators should “just focus on sports,” or more bluntly, “shut up and dribble.” To the contrary, in order for our scholarship to be of the highest quality it is necessary for us to engage with political, social, and historical context to the extent that it is relevant to the research we are conducting. To ignore that context by burying our heads in the proverbial sands (perhaps on a beach volleyball court) will only serve to limit the impact and excellence of our scholarship.