Causal inference with textual data: A quasi-experimental design assessing the association between author metadata and acceptance among ICLR submissions from 2017 to 2022

1.1 Peer-review process

Peer-review has been the cornerstone of scientific research. It is important that the peer-review process be fair and impartial, especially for early-career researchers. In recent years, the peer-review process has been under a lot of scrutiny. For instance, in 2014, the organizers of the Conference on Neural Information Processing Systems (NeurIPS) randomly duplicated 10 % of the submissions and assigned them to two independent sets of reviewers. The study found that 25.9 % of these submissions received inconsistent decisions. Cortes and Lawrence [1] tracked the fate of submissions rejected in the NeurIPS experiment and found that the peer-review process was good at identifying poor papers but fell short of pinpointing good ones. McGillivray and De Ranieri [2] analyzed 128,454 articles in Nature-branded journals and found that authors from less prestigious academic institutions are more likely to choose double-blind review as opposed to single-blind review. To further illuminate the difference between single-blind and double-blind review processes, Tomkins et al. [3] assigned each submission from 2017 Web Search and Data Mining conference to two single-blinded reviewers and two additional double-blinded reviewers, and they found single-blind reviewing conferred a significant advantage to papers with authors from high-prestige institutions. More recently, Sun et al. [4] studied 5,027 papers submitted to the International Conference on Learning Representations (ICLR) and found that scores given to the most prestigious authors significantly decreased after the conference switched its review model from single-blind review to double-blind review. Smirnova et al. [5] evaluated a policy that encouraged (but did not force) authors to anonymize their submissions and found that the policy increased positive peer-reviews by 2.4% and acceptance by 5.6% for low-prestige authors and slightly decreased positive peer-reviews and acceptance rate for high-prestige authors. Many of these studies identified associations between decision makers’ perception of certain aspects of articles’ author metadata (e.g., authors’ prestige or identity) and final acceptance decisions of these articles, and suggested various forms of implicit bias in the peer-review processes, especially among those that adopt a single-blind model.

1.2 Hypothetical experiment

In a seminal paper, Bertrand and Mullainathan [6] measured racial discrimination in labor markets by sending resumes with randomly assigned names, one African American sounding and the other White sounding (e.g., Lakisha versus Emily), to potential employers. Bertrand and Mullainathan’s [6] study was elegant for two reasons. First, it was a randomized experiment free of confounding bias, observed or unobserved, although to what extent the found effect could be attributed to the bias toward applicants’ race and ethnicity versus toward other personal traits signaled by the names is unclear (see, e.g., related discussions in Bertrand and Mullainathan [6, Section IV] and Greiner and Rubin [7]). Second, the study illustrated a general strategy to measure the causal effect due to an immutable trait: instead of imagining manipulating the immutable trait itself, the study manipulated employers’ perception of this immutable trait.

In a recent high-profile study published in the Proceedings of the National Academy of Sciences, Huber et al. [8] designed a field experiment in the similar spirit as Bertrand and Mullainathan [6]. Huber et al. [8] measured the extent of the status bias, defined as a differential treatment of the same paper by prominent versus less established authors in the peer-review process, by randomizing over 3,300 researchers to one of the three arms: one arm assigned an article with a prestigious author, one arm assigned an anonymized version of the same article, and the other arm assigned the same article but with a less established author. Huber et al. [8] found strong evidence that the prominence of authorship markedly increased the acceptance rate by as much as sixfold. Using a similar study design, Nielsen et al. [9] examined the extent to which country- and institution-related status bias in six disciplines, namely, astronomy, cardiology, materials science, political science, psychology, and public health, and found inconclusive evidence supporting a status bias.

Bertrand and Mullainathan [6] and Huber et al.’s [8] studies illuminate a randomized experiment that conference organizers and journal editorial offices could carry out, at least hypothetically, in order to understand various forms of bias. For instance, if the policy interest is to evaluate the effect of reviewers’ perception of certain aspects of authors (e.g., authors’ identity and institution), then a hypothetical experiment would forward to reviewers articles with randomly assigned aspects of interest. Although such an experiment is conceivable, it is difficult to implement due to practical constraints.

1.3 Quasi-experimental design using a corpus of ICLR papers

In the absence of a randomized controlled trial (RCT) a quasi-experimental design aims to fill in the gap by constructing two groups, one treatment group and the other comparison group, that are as similar as possible in pre-treatment variables from retrospective, observational data. Statistical matching is a popular quasi-experimental design device [10,11]. In this article, we describe an effort to conduct a matched observational study that investigates the effect of authorship metadata on articles’ final decisions using state-of-the-art natural language processing (NLP) tools and quasi-experimental design devices.

Our database was constructed from a carefully curated corpus of articles from ICLR, a premium international machine learning (ML) conference. The database is the first of its kind to provide an empirical evidence base for investigating the peer-review process. In particular, the database is feature-rich, in the sense that it contains not only explicit/structural features of an article such as its keywords, number of figures, and author affiliations, but also more subtle and higher-level features such as topic and textual complexity as reflected by articles’ text, and reviewers’ sentiment as reflected by their publicly available comments. Building upon Neyman and Rubin’s potential outcomes (PO) framework [12,13] and discussions of immutable traits regarding human subjects, for instance, those in Greiner and Rubin [7], we elaborate on the essential elements of causal inference that facilitate a cause-and-effect analysis between authorship and papers’ final decisions; in particular, we will carefully define and state the treatment of interest including its timing, pre-treatment variables, causal identification assumptions, causal null hypothesis to be tested, and how to interpret the results, all in the context of study units being textual data.

The conference submission and peer-review process consists of multiple steps. For a typical ML conference such as ICLR, articles need to be submitted by authors before a pre-specified deadline. Valid submissions are then forwarded to a number of reviewers (typically three to four) for feedback and a numerical rating. This part of the peer-review process is double-blind so the reviewers and authors in principle do not know each other, although in practice, reviewers could identify authors from penmanship or because the authors might have uploaded their articles to the preprint platform arXiv.org. Authors are then given the chance to answer reviewers’ comments and feedback and provide a written rebuttal. Reviewers are allowed to modify their previous ratings taking into account the rebuttal. Finally, an area chair (similar to an associate editor of an academic journal) reviews the article and its ratings and then makes a final decision. Submitted articles, author metadata, reviewers’ written comments, authors’ written rebuttals, reviewers’ ratings and area chairs’ final decisions are all openly available from the website openreview.net.

Our compiled database allows us to study many different aspects of the peer-review process. In this article, we will focus specifically on the last stage of the peer-review process and investigate the effect of authorship metadata on area chairs’ final decisions. Our focus was motivated by several considerations. First, it is an empirical fact that articles receiving identical or near-identical ratings could receive different final decisions (see, e.g., the stacked bar graph in Panel a of Figure 1). It is not unreasonable for authors, especially those who are junior and less established, to wonder if they are fairly treated. Second, any endeavor to establish a cause-and-effect relationship using observational data is challenging because of unmeasured confounding. In our analysis, articles have many implicit features such as novelty and thoughtfulness, and these unmeasured confounders could, in principle, explain away any association between author metadata and final decisions. This problem is greatly attenuated when we focus on area chairs’ final decisions and have reviewers’ ratings as pre-treatment covariates: reviewers’ ratings should be a good, albeit not perfect, proxy for the innate quality of an article.

Figure 1

Illustration of ratings and high-level textual features: (a) stacked bar graphs of acceptance decisions. The decisions of borderline articles (average rating between 5 and 7) are not clearcut, (b) T -SNE plot of the Specter embedding together with arXiv primary categories and primary keywords among submissions that have been put onto arXiv, (c) sentiment and ratings of reviews, and (d) distribution of complexity scores of a random samples of arXiv articles across four primary categories.

The plan for the rest of the article is as follows. In Section 2, we briefly describe our compiled database. In Section 3, we lay out the PO framework under which we will conduct the matched observational study. Section 4 describes two concrete study designs. We also report outcome analysis results in Section 4. Section 5 discusses how to interpret our findings.

2.1 Database construction and structural features

We used the ICLR database^[1] collected and complied by Zhang et al. [14]. Motivated by the observation that area chairs’ final decisions of submissions with an average rating between 5 and 7 are not deterministic, as shown in Panel a of Figure 1, we restricted ourselves to this subset of borderline submissions. We first briefly recall the data collection and cleaning process here for completeness.

The OpenReview API was used to crawl data of 10,289 submissions to the ICLR from 2017 to 2022. The crawled data include (i) submissions and their metadata, (ii) author metadata, (iii) review/rebuttal data and (iv) area-chair/program chair decision. Structural features, including the number of sections, figures, tables, and bibliographies, were extracted in a relatively straightforward manner. We also extracted and ranked self-reported keywords from each submission to form primary and secondary keywords. Author profiles include an optional self-reported gender; we used the first name dictionary developed by Tsai et al. [15] to provide a numerical score based on the first names of the authors, where 0 signifies female and 1 otherwise. Author profiles were then augmented via Google Scholar API to obtain author citation and h -index data. Author institution was matched using the domain name of the author email.^[2] Although CSRanking data are available, it does not have full coverage of all authors’ institutions. As such, we mainly used the institutional ranking derived from the cumulative number of accepted papers to the ICLR in the past. For example, the ranking in 2020 of institution A was determined by all papers accepted to ICLR 2017–2020 that had at least one author from it. The review data include rating, confidence, and textual reviews. In some years, for example, 2020 and 2022, there were additional assessments such as technical soundness or novelty. Since these additional assessments were not available for all years, we restricted our attention to ratings, confidence, and higher-level features derived from textual reviews to be discussed shortly. Finally, we dichotomized the paper decision by grouping various acceptance designations (spotlight, poster, short talk) into “Accept” and “Reject” or “invited to workshop track” as “Reject.”

We also identified if a submission was posted on the preprint platform arXiv.org  before the review deadline by (i) searching for five most relevant results based on the title and abstract corresponding to each article from arXiv.org, (ii) computing the Jaccard similarity and normalized Levenshtein similarity between authors, and (iii) calculating the cosine-similarity of the title-abstract embedding. Using the arXiv timestamp, we then identified which submissions were posted prior to the end of the review process. Among the subset of papers that had arXiv counterparts, we also obtained their arXiv-related metadata such as primary and secondary categories.

2.2 Derived higher-level features

Although the structural features described so far contain abundant information, we considered further leveraging language models to extract additional higher-level features directly from textual data. These higher-level features, such as topics, quality of writing and mathematical complexity, or rigor, may help quantify residual confounding not captured by structural aspects (e.g., those described in Section 2.1) of an article. Furthermore, in a matched observational study, it is desirable to have a characterization of the “similarity” among study units to serve as a criterion for matching. Therefore, we derived the following higher-level features and a similarity measure based on embeddings from language models to facilitate our study design.

3.1 Matched cohort study

In the absence of a well-controlled experiment (e.g., the hypothetical RCT envisioned in Section 1.2), observational studies, up to their important limitations, provide an alternative to explore a cause-and-effect relationship [21]. In an observational study, study units receiving different levels of treatment may differ systematically in their observed covariates, and this induces the so-called overt bias [10, Section 3]. In our case study, articles with different author metadata, for instance, those with authors from high-ranking versus relatively lower-ranking academic institutions, could differ systematically in topics, keywords, number of figures, and equations, among others, and this would invalidate a naïve comparison.

Statistical matching is a commonly used strategy to adjust for confounding in empirical studies [10,11,22–25]. The goal of statistical matching is to embed non-randomized, observational data into an approximate randomized controlled experiment by designing a matched control (or comparison) group that resembles the treated group in observed pre-treatment covariates by matching on these covariates [22,23], a balancing score derived from these covariates, e.g., Rosenbaum and Rubin’s [26] propensity score, or a combination of both [27].

We note that there are multiple ways to adjust for observed covariates and draw causal conclusions under the PO framework, statistical matching being one of them. Other commonly used methods include weighting, modeling the PO, and a combination of both. We found a matched observational study particularly suited for our case study for three reasons. First, it facilitates testing Fisher’s sharp null hypothesis of no effect, which is an appropriate causal null hypothesis encoding an intriguing notion of fairness, as we will discuss in detail in Section 3.4. Second, a matched design naturally takes into account similarity of textual data (for instance, as measured by cosine similarity based on their embeddings) and is capable of balancing some high-dimensional covariates like keywords in our data analysis. A third strength is mostly stylistic: a matched comparison best resembles Bertrand and Mullainathan’s [6] seminal field experiment and is perhaps the easiest-to-digest way to exhibit statistical analysis results to a non-technical audience.

In the rest of this section, we articulate essential elements in our analysis, including study units, treatment to be hypothetically manipulated, PO, timing of the treatment, pre-treatment variables, and causal null hypothesis of interest.

3.2 Study units; treatment and its timing; POs

As discussed in detail in Greiner and Rubin [7], there are two agents in our analysis of the effect of authorship metadata on area chairs’ final decisions: an ICLR article peer-reviewed and having received reviewers’ ratings, and a decision maker, i.e., an area chair or meta-reviewer (AC), who assigned the final acceptance status to the article. In our analysis, each study unit is a (peer-reviewed ICLR article, area chair) pair. There are a total of N = 10,289 study units in our compiled database, and 5,313 of them have three or four reviewers and an average rating between 5 and 7. We will write the i th study unit as S U i = ( article i , AC i ) .

We define the treatment of interest as an area chair’s perception of a peer-reviewed article’s authorship metadata. This definition is modeled after Bertrand and Mullainathan [6] and Greiner and Rubin [7] and implies the timing of the treatment: we imagine a hypothetical randomized experiment where peer-reviewed ICLR articles, whose constituent parts include text, tables, figures, reviewers’ ratings and comments, are randomly assigned authorship metadata and presented to the area chair for a final decision. This timing component of the treatment is critical because it implies which variables are “pre-treatment variables” under Neyman–Rubin’s causal inference framework, as we will discuss in detail in Section 3.3.

In principle, the most granular author metadata is a complete list of author names with their corresponding academic or research institutions. Let A → = a → denote author metadata and A the set of all possible configurations of author metadata. There is one PO Y i ( a → ) associated with unit i and each a → ∈ A ; in words, there is one final decision associated with each peer-reviewed article had the author metadata been a → . We will assume the consistency assumption so that the observed outcome Y i obs = Y i ( a → obs ) . One may adopt a variant of the stable unit treatment value assumption or SUTVA [28] to reduce the number of POs. For instance, one may further assume that the PO Y i ( a → ) depends on author metadata a → only via authors’ academic institutions. Let f ( ⋅ ) denote a mapping from author metadata to authors’ academic institutions, then this “stability” assumption amounts to assuming Y i ( a → ) = Y i ( a → ′ ) when f ( a → ) = f ( a → ′ ) . We do not a priori make such stability assumptions.

3.3 Observed and unobserved pre-treatment variables

According to Rubin [29], covariates refer to “variables that take their values before the treatment assignment or, more generally, simply cannot be affected by the treatment.” In the following, we briefly review a dichotomy of pre-treatment variables in the context of drawing causal conclusions from textual data [30].

In human populations research, pre-treatment variables or covariates are often divided into two broad categories: observed and unobserved (see, e.g., Rosenbaum [10,11]). A randomized controlled experiment like the one in Bertrand and Mullainathan [6] had the key advantage of balancing both observed and unobserved confounding, while drawing causal conclusions from observational data inevitably suffers from the concern of unmeasured confounding and researchers often control for a large number of observed covariates in order to alleviate this concern.

When study units are textual data, Zhang and Zhang [30] divided observed covariates into two types: explicit observed covariates X obs exp that could be derived from textual data at face value, e.g., number of equations, tables, and illustrations in the article, and implicit observed covariates X obs imp that capture higher-level aspects of textual data, e.g., the topic, flow, and novelty of the article. In our case study, we will consider the following explicit observed covariates: year of submission, reviewers’ ratings, number of authors, sections, figures and references, and keywords. We will also consider each article’s complexity, topic and reviewers’ sentiment extracted using the state-of-the-art, NLP models as described in Section 2.

Unmeasured confounding is a major concern for any attempt to draw a cause-and-effect conclusion from observational data, regardless of the covariance adjustment method. Despite researchers’ best intention and effort to control for all relevant pre-treatment variables via matching, there is always a concern about unmeasured confounding bias as we are working with observational data. In our analysis of ICLR papers, we identified two sources of unmeasured confounding. First, there could be residual confounding due to the insufficiency of language models (such as the SPECTER model) in summarizing or extracting implicit observed covariates X obs imp such as topics, flow, and sentiment. Second, in our analysis, we used numeric ratings from reviewers as a proxy of the quality and novelty of the article. Reviewers’ ratings may not be sufficient in summarizing the quality of the articles. Unmeasured confounding may lead to a spurious causal conclusion, and researchers routinely examine the robustness of the putative causal conclusion using a sensitivity analysis [10,11,31].

3.4 Causal null hypothesis: A case for Fisher

A causal statement is necessarily a comparison among POs. In the context of a many-level treatment assignment, Fisher’s sharp null hypothesis states the following:

Fisher’s sharp null hypothesis prescribes a notion of fairness that, arguably, best suits our vision: area chairs’ final decisions of the articles are irrelevant of author metadata; in other words, the decision Y i ( a → ) could potentially depend on any substantive aspect of the article i , including its topic, quality of writing, and reviewers’ ratings, but would remain the same had we changed author metadata from a → to a → ′ .

In addition to Fisher’s sharp null hypothesis, Neyman’s weak null hypothesis, which states that the sample average treatment effect is zero, is another commonly tested causal null hypothesis. Unlike Fisher’s sharp null, Neyman’s weak null hypothesis allows perception bias of varying magnitude for all article-AC pairs, as long as these biases would cancel out each other in one way or another. We found this a sub-optimal notion of fairness compared to that encoded by Fisher’s sharp null hypothesis, and we will focus on testing Fisher’s sharp null hypothesis in our data analysis.

Unlike Bertrand and Mullainathan’s [6] field experiment that randomly assigned resume names, our cohort of ICLR articles are not randomly assigned authorship metadata. It is conceivable that articles with more “prestigious” authors, however one might want to define the concept of “prestige,” could differ systematically in their reviewers’ ratings, topics, etc., and this difference in baseline covariates could potentially introduce a spurious association between author metadata and area chairs’ final decisions. To overcome this, we embed the observational data into a matched-pair design by constructing I matched pairs, each with two peer-reviewed articles, indexed by j = 1 , 2 , such that these two articles are as similar as possible in their covariates but with different author metadata. Let a → i j denote the author metadata associated with article j in the matched pair i . Such a matched-pair design enables us to test the following sharp null hypothesis:

We note that H 0 , sharp in (1) implies H 0 , sharp ′ in (2), so rejecting H 0 , sharp ′ would then provide evidence against H 0 , sharp . Such a design is termed “near-far” design in the literature [32,33] and has been used in a number of empirical studies [34–36].

4.1 A first matched comparison: design M1

We restricted our attention to 5,313 borderline articles that were peer-reviewed by three or four reviewers and received an average rating between 5 and 7. We first considered a study design M1 where each matched pair consisted of one article whose authors’ average institution ranking was among the top 30% of these 5,313 submissions and the other article whose authors’ average institution ranking was among the bottom 70 % . Columns 2 and 3 of Table 1 summarize the characteristics, including structural features and derived higher-level features, of 1,585 top-30% articles and those of the other 3,728 articles. As one closely examines these two columns, a number of features, including submission year, number of figures, complexity score as judged by the language model, keyword, and topic, differ systematically among these submissions. Matching helps remove most of the overt bias: in the matched comparison group (which is a subset of size n = 1,585 from the reservoir of 3,728 articles), standardized mean differences of all but one covariates are less than 0.1, or one-tenth of one pooled standard deviation. In fact, design M1 required near-exact matching on important covariates such as reviewers’ ratings and year of submission and achieved near-fine balance for categorical variables such as topic cluster and primary keyword [37]. Algorithms used to construct the matched design M1 will be described in detail in Section 4.3. Panel c in Figure 2 assesses how different two articles in each matched pair are in their authors’ average institution rankings (Figure A1 in Appendix for similar plots in the four-reviewer stratum). The average, within-matched-pair difference in authors’ average institution ranking is 74.3 among 983 matched pairs in the three-reviewer stratum (median: 49.6; interquartile range: 26.6–96.7) and 99.6 among 602 matched pairs in the four-reviewer stratum (median: 63.5; interquartile range: 28.5–135.0). On the other hand, the average, within-matched-pair difference in the highest ranked institution among co-authors is 43.8 among 983 matched pairs in the three-reviewer stratum (median: 20.5; interquartile range: 6.0–50.3) and 55.6 among 602 matched pairs in the four-reviewer stratum (median: 20.0; interquartile range: 6.0–56.5). We concluded that there was a sizable difference in author metadata between two articles in each matched pair.

Figure 2

Diagnostics of design M1 . Panel a: estimated propensity score top-30% articles, bottom-70% articles and matched comparison articles; Panel b: distribution of between-article cosine similarity before (median = 0.71) and after matching (median = 0.80); Panel c: boxplots of authors’ average (highest) institution rankings and matched-pair differences in authors’ average (highest) institution rankings; Panel d: permutation distribution and the observed test statistic of the classification permutation test (CPT).

To further demonstrate two groups are well comparable, Panel a of Figure 2 displays the distribution of the estimated “propensity score,” defined as the probability that authors’ average ranking of an article was among top 30%, in each of the following three groups: top-30% articles (red), bottom-70% articles (blue), and matched comparison articles (yellow), all in the three-reviewer stratum. Similar plots for articles with four reviewers can be found in Appendix. As is evident from the figure, the propensity score distribution of the matched comparison articles is more similar to that of the top-30% articles. Panel b of Figure 2 further plots the cosine similarity calculated from the raw textual data of each article. It is also evident that two articles in the same matched pair now have improved cosine similarity compared to that from two randomly drawn articles prior to matching. Our designed matched comparison M1 appears to be well balanced in many observed covariates and resembles a hypothetical RCT where we randomly assign author metadata.

The question remains as to whether the balance is sufficiently good compared to an authentic RCT and could justify randomization inference. We conducted a formal diagnostic test using Gagnon–Bartsch and Shem–Tov’s [38] CPT based on random forests to test the randomization assumption for the matched cohort. The randomization assumption cannot be rejected in either the three-reviewer or four-reviewer stratum ( p -value = 0.194 and 0.641, respectively) (see the null distribution and test statistic in Panel d of Figure 2 and Figure A1 in Appendix).

Panel a of Table 2 summarizes the outcomes of 1,585 matched pairs of two articles. We tested Fisher’s sharp null hypothesis H 0 , sharp ′ reviewed and discussed in Section 3.4 using a two-sided, exact McNemar’s test [39] and obtained a p -value of 0.050, which suggested some weak evidence that authorship metadata was associated with AC’s final decisions. We further examined the effect heterogeneity by repeating the analysis among matched pairs where both articles had an average score below 6 and matched pairs where both articles had an average score greater than or equal to 6. Interestingly, among the subgroup of below-6 borderline articles, we had strong evidence that author metadata was associated with area chairs’ final decisions ( p -value < 0.001 ). Under an additional stability assumption stating that the potential acceptance status Y ( a → ) depended on author metadata a → only via authors’ average institution ranking and remained unchanged when the average ranking is among the top-30% or among the bottom-70%, we estimated the odds ratio (OR) to be 0.57 (95% CI: [0.41, 0.78]), providing evidence that below-6 borderline articles from top-30% institutions were less favored by area chairs compared to their counterparts in the comparison group. On the other hand, among borderline articles with an average score greater than or equal to 6, we had no evidence that author metadata was associated with area chairs’ final decisions (OR = 1.04; 95% CI: [0.77, 1.41]; p -value = 0.824).

Finally, we conducted a sensitivity analysis where we excluded matched pairs with at least one article arxived before the end of the peer-review process. Approximately 15% of all submissions in our dataset were matched to their arxived versions. By excluding these articles, reviewers’ ratings and comments were more likely to be a faithful reflection of the true quality of the article. Because the proportion of arxived papers was relatively small, results remained similar (below-6 submissions: OR = 0.61; 95% CI: [0.41, 0.89]; p -value = 0.001; above-6 submissions: OR = 1.16; 95% CI: [0.82, 1.66]; p -value = 0.437).

Our analysis seems to defy the common wisdom that there is a “status bias” favoring higher-profile authors. It is then natural to ask, if author metadata were even more drastically different, shall we then see some evidence for status bias that would better align with previous findings? This inquiry led to a second, strengthened design M2 .

4.2 Strengthened design M2

Baiocchi et al. [32] first considered “strengthening” an instrumental variable (the differential distance) in an analysis of the effect of low-level versus high-level neonatal intensive care units (NICUs) on mortality rate. In their analysis, Baiocchi et al. [32] compared mothers who lived near to and far from a high-level NICU and “strengthened” the comparison by restricting their analysis to a smaller subset of comparable mothers who lived very near to and very far from a high-level NICU (see also Zhang et al. [40] and Chen et al. [41] for some related recent development). We adopted this idea here and constructed a strengthened design M2 where one article in each matched pair is a top-20% article (as opposed to a top-30% article in design M1 ) and the other matched comparison article was from the reservoir of bottom-80% articles. We also added a “dose-caliper” in the statistical matching algorithm to further separate the average ranking within each matched pair (see Section 4.3 for details).

A total of 1,051 matched pairs were formed. Panel c of Figure 3 summarizes the within-matched-pair difference in authors’ average institution ranking across 658 matched pairs in the three-reviewer stratum of the strengthened design M2 . In this strengthened design, the matched-pair-difference in institution ranking (averaged over all co-authors) now increases from 74.3 (as in the design M1) to 108.8, and the difference in the ranking of the highest ranked co-author increases from 43.8 to 64.4. Importantly, the cohort of top-20% articles and their matched comparison group are still comparable in all baseline covariates, as summarized in Table 3. Similar to the design M1, we cannot reject the randomization assumption based on the CPT ( p -value = 0.961).

Figure 3

Diagnostics of design M2 . Panel a: estimated propensity score top-30% articles, bottom-70% articles and matched comparison articles; Panel b: distribution of between-article cosine similarity before (median = 0.71) and after matching (median = 0.78). Panel c: boxplots of authors’ average (highest) institution rankings and matched-pair differences in authors’ average (highest) institution rankings. Panel d: permutation distribution and the observed test statistic of the CPT.

Panel b of Table 2 summarizes the outcomes of 1,051 matched pairs of two articles in the strengthened design M2 and associated subgroup analyses. We observed results similar to those derived from the design M1 , although the estimates were slightly less precise as a result of a smaller sample size (1,051 in M2 versus 1,585 in M1). Consistent results across two study designs help reinforce the conclusion that we did not find evidence supporting a “status bias” favoring authors from high-ranking institutions in this cohort of borderline articles. Quite contrary, below-6 borderline articles from top institutions appeared to be less favored by area chairs.

4.3 Matching algorithm: matching one sample according to multiple criteria

The matched sample M1 displayed in Table 1 was constructed using an efficient, network-flow-based optimization algorithm built upon a tripartite network [42] as opposed to a traditional, bipartite network [43]. Compared to a bipartite network, a tripartite network structure helps separate two tasks in the design of an observational study: (1) constructing closely matched pairs and (2) constructing a well-balanced matched sample. A detailed account of the algorithm can be found in the study of Zhang et al. [42]; in the following, we described how we designed the cost in the tripartite network and achieved the features of M1 and M2 described in Sections 4.1 and 4.2.

Figure 4 illustrates the basic tripartite network structure with three units, { γ 1 , γ 2 , γ 3 } , from the top-30% articles and five units, { τ 1 , … , τ 5 } , from the reservoir of bottom-70% articles. The goal of tripartite-graph-based statistical matching algorithm is to select a subset of units from { τ 1 , … , τ 5 } so that this selected subset resembles { γ 1 , γ 2 , γ 3 } in some aspects quantified by δ γ i , τ j and possibly some other aspects quantified by Δ γ ¯ i , τ ¯ j . This can be efficiently achieved by running a network-flow-based algorithm, where a feasible flow of magnitude 3 emanates from the source node ξ , subsequently travels through the layer { γ 1 , … , γ 3 } , layer { τ 1 , … , τ 5 } , layer { τ ¯ 1 , … , τ ¯ 5 } , layer { γ ¯ 1 , … , γ ¯ 3 } , and finally arrives at the sink ξ ¯ . The blue edges in Figure 4 illustrate one feasible flow, which picks τ 1 , τ 2 , τ 5 and pairs τ 1 to γ 1 , τ 2 to γ 2 , and τ 5 to γ 3 .

Figure 4

An illustrative plot of a tripartite network consisting of three treated units and five candidate control units. The blue edges correspond to one feasible flow that selects { τ 1 , τ 2 , τ 5 } and pairs τ 1 to γ 2 , τ 2 to γ 1 , and τ 5 to γ 3 .

Each feasible flow is associated with a total cost [42]. This total cost consists of two important pieces. The cost δ γ i , τ j associated with each edge e connecting γ i and τ j in the left part of the network encodes criteria for closely matched pairs. To construct the matched sample M1 , we let δ γ i , τ j be the cosine similarity derived from the SPECTER embeddings of article γ i and τ j . We then enforce near-exact matching on submission year and reviewers’ ratings by adding a large penalty to δ γ i , τ j if articles γ i and τ j were submitted to the ICLR conference in different calendar years or did not receive same ratings. As a result, two articles in each matched pair were exactly or near-exactly matched on reviewers’ ratings. For instance, a top-30% article γ i submitted at 2017 and peer-reviewed by three reviewers and received ratings of 6, 5 and 5 (from highest to lowest) was matched to a 2017-submitted, ( 6 , 5 , 5 ) -rated bottom-70% article τ j . This conscious design of δ γ i , τ j helped achieve improved within-matched-pair cosine similarity and near-exact match on year of submission and reviewers’ ratings in M1.

The cost Δ γ ¯ i , τ ¯ j associated with edge connecting γ ¯ i and τ ¯ j in the right part of the network encodes criteria for good overall balance when groups are viewed as a whole. To construct M1 , we estimated the propensity score based on article metadata and set Δ γ ¯ i , τ ¯ j to be the difference in the estimated propensity score to minimize earth-mover’s distance between the propensity score distributions of the top-30% articles and their matched comparison articles. The “balancing” property of the propensity score [26] then helped balance the covariates used to estimate it. One limitation of the propensity score is that its stochastic balancing property often does not suffice when balancing nominal variables with many categories; in these scenarios, a design technique known as fine balance is often used in conjunction with propensity score matching [11,37]. In short, fine balance is a technique that forces the frequency of one or more nominal variables to be identical or as close as possible in two groups. We finely balanced the topic clusters and keywords by adding a large penalty to Δ γ ¯ i , τ ¯ j when γ i and τ j differed in the topic cluster or keyword. Finally, matched pairs in M1 were obtained as a result of solving the minimum-cost flow optimization problem associated with this tripartite network. We conducted matching in the stratum of articles with three reviewers and four reviewers, respectively, because articles in the four-reviewer stratum have two additional covariates: a fourth reviewer rating and a fourth reviewer sentiment.

Our construction of the design M2 was analogous to that of M1, except that we further added a “dose-caliper,” defined as a large penalty when two articles γ i and τ j differ in their authors’ average institution rankings by less than a pre-specified caliper size, to the cost δ γ i , τ j . The average within-matched-pair difference in authors’ average rankings is 74.3 in M1; hence, we set the caliper size to be 80 when constructing M2. In this way, the within-matched-pair difference in authors’ average ranking was as large as 108.8 in the design M2, representing a meaningful improvement over that in M1.