Volume 12, Issue 1

Configurational Comparative Methods (CCMs) aim to learn causal structures from datasets by exploiting Boolean sufficiency and necessity relationships. One important challenge for these methods is that such Boolean relationships are often not satisfied in real-life datasets, as these datasets usually contain noise. Hence, CCMs infer models that only approximately fit the data, introducing a risk of inferring incorrect or incomplete models, especially when data are also fragmented (have limited empirical diversity). To minimize this risk, evaluation measures for sufficiency and necessity should be sensitive to all relevant evidence. This article points out that the standard evaluation measures in CCMs, consistency and coverage, neglect certain evidence for these Boolean relationships. Correspondingly, two new measures, contrapositive consistency and contrapositive coverage, which are equivalent to the binary classification measures specificity and negative predictive value, respectively, are introduced to the CCM context as additions to consistency and coverage. A simulation experiment demonstrates that the introduced contrapositive measures indeed help to identify correct CCM models.

This article proposes a class of M-estimators that double weight for the joint problems of nonrandom treatment assignment and missing outcomes. Identification of the main parameter of interest is achieved under unconfoundedness and missing at random assumptions with respect to the treatment and sample selection problems, respectively. Given the parametric framework, the asymptotic theory of the proposed estimator is outlined in two parts: first, when the parameter solves an unconditional problem, and second, when it solves a stronger conditional problem. The two parts help to summarize the misspecification scenarios permissible under the given framework and the role played by double weighting in each. As illustrative examples, the article also discusses the estimation of causal parameters like average and quantile treatment effects. With respect to the average treatment effect, this article shows that the proposed estimator is doubly robust . Finally, a detailed application to Calónico and Smith’s (The women of the national supported work demonstration. J Labor Econom. 2017;35(S1):S65–S97.) reconstructed sample from the National Supported Work training program is used to demonstrate the estimator’s performance in empirical settings.

Following Efron (2014), we propose an algorithm for estimating treatment effects for use by researchers employing a regression-discontinuity (RD) design. This algorithm generates a set of estimates of the treatment effect from bootstrapped samples, wherein the polynomial-selection algorithm developed by Pei, Lee, Card, and Weber (2021) is applied to each sample, the average of these RD treatment effect (RDTE) estimates is computed and serves as the overall estimate of the RDTE. Effectively, this procedure estimates a set of plausible RD estimates and weights the estimates by their likelihood of being the best estimate to form a weighted-average estimate. We discuss why this procedure may lower the estimate’s root mean squared error (RMSE). In simulation results, we show that this better performance is achieved, yielding up to a 5% reduction in RMSE relative to PLCW’s method and a 16% reduction in RMSE relative to Calonico, Cattaneo, and Titiunik’s (2014) method for bandwidth selection (with default settings).

Bias in causal comparisons has a correspondence with distributional imbalance of covariates between treatment groups. Weighting strategies such as inverse propensity score weighting attempt to mitigate bias by either modeling the treatment assignment mechanism or balancing specified covariate moments. This article introduces a new weighting method, called energy balancing, which instead aims to balance weighted covariate distributions. By directly targeting distributional imbalance, the proposed weighting strategy can be flexibly utilized in a wide variety of causal analyses without the need for careful model or moment specification. Our energy balancing weights (EBW) approach has several advantages over existing weighting techniques. First, it offers a model-free and robust approach for obtaining covariate balance that does not require tuning parameters, obviating the need for modeling decisions of secondary nature to the scientific question at hand. Second, since this approach is based on a genuine measure of distributional balance, it provides a means for assessing the balance induced by a given set of weights for a given dataset. We demonstrate the effectiveness of this EBW approach in a suite of simulation experiments, and in studies on the safety of right heart catheterization and on three additional studies using electronic health record data.

Discussions on causal relations in real life often consider variables for which the definition of causality is unclear since the notion of interventions on the respective variables is obscure. Asking “what qualifies an action for being an intervention on the variable X X ” raises the question whether the action impacted all other variables only through X or directly , which implicitly refers to a causal model. To avoid this known circularity, we instead suggest a notion of “phenomenological causality” whose basic concept is a set of elementary actions . Then the causal structure is defined such that elementary actions change only the causal mechanism at one node (e.g. one of the causal conditionals in the Markov factorization). This way, the principle of independent mechanisms becomes the defining property of causal structure in domains where causality is a more abstract phenomenon rather than being an objective fact relying on hard-wired causal links between tangible objects. In other words, causal relations between variables get defined by the interface between the system and an external agent (who is able to perform the elementary actions), rather than being an internal property of links between the variables. We describe this phenomenological approach to causality for toy and hypothetical real-world examples and argue that it is consistent with the causal Markov condition when the system under consideration interacts with other variables that control the elementary actions.

Conditional effect estimation has great scientific and policy importance because interventions may impact subjects differently depending on their characteristics. Most research has focused on estimating the conditional average treatment effect (CATE). However, identification of the CATE requires that all subjects have a non-zero probability of receiving treatment, or positivity, which may be unrealistic in practice. Instead, we propose conditional effects based on incremental propensity score interventions, which are stochastic interventions where the odds of treatment are multiplied by some factor. These effects do not require positivity for identification and can be better suited for modeling scenarios in which people cannot be forced into treatment. We develop a projection approach and a flexible nonparametric estimator that can each estimate all the conditional effects we propose and derive model-agnostic error guarantees showing that both estimators satisfy a form of double robustness. Further, we propose a summary of treatment effect heterogeneity and a test for any effect heterogeneity based on the variance of a conditional derivative effect and derive a nonparametric estimator that also satisfies a form of double robustness. Finally, we demonstrate our estimators by analyzing the effect of intensive care unit admission on mortality using a dataset from the (SPOT)light study.

Most classical methods popularly used in causal mediation analysis can only estimate the average causal effects and are difficult to apply to precision medicine. Although identifying heterogeneous causal effects has received some attention, the causal effects are explored using the assumptive parametric models with limited model flexibility and analytic power. Recently, machine learning is becoming a major tool for accurately estimating individualized causal effects, thanks to its flexibility in model forms and efficiency in capturing complex nonlinear relationships. In this article, we propose a novel method, conditional generative adversarial network (CGAN) for individualized causal mediation analysis (CGAN-ICMA), to infer individualized causal effects based on the CGAN framework. Simulation studies show that CGAN-ICMA outperforms five other state-of-the-art methods, including linear regression, k-nearest neighbor, support vector machine regression, decision tree, and random forest regression. The proposed model is then applied to a study on the Alzheimer’s disease neuroimaging initiative dataset. The application further demonstrates the utility of the proposed method in estimating the individualized causal effects of the apolipoprotein E- ε 4 allele on cognitive impairment directly or through mediators.

We review standard mediation assumptions as they apply to identifying antibody effects in a randomized vaccine trial and propose new study designs to allow the identification of an estimand that was previously unidentifiable. For these mediation analyses, we partition the total ratio effect (one minus the vaccine effect) from a randomized vaccine trial into indirect (effects through antibodies) and direct effects (other effects). Identifying λ \lambda , the proportion of the total effect due to an indirect effect, depends on a cross-world quantity, the potential outcome among vaccinated individuals with antibody levels as if given placebo, or vice versa . We review assumptions for identifying λ \lambda and show that there are two versions of λ \lambda , unless the effect of adding antibodies to the placebo arm is equal in magnitude to the effect of subtracting antibodies from the vaccine arm. We focus on the case when individuals in the placebo arm are unlikely to have the needed antibodies. In that case, if a standard assumption (given confounders the potential mediators and potential outcomes are independent) is true, only one version of λ \lambda is identifiable, and if not neither is identifiable. We propose alternatives for identifying the other version of λ \lambda , using experimental design to identify a formerly cross-world quantity. Two alternative experimental designs use a three-arm trial with the extra arm being passive immunization (administering monoclonal antibodies), with or without closeout vaccination. Another alternative is to combine information from a placebo-controlled vaccine trial with a placebo-controlled passive immunization trial.

In a seminal article, Ding and VanderWeele proposed a method of constructing bounds for causal effects that has become widely recognized in causal inference. This method requires the analyst to provide guesses of certain “sensitivity parameters,” loosely defined as the maximal strength of association that an unmeasured confounder may have with the exposure and with the outcome. Ding and VanderWeele stated that their bounds are sharp, but without defining this term. Using a common definition of sharpness, Sjölander A. A note on a sensitivity analysis for unmeasured confounding, and the related E-value. J Causal Inference. 2020;8(1):229–48 showed that Ding and VanderWeele’s bounds are sharp in some regions of the sensitivity parameters, but are non-sharp in other regions. In this note, we follow up the work by Sjölander A. A note on a sensitivity analysis for unmeasured confounding, and the related E-value. J Causal Inference. 2020;8(1):229–48, by deriving bounds that are guaranteed to be sharp in all regions of Ding and VanderWeele’s sensitivity parameters. We illustrate the discrepancy between Ding and VanderWeele’s bounds and the sharp bounds with a real data example on vitamin D insufficiency and urine incontinence in pregnant women.

We propose a model of treatment interference where the response of a unit depends only on its treatment status and the statuses of units within its K-neighborhood. Current methods for detecting interference include carefully designed randomized experiments and conditional randomization tests on a set of focal units. We give guidance on how to choose focal units under this model of interference. We then conduct a simulation study to evaluate the efficacy of existing methods for detecting network interference. We show that this choice of focal units leads to powerful tests of treatment interference that outperform current experimental methods.

Causal inference from observational data often rests on the unverifiable assumption of no unmeasured confounding. Recently, Tchetgen Tchetgen and colleagues have introduced proximal inference to leverage negative control outcomes and exposures as proxies to adjust for bias from unmeasured confounding. However, some of the key assumptions that proximal inference relies on are themselves empirically untestable. In addition, the impact of violations of proximal inference assumptions on the bias of effect estimates is not well understood. In this article, we derive bias formulas for proximal inference estimators under a linear structural equation model. These results are a first step toward sensitivity analysis and quantitative bias analysis of proximal inference estimators. While limited to a particular family of data generating processes, our results may offer some more general insight into the behavior of proximal inference estimators.

The evidence-based medicine approach to causal medical inference is the dominant account among medical methodologists. Competing approaches originating in the philosophy of medicine seek to challenge this account. In order to see how successful these challenges are, we need to assess the performance of all approaches in real world medical inference. One important real world problem all approaches could be applied to is the assessment of drugs for approval by drug regulation agencies. This study assesses the success of the status quo against an empirical non-systematically obtained body of evidence and we scrutinise the alternative approaches from the armchair, contemplating how they would fare in the real world. We tentatively conclude that the status quo is regularly not successful at its primary task as it regularly fails to correctly assess effectiveness and safety and suggest that this is due to inherent factors of the “messy real world.” However, while all alternatives hold promise, they are at least as susceptible to the real world issues that beset the status quo . We also make recommendations for changes to current drug approval procedures, identify lacunae to fill in the alternatives, and finally, call for a continuation of the development of alternative approaches to causal medical inference and recommendations for changes to current drug approval procedures.

Discovering causal relationships from observational data is a challenging task that relies on assumptions connecting statistical quantities to graphical or algebraic causal models. In this work, we focus on widely employed assumptions for causal discovery when objects of interest are (multivariate) groups of random variables rather than individual (univariate) random variables, as is the case in a variety of problems in scientific domains such as climate science or neuroscience. If the group level causal models are derived from partitioning a micro-level model into groups, we explore the relationship between micro- and group level causal discovery assumptions. We investigate the conditions under which assumptions like causal faithfulness hold or fail to hold. Our analysis encompasses graphical causal models that contain cycles and bidirected edges. We also discuss grouped time series causal graphs and variants thereof as special cases of our general theoretical framework. Thereby, we aim to provide researchers with a solid theoretical foundation for the development and application of causal discovery methods for variable groups.

It is often of interest to decompose a total effect into an indirect effect, relayed through a particular mediator, and a direct effect. However, these effect components are not identified if there are unmeasured confounding of the mediator and the outcome. We derive nonparametric bounds on the natural direct and indirect effects, and Cornfield inequalities that the unmeasured confounders must satisfy to explain away an “observed” effect. We demonstrate, analytically and by simulation, that these bounds and Cornfield inequalities are sharper than those previously proposed in the literature. We illustrate the methods with an application to cholestyramine treatment for coronary heart disease.

I thank Thomas Richardson and James Robins for their discussion of my article, and discuss the similarities and differences between their approach to causal modelling, based on single world intervention graphs, and my own decision-theoretic approach.

There is a growing number of studies benchmarking the performance of configurational comparative methods (CCMs) of causal data analysis. A core benchmark criterion used in these studies is a dichotomous (i.e., non-quantitative) correctness criterion, which measures whether all causal claims entailed by a model are true of the data-generating causal structure or not. To date, Arel-Bundock [ The double bind of Qualitative Comparative Analysis ] is the only one who has proposed a measure quantifying correctness. That measure, however, as this study argues, is problematic because it tends to overcount errors in models. Moreover, we show that all available correctness measures are unsuited to assess relations of indirect causation. We therefore introduce a new correctness measure that adequately quantifies errors and does justice to indirect causation. We also offer a new completeness measure quantifying the informativeness of CCM models. Together, these new measures broaden and sharpen the resources for CCM benchmarking.

There is a growing literature on design-based (DB) methods to estimate average treatment effects (ATEs) for randomized controlled trials (RCTs) for full sample analyses. This article extends these methods to estimate ATEs for discrete subgroups defined by pre-treatment variables, with an application to an RCT testing subgroup effects for a school voucher experiment in New York City. We consider ratio estimators for subgroup effects using regression methods, allowing for model covariates to improve precision, and prove a new finite population central limit theorem. We discuss extensions to blocked and clustered RCT designs, and to other common estimators with random treatment-control sample sizes or summed weights: post-stratification estimators, weighted estimators that adjust for data nonresponse, and estimators for Bernoulli trials. We also develop simple variance estimators that share features with robust estimators. Simulations show that the DB subgroup estimators yield confidence interval coverage near nominal levels, even for small subgroups.

We propose a nonlinear difference-in-differences (DiD) method to estimate multivariate counterfactual distributions in classical treatment and control study designs with observational data. Our approach sheds a new light on existing approaches like the changes-in-changes estimator and the classical semiparametric DiD estimator, and it also generalizes them to settings with multivariate heterogeneity in the outcomes. The main benefit of this extension is that it allows for arbitrary dependence between the coordinates of vector potential outcomes and includes higher-dimensional unobservables, something that existing methods cannot provide in general. We demonstrate its utility on both synthetic and real data. In particular, we revisit the classical Card & Krueger dataset, which reports fast food restaurant employment before and after a minimum wage increase. A reanalysis with our methodology suggests that these restaurants substitute full-time labor with part-time labor on aggregate in response to a minimum wage increase. This treatment effect requires estimation of the multivariate counterfactual distribution, an object beyond the scope of classical causal estimators previously applied to this data.

In causal inference, interference takes place when the intervention on one unit affects the outcome of other units. Most of the previous methods for estimating network causal effects assume that the covariate information is complete, which may lead to biased estimates when missingness exists. In this study, we consider the partial and direct interference setting. Specifically, the whole population can be divided into different clusters. Within each cluster, the outcome of each unit is dependent on the intervention received by other units, but not dependent on the confounders or outcomes of other units within the same cluster or of those in different clusters. We also assume that the confounders are subject to non-ignorable missingness, and a confounder is considered as missing if any component of it is missing. We propose three consistent estimators for the direct, indirect, total, and overall effect of the intervention on the outcome, and derive the asymptotic results accordingly. A comprehensive study is carried out as well to investigate the finite sample properties of the proposed estimators. We illustrate the proposed methods by analyzing the dataset collected from an acid rain program, which was launched to reduce air pollution in the United States by encouraging the scrubber’s installation on power plants, where the records of some operating characteristics of the power generating facilities are subject to missingness.

We consider an experiment with at least two stages or batches and O ( N ) O\left(N) subjects per batch. First, we propose a semiparametric treatment effect estimator that efficiently pools information across the batches, and we show that it asymptotically dominates alternatives that aggregate single batch estimates. Then, we consider the design problem of learning propensity scores for assigning treatment in the later batches of the experiment to maximize the asymptotic precision of this estimator. For two common causal estimands, we estimate this precision using observations from previous batches, and then solve a finite-dimensional concave maximization problem to adaptively learn flexible propensity scores that converge to suitably defined optima in each batch at rate O p ( N − 1 ⁄ 4 ) {O}_{p}\left({N}^{-1/4}) . By extending the framework of double machine learning, we show this rate suffices for our pooled estimator to attain the targeted precision after each batch, as long as nuisance function estimates converge at rate o p ( N − 1 ⁄ 4 ) {o}_{p}\left({N}^{-1/4}) . These relatively weak rate requirements enable the investigator to avoid the common practice of discretizing the covariate space for design and estimation in batch adaptive experiments while maintaining the advantages of pooling. Our numerical study shows that such discretization often leads to substantial asymptotic and finite sample precision losses outweighing any gains from design.

This article establishes the functional average as an important estimand for causal inference. The significance of the estimand lies in its robustness against traditional issues of confounding. We prove that this robustness holds even when the probability distribution of the outcome, conditional on treatment or some other vector of adjusting variables, differs almost arbitrarily from its counterfactual analogue. This article also examines possible estimators of the functional average, including the sample mid-range, and proposes a new type of bootstrap for robust statistical inference: the Hoeffding bootstrap. After this, the article explores a new class of variables, the U {\mathcal{U}} class, that simplifies the estimation of functional averages. This class of variables is also used to establish mean exchangeability in some cases and to provide the results of elementary statistical procedures, such as linear regression and the analysis of variance, with causal interpretations. Simulation evidence is provided. The methods of this article are also applied to a National Health and Nutrition Survey data set to investigate the causal effect of exercise on the blood pressure of adult smokers.

The causal dose–response curve is commonly selected as the statistical parameter of interest in studies where the goal is to understand the effect of a continuous exposure on an outcome. Most of the available methodology for statistical inference on the dose-response function in the continuous exposure setting requires strong parametric assumptions on the probability distribution. Such parametric assumptions are typically untenable in practice and lead to invalid inference. It is often preferable to instead use nonparametric methods for inference, which only make mild assumptions about the data-generating mechanism. We propose a nonparametric test of the null hypothesis that the dose-response function is equal to a constant function. We argue that when the null hypothesis holds, the dose-response function has zero variance. Thus, one can test the null hypothesis by assessing whether there is sufficient evidence to claim that the variance is positive. We construct a novel estimator for the variance of the dose-response function, for which we can fully characterize the null limiting distribution and thus perform well-calibrated tests of the null hypothesis. We also present an approach for constructing simultaneous confidence bands for the dose-response function by inverting our proposed hypothesis test. We assess the validity of our proposal in a simulation study. In a data example, we study, in a population of patients who have initiated treatment for HIV, how the distance required to travel to an HIV clinic affects retention in care.

Many software packages have been developed to assist researchers in drawing directed acyclic graphs (DAGs), each with unique functionality and usability. We examine five of the most common software to generate DAGs: Ti k Z, DAGitty, ggdag, dagR, and igraph. For each package, we provide a general description of its background, analysis and visualization capabilities, and user-friendliness. In addition, in order to compare packages, we produce two DAGs in each software, the first featuring a simple confounding structure and the second with a more complex structure with three confounders and a mediator. We provide recommendations for when to use each software depending on the user’s needs.

Optimizing the allocation of units into treatment groups can help researchers improve the precision of causal estimators and decrease costs when running factorial experiments. However, existing optimal allocation results typically assume a super-population model and that the outcome data come from a known family of distributions. Instead, we focus on randomization-based causal inference for the finite-population setting, which does not require model specifications for the data or sampling assumptions. We propose exact theoretical solutions for optimal allocation in 2 K {2}^{K} factorial experiments under complete randomization with A-, D-, and E-optimality criteria. We then extend this work to factorial designs with block randomization. We also derive results for optimal allocations when using cost-based constraints. To connect our theory to practice, we provide convenient integer-constrained programming solutions using a greedy optimization approach to find integer optimal allocation solutions for both complete and block randomizations. The proposed methods are demonstrated using two real-life factorial experiments conducted by social scientists.

Given a binary treatment and a binary mediator, mediation analysis decomposes the total effect of the treatment on an outcome variable into various sub-effects, and there appeared two-, three-, and four-way decompositions in the literature. Using “principal stratification” based on the potential mediator types, we consider sub-treatment effects for “mediative never-takers, compliers, defiers, and always takers.” In this approach, although it is difficult to pick any one decomposition over the others in general, a particular three-way decomposition becomes well suited, which is thus advocated to use. We present identification conditions for the effects using conditional means, which is then followed by simple estimators that are applicable to any outcome variable (binary, count, continuous, etc.). We also provide simulation and empirical studies.

Out of the participants in a randomized experiment with anticipated heterogeneous treatment effects, is it possible to identify which subjects have a positive treatment effect? While subgroup analysis has received attention, claims about individual participants are much more challenging. We frame the problem in terms of multiple hypothesis testing: each individual has a null hypothesis (stating that the potential outcomes are equal, for example), and we aim to identify those for whom the null is false (the treatment potential outcome stochastically dominates the control one, for example). We develop a novel algorithm that identifies such a subset, with nonasymptotic control of the false discovery rate (FDR). Our algorithm allows for interaction – a human data scientist (or a computer program) may adaptively guide the algorithm in a data-dependent manner to gain power. We show how to extend the methods to observational settings and achieve a type of doubly robust FDR control. We also propose several extensions: (a) relaxing the null to nonpositive effects, (b) moving from unpaired to paired samples, and (c) subgroup identification. We demonstrate via numerical experiments and theoretical analysis that the proposed method has valid FDR control in finite samples and reasonably high identification power.

A century ago, Neyman showed how to evaluate the efficacy of treatment using a randomized experiment under a minimal set of assumptions. This classical repeated sampling framework serves as a basis of routine experimental analyses conducted by today’s scientists across disciplines. In this article, we demonstrate that Neyman’s methodology can also be used to experimentally evaluate the efficacy of individualized treatment rules (ITRs), which are derived by modern causal machine learning (ML) algorithms. In particular, we show how to account for additional uncertainty resulting from a training process based on cross-fitting. The primary advantage of Neyman’s approach is that it can be applied to any ITR regardless of the properties of ML algorithms that are used to derive the ITR. We also show, somewhat surprisingly, that for certain metrics, it is more efficient to conduct this ex-post experimental evaluation of an ITR than to conduct an ex-ante experimental evaluation that randomly assigns some units to the ITR. Our analysis demonstrates that Neyman’s repeated sampling framework is as relevant for causal inference today as it has been since its inception.

Neyman’s 1923 paper introduced the potential outcomes framework and the foundations of randomization-based inference. We discuss the influence of Neyman’s paper on four introductory to intermediate-level textbooks by Berkeley faculty members (Scheffé; Hodges and Lehmann; Freedman, Pisani, and Purves; and Dunning). These examples illustrate that Neyman’s key insights can be explained in intuitive and interesting ways to audiences at all levels, including undergraduates in introductory statistics courses. We have found Freedman, Pisani, and Purves’s box-of-tickets model to be a valuable expository tool, and we also find their intuitive explanation of Neyman’s variance result helpful: It is a “minor miracle” that in randomized experiments, the two-sample z z -test is conservative because of “two mistakes that cancel.” All four books take a more positive view of Neyman’s results than Neyman himself did. We encourage educators and researchers to explore ways to communicate Neyman’s ideas that are helpful for their own audiences.

In this article, we discuss both prospective and retrospective causal inferences, building on Neyman’s potential outcome framework. For prospective causal inference, we review criteria for confounders and surrogates to avoid the Yule–Simpson paradox and the surrogate paradox, respectively. For retrospective causal inference, we introduce the concepts of posterior causal effects given observed evidence to quantify the causes of effects. The posterior causal effects provide a unified framework for deducing both effects of causes in prospective causal inference and causes of effects in retrospective causal inference. We compare the medical diagnostic approaches based on Bayesian posterior probabilities and posterior causal effects for classification and attribution.

Many recent studies have probed status bias in the peer-review process of academic journals and conferences. In this article, we investigated the association between author metadata and area chairs’ final decisions (Accept/Reject) using our compiled database of 5,313 borderline submissions to the International Conference on Learning Representations from 2017 to 2022 under a matched observational study framework. We carefully defined elements in a cause-and-effect analysis, including the treatment and its timing, pre-treatment variables, potential outcomes (POs) and causal null hypothesis of interest, all in the context of study units being textual data and under Neyman and Rubin’s PO framework. We found some weak evidence that author metadata was associated with articles’ final decisions. We also found that, under an additional stability assumption, borderline articles from high-ranking institutions (top-30% or top-20%) were less favored by area chairs compared to their matched counterparts. The results were consistent in two different matched designs (odds ratio = 0.82 [95% confidence interval (CI): 0.67 to 1.00] in a first design and 0.83 [95% CI: 0.64 to 1.07] in a strengthened design) and most pronounced in the subgroup of articles with low ratings. We discussed how to interpret these results in the context of multiple interactions between a study unit and different agents (reviewers and area chairs) in the peer-review system.

Neyman’s seminal work in 1923 has been a milestone in statistics over the century, which has motivated many fundamental statistical concepts and methodology. In this review, we delve into Neyman’s groundbreaking contribution and offer technical insights into the design and analysis of randomized experiments. We shall review the basic setup of completely randomized experiments and the classical approaches for inferring the average treatment effects. We shall, in particular, review more efficient design and analysis of randomized experiments by utilizing pretreatment covariates, which move beyond Neyman’s original work without involving any covariate. We then summarize several technical ingredients regarding randomizations and permutations that have been developed over the century, such as permutational central limit theorems and Berry–Esseen bounds, and we elaborate on how these technical results facilitate the understanding of randomized experiments. The discussion is also extended to other randomized experiments including rerandomization, stratified randomized experiments, matched pair experiments, and cluster randomized experiments.