The necessity of construct and external validity for deductive causal inference

1 Introduction

The past few decades have witnessed the rise of the “Credibility Revolution” in quantitative causal inference [1–7]. Researchers influenced by the Credibility Revolution advocate for a design-based approach to evaluating causal effects. Arguably, this approach has come to dominate quantitative research in the social sciences and in policy evaluation [8,9].^[1] The design-based approach is at the core of popular textbooks on quantitative research design for PhD and MPP training [10–12]. To recognize the impact of this approach on quantitative social science, two of the key innovators of the Credibility Revolution, Joshua Angrist and Guido Imbens, were awarded the Nobel Prize in 2021. In its announcement, the Nobel Committee stated the Credibility Revolution “improved researchers’ ability to answer causal questions of great importance for economic and social policy [...]”.

The Credibility Revolution has contributed to the study of causality in at least two important ways [9]. First, the field promotes a deductive approach to discovering causal effects by requiring that assumptions be made explicit. In this tradition, a causal claim is said to be “credible” when the connection between a statistical result and a causal effect of interest deductively follows from transparently stated assumptions [9,14,15]. In a deductive view, claims regarding causality are not justified by data and analysis alone.^[2] Instead, such claims require an explicit statement of the assumptions that deductively guarantee a causal conclusion.

Second, tightly related to the first contribution, the Credibility Revolution advances designs that warrant assumptions sufficient to “identify” causal effects from observed data [2–5]. In particular, the field advances designs that warrant an assumption of internal validity. While we will highlight the randomized controlled trial (RCT) design, this tradition also advances natural experimental designs such as instrumental variables, matching, difference-in-differences, and regression discontinuity that each posit different assumptions [16]. Each of these designs seeks to ensure that the contrast of interest to the research question is not confounded and hence is internally valid. Given the design assumptions, the quantitative analyst can compare aggregate outcomes between counterfactual states of the world to detect the presence of a cause.

While causal identification strategies primarily focus on internal validity, they routinely omit considerations of construct and external validity [17, p. 366].^[3] This article reviews the foundations of identification for quantitative causal inference and demonstrates this neglect in the literature results in a substantial inferential gap. Identifying a causal effect among measured variables in an internally valid research design is insufficient on its own to ensure a deductive claim about what actually causes what. Without construct validity, one risks mislabeling the cause or outcome [18]. Without external validity, one risks misunderstanding the conditions enabling the cause to have an effect [19]. Thus, the causal inference literature must incorporate assumptions regarding construct and external validity within identification strategies to enable deductive, generalized claims about causes and effects.

As even Angrist and Pischke [1, p. 23] acknowledge, the most that internal validity allows the analyst to deduce is the simple fact that some cause occurred in the course of an experiment. But internal validity sheds no light on what were the cause and effect, or why the effect occurred. As a result, internal validity provides no guidance on how the presence of causality shown in a given study or across studies can accumulate into deductively formed knowledge. Instead, in their view, any substantive claims about what was the cause, what was the effect, and what were the enabling conditions can at best be – in their words – “speculative” (see also [20, p. 959]). Without explicit assumptions about construct and external validity, however, such speculations remain tentative in a way that undermines the deductive “credibility” of a causal claim, contradicting the very purpose of the Credibility Revolution.

In practice, applied researchers – even leading proponents of the Credibility Revolution – routinely make general causal claims without explicitly declaring what is speculation, and thus fail to make their assumptions transparent or their causal claims deductively “credible.” For example, Gerber et al. [21, p. 33] claim that mailing postcards that reveal one’s voting history to neighbors “demonstrate[s] the profound importance of social pressure as an inducement to political participation.” Likewise, Angrist et al. [22, p. 858] claim that a specific charter school design “generated substantial score gains,” attributing causality to the school design itself. Based on RCTs of an anti-poverty program, Banerjee et al. [23] claim: “It is possible to make sustainable improvements in the economic status of the poor with a relatively short-term intervention.” These authors make claims about actual causes and actual effects without sufficiently clarifying that they had intended these internally valid but not necessarily construct or externally valid claims to rely on – and knowledge in their field to accumulate from – “speculation.”

Using insights from the philosophy of science, this article provides a critique that aims to close this gap in the literature on causal inference and the Credibility Revolution. We explain how, whether the analyst recognizes it or not, any deductively “credible” generalized causal claim requires assumptions of construct and external validity – assumptions that ultimately derive from theory [26–29] and substantive knowledge [30,31]. Thus, deduction in quantitative causal inference fundamentally depends on a combination of theory and descriptively, historically, or qualitatively derived substantive knowledge along with any design-based statistical evidence.

To be clear: the argument for this article is not simply repeating the widespread view that “construct and external validity are important.” We fully agree with Shadish et al. [18] that each type of validity is necessary for causal generalization. But our argument is more specific and goes beyond that understanding. We demonstrate that like internal validity, construct and external validity also are necessary to preserve the deductiveness of a causal claim.^[4] Nor do we take a position on whether researchers do or do not ignore construct or external validity in practice. Even if the discussion of these types of validities were pervasive, the current understanding of their necessity for deduction is absent or at best implicit.^[5] We show that neglecting the assumptions for construct and external validity undermines deduction – i.e., the very task scholars in the Credibility Revolution set out to accomplish.

To demonstrate the consequences of neglecting construct and external validity, we develop a framework we call causal specification.^[6] This framework formalizes assumptions regarding internal, construct, and external validity within a single causal expression and shows that causal deduction requires a rebalancing that equally values all three validities. Within a deductive understanding of causality, internal validity has no special status or lexical priority. The causal specification framework explicitly recognizes the contributions of theory and substantive knowledge to quantitative causal inference, and charts a way forward for social scientists who aim to make deductive causal generalizations.

2 Credibility revolution

Many of the advances in the Credibility Revolution have been governed by one of two comprehensive frameworks for causal inference, the potential outcomes framework, also known as the “Rubin causal model” (RCM) [20], and the structural causal model (SCM) framework developed by Pearl [3]. In either framework, a causal effect is defined by comparing the counterfactual outcomes – i.e., what would have happened if the cause had been present versus absent, while everything else had remained the same [36–38]. Because it is not possible to observe events that do not actually occur, at least half of the relevant cases remain unobserved post-intervention. A causal effect is said to be identified only if the effect described in counterfactual terms can be deduced from stated assumptions and the data, in the ideal circumstances where the analyst had an infinite amount of data [9,39,40]. Because it involves inference from observed data to unobserved counterfactuals, identification requires a set of assumptions about the relation between the observed data and the causal effect of interest.

We illustrate identification within the RCT design using a fictional vignette in which The Gold Standard Lab (GSL) undertakes a quantitative evaluation of the causal effect of an intervention aiming to increase juror turnout.^[7] Jury service is a form of democratic participation and is a political right that governments can coerce [41]. Of course, not everyone who receives a summons actually shows up at the courthouse for jury service, so courts routinely seek low-cost methods to increase the yield for jury summonses [42]. Inspired by get out the vote (GOTV) studies (e.g., refs [43,44]), GSL exposes residents in the court’s jurisdiction to different messages using jury summons reminder postcards (replicating [45]) to evaluate the messages’ causal effect.

Fortunately, the researchers in GSL are well trained in design-based causal inference and so they conduct a gold-plated RCT. GSL asks the Riverside (California) County Superior Court to mail official government postcards to residents who recently received a jury summons, randomizing so that half receive a standard reminder postcard and the other half receive a postcard, indicating that failure to appear could result in fines or imprisonment. The “enforcement” condition results in a statistically significant 10% increase in turnout relative to the “reminder” condition – an effect size more than 20 times that typically found in GOTV postcard experiments.

Given these strong results, GSL recommends courts adopt the enforcement message as a policy, and they publish an article containing the causal generalization: “Enforcement messages cause juror turnout.” Eager to demonstrate the efficacy of the enforcement message in other contexts, GSL next collaborates with the superior court in Orange County, California – a more affluent adjacent county – to implement the identical gold-plated evaluation. Much to their surprise, the enforcement postcard shows no treatment effect, from which they reason that affluence suppresses the enforcement message effect.

The GSL design very much adheres to the identification strategies of the Credibility Revolution. We can formalize the GSL study design and analysis as follows. There are n units indexed by i ∈ { 1 , … , n } . Throughout, we assume that each unit i is a randomly selected unit of the sampling frame S (on the role of sampling in RCTs [46]). We assume a binary representation of the causal variable, where A = 1 indicates the treatment state (having received a postcard with the enforcement message) and A = 0 indicates the control state (having received a postcard with the reminder message). B is a binary outcome variable, where B i = 1 if the unit reported for jury duty and B i = 0 if they did not. B i ( A = 1 ) represents the counterfactual outcome B conditioned on unit i ’s being in the treatment state, regardless of which postcard the unit actually received. Likewise, B i ( A = 0 ) represents the counterfactual outcome conditionally upon that unit being in the control state. For every unit of analysis i , these terms represent the potential outcome B that would have occurred had A = 1 occurred, or respectively, not occurred.

In an RCT, internal validity is present if the experimental units’ assignment to treatment or control is unrelated to their outcomes in either state, represented formally as

The symbol ⊥ ⊥ means “is independent of.” A weaker version of (1) only requires the claim to be true within strata of covariates. The assumption in (1) implies that the units in treatment and control have identical distributions of potential outcomes in expectation, and hence, each group can supply the missing counterfactual for the other. It follows that the “average treatment effect” estimand (ATE) is identified and can be estimated using the observed data:

where E S is the expectation over S and estimation is over the observed data.

The right-hand side of (2) is the estimand of the A T E since, under assumption (1), the expected difference is not driven by bias that would otherwise occur from confounding [10, p. 38]. Under assumption (1), the only systematic difference between those in the A = 1 condition and those in the A = 0 condition is their exposure to A . Researchers can interpret the estimated statistical relationship between A and B as a counterfactual causal claim of identifying the A T E if assumption (1), the core assumption of internal validity is true.^[8]

We focus on RCTs where the assumption of internal validity is well justified by the randomization of unit assignments, i.e., randomization renders the assumption of internal validity relatively weak. The claim to have identified a causal effect, however, in no way depends on the strength of the assumptions associated with any specific research design. The conclusion that a design identified a causal effect deductively follows from the premises encoded in the assumptions laid out in the formal apparatus of identification, such as selection on observables for matching and regression, continuity for regression discontinuity, or the parallel path assumption for difference-in-differences [3,16]. Once the assumptions are made, the conclusion follows deductively.

The Credibility Revolution was motivated by previous generations’ naïve reliance on regression models of observational data to test for causality. In that context, applied researchers invoked verbal assurances that they included all needed control variables. These assurances typically strained credulity [8]. In turn, many researchers developed what Stokes [47] refers to as “radical skepticism” about unobserved confounders. We develop our arguments with the RCT design so that we can focus on the problems for causal inference even when one has near-perfect internal validity. Radical skepticism was an excess. However, the skeptical thinking that helped motivate the focus on internal validity equally justifies concerns about construct validity and external validity, as will become evident in our framework and discussion below.

3 Validity and causal deduction

The literature informed by the credibility revolution takes the variables that are actually measured as fundamental for understanding causality. As Holland [20] notes, the variables that actually are measured in a scientific procedure – such as A and B – are “primitive” in the RCM, and hence, counterfactuals and identification are each with respect to the measured variables. Researchers can identify that some cause occurred in a given experiment when the expected value of the variable B differs between counterfactual states characterized by A . Equally importantly, identification strategies often leave the scope of generalization vague or implicitly local to the experimental setting.

Nevertheless, researchers rarely interpret their findings in terms of only the measured variables and are often not explicit about their limited scope of generalization. Instead, scholars typically interpret their findings in terms of general causes and general effects that generalize beyond their study’s setting. For instance, GSL might claim “enforcement messages” cause “juror turnout.” As Shadish et al. [18] forcefully argue, researchers’ semantic statements are virtually always in terms of underlying causes and effects and virtually never in terms of the measured variables themselves [48]. Furthermore, all experiments are embedded in a set of conditions that also matter to causality [19]. To be consistent with the intent of the Credibility Revolution, causal claims must be deductively true at the semantic level and hence must be supported by a complete set of assumptions.

Causes, effects, and the conditions in settings are referents in nature, and claims regarding their relationships are ontological. As a result, a deductively correct generalized causal claim requires not only the identification of a causal effect between measured variables (i.e., internal validity) but also the correct specification of the relationship between the measured variables and the actual cause and the actual effect (i.e., construct validity) and correct specification of the scope of the generalization (i.e., external validity).

Accurately specifying which aspects of the measured intervention had which relevant effects is essential to deducing a generalized causal claim [50–54]. Consider a typical causal process about which researchers wish to make a causal inference and claim. In any experiment, the intervention will contain a (potentially null) set of causes, which we label “active ingredients,” along with other elements that are not causal, or “inert ingredients” [55]. Likewise, the outcome will contain elements that are of interest (“the disease”) and not of direct interest (“symptoms”). We refer to the intervention’s active ingredients and the outcome of interest as the causal “relata.” GSL described the manipulated active ingredient as the “enforcement message,” and “juror turnout” as the outcome of interest. The active ingredients in the intervention will be supported or countered by conditions in the setting [56,57]. For GSL, ingredients such as affluence are perhaps present in coastal Orange County but not in inland Riverside County.

Using a classic notation from philosophy, we formalize the (metaphysical) causal relata and conditions using the following simple causal claim:

where α and β are types or classes of ontological events, which can be represented by random variables, and γ is a set of supporting or countering conditions. We assume that causation is a relation between individual concrete events (see Schaffer et al. [58], for a review of metaphysical alternatives), indicated with subscript i . For GSL, α is the class of events in which summoned jurors receive a postcard with such-and-such a content, and α i is an individual event of a particular summoned juror receiving a particular postcard. Following convention in philosophy, we use quotation marks to designate a semantic claim (i.e., a verbal claim about facts in nature, as opposed to actual facts in nature). A particular causal claim is a historical, hypothetical, or predictive statement about the relationship between two individual events: “ α i caused (or would have caused or will cause) β i .” A general causal claim or causal generalization asserts a pattern among the relata, that “events of type α cause events of type β .”

Often, as we have just done, the conditions or settings in which the generalization holds are left implicit or unstated. Although some causal generalizations in physics might be truly universal ( α causes β whenever and wherever α occurs), in social science, causal generalizations are nearly always realistically restricted to settings with relevant local conditions, γ , that are often vaguely stated or understood [19,31].^[9] For example, GSL might only expect postcards to work in functioning democracies and among literate participants, even if they did not explicitly say so.

The statement “ α causes β in γ ” is a generalization: it is a claim that one thing generally causes another in certain conditions [59]. We say that a generalized causal claim is valid if the claim is true, i.e., if the purported cause and purported effect are the actual cause and actual effect. Our definition of validity comes from measurement theory originating in Kelly [60] and is consistent with Borsboom et al. [61].^[10] In our framework, the general causal claim that “ α causes β in γ ” is valid if it is indeed the case that α causes β in γ . Hence, validity is a relationship between a claim and the world – the relationship that holds if and only if the claim correctly reflects reality (cf. ref [18, p. 35]).

There are exactly four ways in which the causal generalization “ α causes β in γ ” can be invalid, corresponding to the four parts of the claim:

1. α

2. causes

3. β

4. in γ .

To illustrate, consider how GSL’s initial causal generalization, “Enforcement messages cause juror turnout,” might fail:

1. Their claim might fail because the researchers misconstrued the nature of the cause, assigning an incorrect semantic label. The postcards’ effect might not be due to the specific words but rather because the text was longer in the enforcement compared to reminder message.

2. Their claim might fail internally, due to chance or poor experimental design. Maybe jurors who were already planning to appear at court disproportionately received the threatening postcards.

3. Their claim might fail because the researchers misconstrued the nature of the outcome, assigning an incorrect semantic label. Maybe juror turnout was mismeasured – for example, if excused absences were classified as successful recruitments.

4. Their claim might fail because the researchers mischaracterized how broadly their claim generalizes. The claim invites the reader to generalize across the US (though unfortunately this remains vague); but perhaps it only works in certain communities.

Generalizations always go beyond the scientific evidence. Researchers will have witnessed only a finite number of events in specific times and places. To make general causal claims that are meaningful to others, researchers must make a causal inference, moving from the evidence to a causal claim on the basis of theory, design-based procedures, substantive knowledge, and even common sense – i.e., assumptions combined with the evidence [15,17,25]. A causal generalization results from an inferential leap to the conclusion that in general, under conditions γ , α -type events cause (or often enough cause, or cause absent interference) β -type events. Such an inference may or may not be warranted, but without an inference, even if a study induced causality it could not support a general causal claim.^[11]

Now consider how one deduces generalized causal claims using evidence from an experiment. Our framework distinguishes active from inert ingredients as events that are bundled within the measures of the intervention and outcome, i.e., we do not take measured variables as the “primitives” of the analysis. To clarify this distinction, we label the ontological elements of the bundles (the causes, effects, and conditions of interest, plus inert ingredients) with lowercase Greek letters, and we label the measured bundles with uppercase Latin letters. In the fully binary case,

Note that ≜ means “is definitionally equal to,” ∧ logically means “and,” (requiring both elements to be true for the expression to be true), and ∨ logically means “or” (requiring one or both elements to be true for the expression to be true). In the binary case, each variable can be either true or false.

The Latin letters A , B , and C are observed measures of the intervention, outcome, and conditions in settings, respectively, here assumed to be measured without error [65]. The Greek letters represent hypothesized causes, effects, causal conditions, and inert elements, which combine into informationally equivalent sets [66]. The elements { α , γ } are “active ingredients” that have a causal effect on the outcome of interest β . The elements { θ α , θ γ } are “inert ingredients” and θ β is a related outcome that is not of interest. For simplicity, we omit interactions between elements of the sets.^[12]

This notation makes clear that measured variables are inherently bundles of the ontological components of interest [53]. In particular, since active and inert ingredients { α , θ α } are bundled in the intervention, removing the inert ingredient θ α in equation (4a) would make A false. For example, in the first GSL trial, the postcards bundled the enforcement message with an emotional tone, numerals related to the relevant statute, amount of ink, and sentence length and complexity (all varying at least slightly between treatment and control). Likewise, C bundles all of the conditions in the setting { γ , θ γ } that remain constant. These include design elements that are identical for treatment and control (e.g., the cardstock and court seal), attributes of the units that are assumed to be balanced through randomization (e.g., employment status of the recipient), and features of the context (e.g., Riverside during the rainy season). Typically, neither these conditions nor C itself is literally “measured” beyond disclosures of experimental procedures, units, and the setting of the RCT.

The disjunction in (4b) represents that the measurement of an outcome might reflect the real outcome of interest, β , or instead a related event not of direct interest, θ β (e.g., a legally valid request for excuse). In most studies, β itself cannot be measured in isolation but must be inferred from self-reports or records ( B ) assumed to stand in some felicitous causal relation to β . For simplicity, our notation omits cases in which β occurs but remains unmeasured, modeling the risk of false positives but not false negatives.

In an RCT, the evidence is limited to the measured variable bundles. Typically, however, researchers’ causal claims reference the ontological events here represented as Greek letters (“enforcement message” and “turnout”), not the measured bundles characterized by Latin letters (“A” and “B”). The definition in (4) makes explicit that the Greek-letter reality behind the Latin-letter measures remains a matter of inference, and this is true even when the causal estimand has internal validity (as in Keele and Minozzi [16]). Under our assumptions, internal validity and textbook identification establish the following specific causal claim as a fact about what has actually been manipulated and measured:

where C bundles the conditions in the setting, which are either balanced or constant across units. The quotes make it explicit that (5) is a claim. Thus, internal validity only warrants claims with respect to the Latin-letter variables – literally only that “the average treatment effect is a causal relationship between variables A and B in setting C ” – not the Greek-letter causal relata and causal conditions that are present in the world and that are of substantive interest. It follows from definition (4) that the fact of claim (5) is not the same as the (generalized) causal process of interest τ , which is expressed semantically as counterfactuals regarding states of α and γ ,

i.e., the substantive, semantic claim about the ontological process of interest – “enforcement messages cause turnout in the absence of affluence.” As our notation makes plain, even if an identification strategy justifies claim (5), that by itself does not deductively justify making the claim about the causal process meaningfully expressed in claim (6). To see this, we expand claim (5) using definition (4) to the equivalent statement,

Plainly, moving from identifying the A T E in (5) to deducing the generalized causal effect τ in (6) requires an extensive set of assumptions about claim (7) that go beyond the assumption of internal validity. Comparing claim (5) to claim (7), expanding A problematizes construct validity of the cause; expanding B problematizes construct validity of the outcome; and expanding C problematizes external validity.

Of course, if researchers literally only care about the measured variables A and B as manifested in the exact setting C , they need not make assumptions about the relationships between A and α , B and β , and C and γ . However, they would then be unable to communicate the meaning of their results beyond saying “Whatever it was we did (designated by the symbol A ), at that one time and place (designated by the symbol C ), had an effect on whatever it was we measured (designated by the symbol B )” – a claim that would never be published and indeed is not even a generalization [35]. Since textbook identification strategies are only with respect to the symbolic representations that indicate measured variables, identification alone can only deductively license a claim such as (5). Claim (5) does not license the meaningful semantic claim (6) that is the verbal statement of interest in applied research [55].

Note that a claim such as (5) can be the result of a deductive test, under assumptions of internal validity, using standard procedures of causal identification. It can be what Gelman and Imbens [14] call a “what if” rather than a “why”-type assessment. In moving implicitly from (5) to (6), however, the researcher transforms a “what if” question among measured variables into a speculative or exploratory “why” accounting of the relata and conditions driving the statistical patterns that support claim (5). This is the case even when claim (5) is identified. When researchers make a generalized causal claim without addressing the relationship between the Latin-letter measured variables and the Greek-letter causal relata and conditions, they are simply hazarding an exploratory guess or “structured speculation” about the underlying causality [67], with the hope that causal knowledge will somehow accumulate coherently from a sequence of results such as (5) [1, p. 23]. This hope for accumulation of knowledge under speculation is similar to when earlier generations of applied statisticians made exploratory guesses and offered verbal assurances about control variables achieving internal validity.

The authors of the Credibility Revolution partially acknowledge this limitation and attempt to place boundaries on claims that are “credibly” established under internal validity through a familiar saying that identification can deductively establish the effects of a cause – that is to establish a fact that a cause occurred by comparing outcomes across counterfactual states of the world – but not the causes of effects that would require naming the actual causes [20]. This distinction fails to establish these bounds, however. Indeed, communicating any description of the counterfactual states over which a cause and effect are detected requires labeling those states in some way. Even if those labels are abstract or generic, this still requires an assumption of construct validity. And to assume the cause will occur in any other time or place, or even generically assume that causal effects are homogeneous, requires an assumption of external validity. Even if one were to try to communicate a finding modestly as an “effect of cause”-type claim, that would not relieve one from considering these aspects of validity.

Because identification is only with respect to measured variables in one setting, identification does not address parts (i), (iii), or (iv) of a generalized causal claim, i.e., the labeling of the relata and the conditions under which the cause will occur. Since a causal generalization is not deductively valid unless all four aspects of the claim are correct, causal identification does not provide sufficient assumptions to deduce a generalized causal claim. Thus, we augment causal identification with what we call causal specification, which formalizes the insights of Shadish et al. [18] into the single expression of claim (7) that shows the equal importance of each type of validity in enabling deductive causal claims. In causal specification, construct validity is present when α and β are correctly specified, and external validity is present when γ is correctly specified.

Causal specification clarifies the intimate relationship between each type of validity and deductive causal inference. Recall that we say a causal claim is “valid” if the claim accurately reflects the ontological causal process found in nature. Since the underlying causal generalization is never directly observed, the deduction can only follow from the assumptions corresponding to internal, construct, and external validity. If any one of the assumptions is missing, the conclusion is not deductively supported. But note as well that if any of the assumptions is false, the claim might follow deductively from the assumptions (and thus be “deductively valid” as commonly used in philosophy) even though it is not a valid causal claim. For causal knowledge to accumulate, the best that the researcher can do is to provide sufficient warrants for each assumption – using theory, research design, qualitative knowledge, or justified intuition.

Causal specification also calls into question any assertion that internal validity has priority over construct or external validity. Such a claim dates to Campbell [68, p. 297] who stated internal validity as a “basic minimum” for science, and when comparing internal to external validity, he writes, “Internal validity is the prior and indispensable consideration.” For a more recent statement, see Guala [69, p. 1198]. The reasoning holds that if an experiment is confounded, nothing can be learned from it, and hence, internal validity has a lexical priority over considerations regarding the other types of validity.

However, in the present framework, deductive validity does not require that the validities must be considered in any specific sequence or ranked in their priorities, or that internal validity must be present before one can consider the other types of validity. For example, it is unclear how one can say that knowing an internally valid (directional, unconfounded) cause occurred must precede knowing what was the cause and what was the effect (i.e., construct validity). Similarly, it is also unclear how one can say knowing an internally valid cause occurred can precede knowing whether the setting has the necessary conditions to enable the cause (i.e., external validity). Because all three validities are necessary, no causal generalization can be successfully deduced without all three. None stands separate and prior.

Table 1 summarizes the key concepts we introduce in this section. In the remainder of this review article, we survey the concepts of construct and external validity as they fit within the framework of causal specification, showing the necessity of each type of assumption to preserve the deductivenss of causal claims. The formal approach using the potential outcomes framework can be extended for each of internal, construct, and external validity. However, we confine those formalizations to Appendix A. In addition, for interested readers, Appendix B shows our arguments in an SCM framework. These appendices can be skipped without loss of continuity.

4 Causal specification for construct validity

Traditionally, construct validity centers on considerations of the quality of observed measures when a criterion measure does not exist, to ensure that the variable that is measured in fact corresponds with the concept of interest [70,71]. In causal analysis, this means the semantic labels assigned to the causal relata are correct [55]. The notion originates in Cronbach and Meehl [72] who proposed assessing whether the pattern of convergences and divergences in a set of correlations meets the theoretical expectations of a “nomological network.” As Borsboom et al. [61] explain, such an analysis of correlations might warrant the match between a measure and the underlying ontological referent, but this only serves as an empirical validation procedure to support an assumption of validity [73].

According to Borsboom et al. [61], a measure is construct valid if measured observations are themselves caused by the underlying (ontological) referent of interest. Referring back to our definition (4), in our framework, a causal generalization is construct valid only if α and β , inferred from observations of A and B , are the real underlying cause and effect. As ontological referents, α and β are latent and so not normally readily measurable. Instead, the correspondence between the measured variables and the intended relata is a (possibly warranted) assumption, governed by considerations of construct validity, just as the presence of internal validity is an assumption. Assigning correct semantic labels “ α ” and “ β ” to the causal relata thus stands as one of the core inferential risks when making deductive causal claims based on the statistical relationship between A and B [48]. Without an explicit assumption and justification for their semantic labels, researchers cannot properly claim to have deduced a generalized causal effect. One knows only that something caused something, not what causes what.

5 Causal specification for external validity

Traditionally, external validity focuses on whether an identified causal effect extrapolates, transports, or is generalizable to other settings [17,18,77–79].^[13] “Settings” include different countries, time periods, populations, contexts, and laboratories. Although this definition is standard, it is often viewed as an unattainable ideal [17,80]. Very few social science studies yield the same results across all settings of human existence [77,78]. Indeed, despite internal validity, RCTs usually yield substantially varying results across settings [80–85]. This is the case, of course, if there has been inconsistent execution of RCTs. However, low external validity in the traditional sense is quite common across RCTs even when executed consistently [80,86,87].

6 Causal specification and paths forward toward deductive causal inference

Our causal specification framework clarifies the assumptions that must be added to prevailing textbook identification strategies to support credible deductive claims about causal effects. In the absence of construct and external validity, the design-based researcher converts a deductive “what if” question to an exploratory “why” question [14]. Researchers might implicitly speculate that α is the active ingredient in A , that B accurately tracks β , and that the causal conditions γ in C operate similarly elsewhere. These, however, are assumptions that underwrite deductive causal claims of the nature of the cause, the nature of the effect, and the scope of the generalization. Failing to make it explicit that these are assumptions undermines the deductiveness of any causal claim even when internal validity is present. Indeed, as practitioners of the Credibility Revolution know well, internal validity itself remains an assumption even after randomization and even after balance tests have been passed [100,101].

The inescapable role of assumptions leads to our first recommendation. Our causal specification framework makes clear that ontological problems of how to assign semantic labels to causes, outcomes, and conditions cannot be solved by statistical procedures [48]. Instead, as Cook [100], Deaton and Cartwright [101], Kocher and Monteiro [30, p. 953], Pearl and MacKenzie [25], Slough [26] – and many others – emphasize, the assumptions that underwrite a design for causal inference are not statistical but instead are derived from theory, and substantive knowledge, including hypotheses about the concrete mechanisms involved [54,91,102] (e.g., that GSL’s prospective jurors were motivated by fear of punishment).

This means that substantive domain knowledge must always be a partner in deductive causal inference. One can develop a theoretical and substantive understanding of the underlying referents that correspond to the actual causal relata and the relevant conditions in settings through immersion in descriptive [103], historical [30], or qualitative empirical evidence [104], or even by well-reasoned intuition [25]. Such theory and substantive knowledge enable researchers to specify relatively more plausible background assumptions concerning construct and external validity. Acknowledging this highlights the essential role of descriptive, historical, and qualitative evidence for scientific progress even within the most sophisticated statistical techniques [30,105–107]. Acknowledging this also highlights the essential role of interdisciplinarity and collaboration between methodologists and substantive experts.

We propose three other directions forward beyond recognizing the inescapable role of assumptions, theory, and substantive knowledge. First, claims regarding construct and external validity must be supported with the same rigor, precision, and care as claims regarding internal validity. As we mention above, the credibility revolution ushered in a normative change about language regarding causal effects. The causal specification framework encourages another normative change toward greater circumspection and modesty about what can be claimed based on internal validity alone. For instance, even if researchers want to keep referring to “causal effects” based on internal validity, we urge a shift to “a” rather than “the” causal effect. Even if one claims “a” causal effect of a certain treatment in one setting, that certainly does not warrant claims about “the” causal effect in general and across settings.

Closely related, we recommend that researchers who wish to make deductive contributions to our understanding of causal processes explicitly specify their α s, β s, γ s and defend the inertness of their θ s. For researchers who already take construct and external validity explicitly into account, this might amount to only a more formal statement of their assumptions. For other researchers, like GSL, this might require confronting and clarifying causal assumptions through substantive knowledge that they would otherwise disregard and leave implicit in the background.

Second, the scientific community should be encouraged to conduct what Ankel-Peters and colleagues [108] call “policing replications” of studies evidencing internal validity. Rather than presuming that strong internal validity and causal identification provide strong support by itself for causal generalization, the scientific community should treat such evidence as only preliminary. Such evidence should always need to be tested in other settings to understand external validity, and such evidence should always be scrutinized for construct validity.

In recent years, many have tested for variation in treatment effects across settings [109,110]. Often, such replications reveal serious questions about the generalized scope of a causal effect. For instance, Henrich and colleagues [86] show that internally valid psychological experiments often fail to generalize outside rich, Western democracies. Development economics has seen similar failures for RCTs to replicate [82,83,85]. By demonstrating that causal effect heterogeneity is pervasive and large, such replications illustrate convincingly the gap between an internally valid effect and a causal generalization.

Beyond external validity, policing replications can also open up questions about construct validity. Such studies force us to ask harder questions about whether an observed treatment is actually validly measuring the same active ingredient of the underlying construct across various settings and respondents. For instance, Gilbert et al. [111] criticize the Psychology Reproducibility Project for assuming different treatments have the same construct validity in replications of experiments across many settings. Gilbert and colleagues point out that it is unclear that an original experiment asking Israelis to imagine the consequences of military service reflects the same underlying construct as a replication asking Americans to imagine the consequences of a honeymoon. Similarly, Stich and Machery [112] demonstrate that philosophical cognition experiments exhibit heterogeneity across settings in part because of demographic differences in what treatments mean and how they are understood by subjects. Their “geography of philosophy” project improves validity by exploring how reactions to thought experiments vary across settings (i.e., external validity), what features of the treatments drive participant reactions (i.e., construct validity of the cause), and how to interpret heterogeneous participants’ reactions (i.e., construct validity of the effect).

Third, we recommend a research agenda seeking to understand the external and construct validity challenges of internally valid studies. Here, researchers take internally valid studies and then model why and how variation in construct and external validity matters. Beyond showing that some experimental psychology lacks external validity [86], Muthukrishna and Henrich [113] investigate how cultural distance from the US explains the variation in treatment effects. Price and colleagues [114] reexamine 194 RCTs to show how anti-Black racism is a critical condition in communities that moderates/undermines the effects of psychotherapy on youths. Also, Price and colleagues [115] show that sexism is a critical condition across settings that moderates how well psychotherapy helps girls. Of course, these studies will need to establish internal validity in the causal effects of such conditions across settings as well.

Finally, we readily acknowledge that this article (intentionally) raises more questions than it can answer. Our hope is to push causal inference scholars to re-examine the concepts of validity precisely because there are so many open and implicit questions. Specifically, it is essential to debate such things as, what criteria should we use to assess whether an article addresses construct and external validity satisfactorily? How best to navigate trade-offs between internal, external, and construct validity? What practical tools developed in the causal inference community can provide a more concrete path forward to best warrant assumptions that causal inference scholars currently leave implicit?

7 Conclusion

Social scientists typically aim to deduce general knowledge about what causes what in what conditions, and not just historical knowledge that something caused something one time in one setting in the past, i.e., social scientists aim to deduce valid causal generalizations. The tight linkage between the concept of internal validity and the concept of causality is encoded in the causal frameworks that have governed the Credibility Revolution. The RCM [2,20] and SCM [3] have made tremendous contributions while being centered on the problem of unconfoundedness and internal validity. However, the Credibility Revolution has not adequately recognized the necessary role of external and construct validity for causal deduction. We explain how these omissions undermine the Credibility Revolution’s own goal to understand causality deductively.

In this essay, we develop a causal specification framework to critically review the literature on quantitative causal inference. The challenge of causal specification is not only the challenge of confirming that in fact something caused something in one setting (the focus of internal validity) but equally the challenge of correctly labeling the nature of the cause, the nature of the effect, and the conditions under which the generalization holds. By itself, even the most rigorous proof of internal validity shows only that some aspect of the manipulation ( A but not necessarily α ) caused some measured outcome ( B but not necessarily β ) in one setting ( C , typically leaving γ implicit). Construct validity is achieved when the semantically asserted cause and effect are the actual cause and effect. External validity is achieved when the scope of the generalization is correctly specified. Unless all three validities are present, a substantive claim that “ α causes β in γ ” is false. Assumptions about all three types of validity are required for deduction; none has priority. Internal, construct, and external validity are three legs of the stool of causal generalization. And the causal generalization is only as strong as the weakest leg.

The causal specification framework shows that the textbook identification assumptions within “credible designs” are insufficient for deducing generalized causal claims. Identification focuses on internal validity but typically neglects construct and external validity. Social scientists who wish to make generalized and deductive causal claims instead must attend equally to internal, construct, and external validity. Specification of assumptions about construct and external validity can augment current approaches to identification and enable researchers to make coherent deductive causal claims. Moreover, these assumptions tend to inevitably rely on substantive, theoretically grounded, and verbally justified labeling of the relata for construct validity, and of the conditions for external validity. These additional assumptions regarding the relata and conditions are as necessary for deducing a generalized causal claim as are assumptions of internal validity.

If applied researchers ignore construct and external validity when stating causal claims, they mistakenly convert an intended deductive claim into a claim based on exploration and speculation – contrary to the fundamental goals of the Credibility Revolution. Our framework for causal specification corrects this, and offers a means for applied researchers to preserve the deductive nature of their claims not only at the level of measured variables but also – more importantly – at the level of relata and conditions. In this way, causal specification clarifies the additional assumptions required for the Credibility Revolution to achieve its aspirations of understanding causal effects.