Context sensitivity and failed replications in linguistics – a reply to Grieve

Grieve (this issue) argues that linguistics must expect to face low replication rates due to the inherent context sensitivity of linguistic phenomena. Thus, while original and replication studies (and, more generally, studies addressing the same phenomenon) may target the same population of language users and structures, studies might systematically differ in their sampled populations and the contexts in which the data are collected. Similar concerns have been raised for experimental research in the social sciences (Barsalou 2016; Cesario 2014; Van Bavel et al. 2016). I fully agree with Grieve that linguists should be aware of these caveats when interpreting and comparing empirical studies concerned with complex social phenomena such as language.

Differences in the sampled population and changes in the social context can and should be considered as possible explanations for conflicting findings. Virtually any linguistic phenomenon is context-sensitive. Some contextual factors relevant for some aspects of language will always vary from one study to the next. Consequently, it would never be possible to determine whether a “failed” replication attempt is (i) due to the fact that the original effect was a false discovery or whether (ii) the sampled population or context between studies has sufficiently changed. Instead of accepting (ii) as an inevitable fate, linguists should direct their efforts towards convincingly ruling out difference in both the sampled population and context effects. Without such efforts, the field lacks one principled way of integrating empirical findings and, ultimately, advancing our understanding of language in an effective manner (see also Zwaan et al. 2018).

Linguists make generalizations under the assumption that they know enough about a phenomenon in order to be able to infer when and how that phenomenon can be observed. Without being able to rule out, with sufficient confidence, the possibility that discrepant results are due to differences in the sampled population or the context in which the data were collected, it is technically impossible to learn something useful from the observed discrepancy. Being able to specify the contexts that reliably result in an effect allows for independent verification and provides a solid footing for studies that extend the original result. If linguists fail to (or cannot) specify these contexts and if they dismiss failed replications due to context sensitivity, scientific progress in linguistics is seriously impeded.

As Grieve rightfully points out, the inherent social and context-dependent nature of language represents a challenge for replicability in linguistics. There are at least three strategies to address this challenge: First, empirical linguists should be explicit about potential boundary conditions for their generalizations (Simons et al. 2017), e.g., they should identify their target populations with sufficient detail. Second, they should provide detailed descriptions of the sampled population and/or contextual factors that might have affected the results such that future replication attempts can take these factors into account (Brandt et al. 2014). Third, they should fit more conservative statistical models to their data, taking varying effects of, e.g., participants, stimuli, contexts, and other such factors into account in order to draw conclusions that better generalize over different relevant sources of variation (Yarkoni 2020).