Some comments on robustness in comparative grammar research: commentary on “Replication and methodological robustness in quantitative typology” by Becker and Guzmán Naranjo

Over the last two decades, an increasing number of scientists have expressed worries as it has been increasingly clear that in a range of fields, a large proportion of scientific results could not be repeated (or “replicated”, or “reproduced”). Is this perhaps also a problem for comparative linguistics?

For comparative grammar (or typology), reproducibility was first discussed two decades ago (Plank 2006), and as more and more research projects use advanced quantitative methods, it is of course important to have some discussion of which are the best methods, and whether different statistical approaches give the same results. It seems clear from discussions in other fields that the way scientists use statistical methods is often highly problematic. In 2005, John Ioannidis shocked researchers by suggesting that “for most study designs and settings, it is more likely for a research claim to be false than true” (Ioannidis 2005). And as recently as 2023, anthropologist Richard McElreath made some highly critical comments:^[1]

“[T]he status quo is the biological and behavioral sciences is terrible. It makes little sense to invest in ambitious research projects when a typical researcher has no ability to define a non-null model of a phenomenon, to explore the implications of such a model, and to evaluate those implications with evidence. Success in the sciences does not, in the present day, depend upon those skills. But we should fix the foundations before wasting more public money.”

Thus, it is no doubt important to examine the tools of quantitative inference closely, just as we need to critically examine other aspects of our methodology – be it elicitation methods, or comparative corpus methods, or transcription methods. I also find it important to compare competing approaches, rather than to pursue only one approach without considering alternatives. The contribution by Becker and Guzmán Naranjo (2025), henceforth B&GN, is thus highly welcome, and may play a similar role in advancing methodological awareness in quantitative typology as was played by Bell’s (1978) seminal paper about language sampling. I also welcome their plea (in the supplementary Appendix) for complete and transparent datasets (ideally including example sentences), something that could not be taken for granted until recently.

In this short comment, I will discuss a few further factors that seem to be important for robustness and reproducibility in our field, in the spirit of the “manifesto for reproducible science” (Munafò et al. 2017). As B&GN observe, robustness also means stability of results (i) across different language samples and (ii) alternative analyses and annotations. Munafò et al. furthermore mention quite a few further possibilities of enhancing robustness, such as (iii) protection against cognitive biases, (iv) improving methodological training, (v) encouraging team science, and (vi) improving incentives.

1. Robustness across different language samples: B&GN mention a few papers where this has been addressed, but there could (and should) of course be a lot of additional work trying to replicate earlier results with new data. Many readily searchable open-access grammars have been published over the last 20 years (many of them accessible via ALT’s Grammar Watch site),^[2] and there is no reason why such replications should not be assigned to masters or even bachelor students. However, some linguists have questioned whether comparative-grammar research should be based on language samples, preferring analyses based on phylogenetic trees (e.g. Jäger and Wahle 2021; Levinson et al. 2011). The advantages and downsides of both methods have not been discussed thoroughly by typologists,^[3] but in any event, when a phylogeny-based study basically confirms the results of earlier sample-based studies (as in Jäger and Wahle’s results on word order correlations), this may be taken as reassuring. Phylogeny-based methods seem to work best (or be applicable exclusively) when one has data on a very large number of languages (e.g. the entire Grambank sample, as in Verkerk et al. 2023), which is rarely the case in typological studies.

2. Alternative analyses and annotations: This is perhaps the greatest stumbling block for reproducibility in comparative linguistics, because the comparative concepts used for cross-linguistic comparison are not standardized in the field, and traditional terms are often understood and used in diverse ways across scholars and subcommunities. For example, what exactly is the difference between “case (marking)” and marking by an adposition? If one wanted to replicate claims about case (e.g. the universality of Blake’s (2001: 156) “case hierarchy”: nom > acc/erg > gen > dat > loc > abl/inst > others), one would have to have a clear way of distinguishing between case affixes and adpositions. The difficulty is mentioned in a paper by two generative linguists:

   “The truth of [the claim above] can only be judged … if one is able to distinguish reliably between an object NP in dative case (which can be agreed-with in languages of the non-Indo-European type) and PPs with a directional adposition like English to (which cannot be agreed with in either type of language). Settling such questions takes considerable work, work of exactly the type that the methodology of generative syntax requires and encourages.” (Baker and McCloskey 2007: 292)

   Baker and McCloskey emphasize that “settling” such questions is “labour-intensive”, but every practitioner of the field knows that they cannot be settled in the sense that all linguists (or all generative linguists) agree – there are simply too many moving parts in the generative approach, despite frequent appeals to “restrictiveness”. Over the last few decades, generative linguists have engaged in many discussions and have discovered a lot of interesting phenomena, but none of the larger issues or analytical questions has been “settled”. By contrast, in the comparative-concept approach (Haspelmath 2018), one defines cross-linguistically applicable concepts in a uniform way, but if one uses traditional terms (as is usual in typology), the definitions are often considered unsatisfactory by other linguists. For example, Dryer (2005) defines article words (as opposed to affixed articles) as elements that do not always occur directly next to the noun, so that the Basque article -a is regarded as a word, not as an affix. Because it is written as a suffix (e.g. etxe ‘house’, etxea ‘the house’, etxe berria ’the new house’), this decision is regarded as unintuitive by many. Thus, typological “analyses” are either subjective (as in generative grammar) or unintuitive (as in the comparative-concept approach), which means that the details of annotation in cross-linguistic studies are often contentious, and cross-linguistic databases such as WALS or Grambank are often regarded as “unreliable”. In the perception of many linguists, this is a far more evident source of uncertainty than non-reproducible quantitative analyses.^[4]

3. Protection against cognitive biases: This contributing factor to robustness has been discussed repeatedly in metascience, but it has played almost no role in the discussions that linguists have had. As Munafò et al. (2017) note, “an effective solution to mitigate self-deception and unwanted biases is blinding”, an approach that has long been used by experimentalists in biomedical fields. In comparative grammar, a principal investigator could perhaps apply “blinding” in the sense that she might instruct her research assistants to examine particular aspects of languages without telling them about the hypothesis that she is interested in. However, de facto linguists have strong cognitive biases, and often they are not even aware of their groupthink: They often “adopt” a certain framework at the beginning of the research and analyze the incoming data from the perspective of that framework and its commitments. The framework is typically an approach practiced by a community of scholars (perhaps with their own conferences and journals), which (by groupthink) reinforces the tendency to view the data exclusively from a particular perspective. This particular problem may not be so acute with large-scale studies of the type discussed by B&GN, but otherwise it is unfortunately quite typical of the field of (non-quantitative) linguistics.^[5]

4. Improving methodological training: This is a very practical matter that is perhaps less relevant to a small field such as typology, because it is not possible to organize larger training sessions for the small number of comparative grammarians. Examining a substantial number of languages worldwide requires substantial background knowledge and willingness to engage with highly technical grammatical descriptions, and acquiring a comprehensive knowledge of statistics in addition is a tall order. Psychologists have included training in statistics in academic curricula for many decades, but in linguistics, this is still rare.^[6] Moreover, in view of the uncertainties surrounding different statistical approaches (McElreath 2020), it is not immediately clear what aspects of statistics one needs most urgently. Perhaps a solution could be collaborative research, or team science, i.e. point (v).

5. Encouraging team science: Munafò et al. (2017) note that many research results are highly uncertain because of low statistical power, and this is certainly true of comparative grammar: Hypotheses about typological connections have often been based on just a few languages, or just languages of particular world regions. In the field of psychology, the “Many Labs” project brought together dozens of psychologist teams in order to increase the replicability of psychological research results (e.g. Klein et al. 2014), and there is a more recent project in developmental psychology called “Many Babies” (Frank et al. 2017). So could there be something like an analogous project “Many Languages”, where dozens of linguists working in different locations collaborate to improve the methodology of comparative grammar? The challenges of experimental research (as in psychology and psycholinguistics) are surely different, but more collaboration will probably be helpful in linguistics as well. There is a great deal of interesting research by individual scholars, often in dissertations, that involves many languages, but does it add up to something bigger, and are the individual studies reliable? Our knowledge would be more certain if we had more collaboration, although comparative linguistics is of course a far smaller field than psychology, and there may not be sufficient incentives (which brings us to the next topic).

6. Improving incentives: Scientists are driven by curiosity, but also by funding options, and while they want to improve their science, they also need to survive. Improving reproducibility thus also requires a way to incentivize it (basically, to fund it). In different disciplinary context (open hardware for microscopy), physicist Julian Stirling observes:

   “Making hardware open source takes considerable time, the noble benefits … do not align well within our current systems for ranking research” (Stirling 2024: 2).

   The same applies in many other fields, including comparative grammar. Current incentives in terms of career prospects and project funding emphasize innovative research, not necessarily robust research. For example, the European Research Council funds “ground-breaking research” that “leads to advances at the frontiers of knowledge” (ERC 2025). Whether such innovations are robust seems to be of secondary importance. Open science and reproducibility efforts have played an increasing role in various fields and have attracted some funding over the last two decades, but whether there will be sustained funding also depends on high-level political decisions, and not so much of what individual researchers do. In a glossy university magazine, a headline such as “Two thirds of last year’s research papers successfully replicated” will not impress readers too much. But of course, the mood may shift, and our societies may want researchers to pay much more attention to robustness, especially in a context when public trust in science can no longer be taken for granted (e.g. Cologna et al. 2025).

B&GN’s paper will hopefully lead to further reflection and to improving comparative grammar research even more, although we should keep in mind that large-scale comparative research is (and will always be) a small corner of the larger field of language science. The great majority of linguists will always study only one major language, and will be able to compare at most a few languages (e.g. Chinese and English, or Spanish and Portuguese, or Polish, Ukrainian and Russian). And there is of course a great danger that linguistics will continue to be subject to all kinds of biases, not only political biases, but also biases deriving from particular perspectives such as strong convictions either about innateness or non-innateness (cf. Pinker 2003), biases deriving from the perceived success or non-success of Large Language Models (cf. Piantadosi 2024), and so on. Bias-free and robust research may be more difficult than we used to think, but there is no alternative to trying harder.