Replicability all the way up: commentary on “Replication and methodological robustness in quantitative typology” by Becker and Guzmán Naranjo

1 Why it is important to ensure replicability in data collection and classification

The practice of making typological datasets publicly accessible has fortunately become increasingly widespread. Researchers typically verify datasets carefully before release. As B&GN note in their appendix, such datasets should ensure transparency regarding the methodological choices that led to a particular classification of the data.

In this regard, we highlight the comment field model in Grambank (Skirgård et al. 2023) as an example of best practice. Recording the rationale for a given classification, this field is particularly valuable because it allows researchers to revise annotations if new data emerge or if their classification criteria differ from those of the Grambank compilers. In contrast, resources like WALS (Dryer and Haspelmath 2013), though highly valuable, remain somewhat opaque, often leaving users uncertain about the reasoning behind a given classification.

We propose that B&GN’s guidelines also include this recommendation: once theoretical and methodological choices are clearly stated, datasets should offer a computationally simple way to reverse or modify them – reversing binary or ternary classifications (like “Y/N/?”), and modifying multi-valued categories. This is not simply a matter of coding hygiene: it is a precondition for enabling rigorous and cumulative science. While this may seem a simple, even trivial, task, its importance should not be underestimated. Otherwise, biases or misrepresentations introduced at the outset risk being carried through the statistical analysis. These biases may often seem like mere background noise, but can we really be sure that is always the case?

2 Replicability, common sense, and the need for a social layer

An important observation in this regard is that while replicability in statistical modeling is crucial for examining the same data from different perspectives, there are cases – such as those noted by B&GN (e.g., their discussion of Chen 2013) – where replicating the statistical analysis is unnecessary to recognize fundamental theoretical weaknesses. In Chen’s study, the hypothesis of a statistical link between future tense marking and a society’s propensity for saving is poorly formulated for several reasons; the fact that the absence of a dedicated future marker does not prevent speakers from referring to the future, or the reliance on a naïve interpretation of linguistic relativity.

The same applies to other, more sophisticated quantitative studies. For instance, Blasi et al. (2019) propose that the shift from a hunter-gatherer to an agricultural society led to an increase in labiodental sounds. While replicating this study with alternative statistical methods might yield valuable insights – open commentaries to Blasi et al. (2019) suggest that accounting for factors like distance from Africa (Tarasov and Uyeda 2020) or consonant inventory size (Berthommier and Boë 2019) produces different results^[1] – the study’s core assumptions remain difficult to substantiate. As Huijbregts (2019, 2020) points out, for instance, changes in bite configuration did not possibly alter the universal phonetic capabilities of humans, which remained unchanged: labiodental sounds do not constitute an evolutionary innovation, as the phonetic features required to produce them were already available in pre-Neolithic human speech systems.^[2] Other significant methodological limitations can be formulated as follows: First, can we reliably ascribe phonetic substance to the reconstructed phonemic inventories of proto-languages? Likely not. These reconstructions are better understood as symbolic representations of systematic correspondences between attested languages – a kind of “warranty” of genetic relatedness rather than actual phonological systems. Second, even if we assume that a reconstructed phonological system reflects a real linguistic entity, the pervasive nature of phonetic variation in living languages is an aspect of the Implicit Uniformitarian Assumption (as formulated and criticized by Moran et al. 2021) that cannot be easily dispensed with. This raises an important question: what ensures that phonemes in the hypothetical proto-language speech community were not realized with considerable phonetic diversity?

It is important to clarify that the aim here is not to diminish the value of Blasi et al.’s quantitative work but rather to emphasize the necessity of two further dimensions of scientific transparency: one is the opportunity for open commentaries on quantitative studies, a practice increasingly adopted by academic journals, but perhaps not so common in linguistics (and linguistic typology) journals. This approach fosters a critical discourse that extends beyond mere replicability, ensuring that both the potential insights of a given classification and its possible limitations can be openly examined and debated. The other one is the opportunity for datasets in typological research to incorporate a social layer, understood, following Cysouw (2007: 63), as an annotation layer intended to foster collaboration and discussion. This layer should minimally include information relevant to a specific research question and to the process of annotation and classification, allowing others interested in the same (or a closely related) topic to easily access the data and draw their own conclusions. Only the incorporation of such a social layer into scholarly endeavors will enhance the robustness of scientific inquiry and contribute to a more dynamic and self-correcting research environment.

3 A case study: investigating the impact of literacy on grammatical structures

In this final section, we draw on an ongoing research project to highlight the importance of implementing a social layer and ensuring methodological transparency, and thus replicability, at the upstream stages of research. These are the stages at which theoretical constructs are operationalized, and data is selected, annotated, and interpreted. Our study on the possible effects of literacy on grammatical structures (Mauri et al. 2025) offers a concrete example of why replicability must extend beyond statistical modeling to include the earlier, often less visible, phases of the research process, together with the challenges this entails.

A major difficulty we have encountered in the very first stages of our research concerns the availability and consistency of literacy data in cross-linguistic repositories. Not only is the information often incomplete or out of date, but it is also based on unclear definitions of what counts as “literacy”. Is literacy to be measured at the national level or at the level of specific language communities? Should it include second-language literacy (e.g., literacy only in a second, often official language), and in this case, how could we disentangle its effects? What thresholds of reading and writing proficiency should be considered relevant for evaluating potential structural effects?

These questions are not purely technical, but they shape the very structure of the dataset and influence the formulation of the research questions. They also affect how we interpret co-occurrences between grammatical features and sociolinguistic variables. For instance, observed patterns in clause combining may correlate with the presence of literacy, but could also result from other sociopolitical processes such as standardization or education policy. This raises the question: are we studying the effects of literacy itself, or of the broader historical and ideological apparatus with which it tends to co-occur?

Our experience suggests that replicability in broad studies like ours, i.e., on phenomena that are somehow controversial, both at the definitory and operationalizing levels, cannot be achieved without full transparency regarding how the foundational theoretical decisions are taken and, crucially, without enabling others to reconfigure the underlying assumptions and categorizations. Inspired by B&GN’s emphasis on methodological robustness, we advocate for the implementation of a social layer within the dataset itself, allowing for annotations to be revisited, challenged, or recalibrated in light of new data or alternative criteria.

Finally, given the complexity of sociolinguistic variables like literacy, which require a deep and expert knowledge of linguistic communities, we see considerable potential in crowd-sourcing initiatives and community-based data collection. These approaches could both enrich the empirical base and embed the research process within a more dialogic and reflexive framework. But again, such strategies are only useful if they are underpinned by transparent documentation and by mechanisms that facilitate the collaborative critique and refinement of the data.

In sum, before applying any statistical model – whether simple or advanced – it is essential to ensure that the data and the theoretical scaffolding supporting it are themselves open to replication and revision. As our case study illustrates, methodological robustness must begin well before model fitting, and it depends as much on epistemological reflexivity as on technical replicability.