Authors’ response to “Replication and methodological robustness in quantitative typology”

2.6.1 Modeling particular contact scenarios

From the qualitative literature, we know that many types of contact effects are not uniform across the world, and some can be relevant for particular regions only or even affect single pairs of languages. Related to this, Miestamo & Sinnemäki highlight the empirical observation that, depending on language ideologies, contact can also lead to linguistic differentiation instead of convergence. Another example is mentioned by Dryer, who notes that marriage patterns in Papua New Guinea lead to women “marry[ing] into villages where the language is more than ten languages away from their native languages”. In such a setting, contact between two languages thus leapfrogs several languages. As far as we can gauge, particular and potentially idiosyncratic effects of language contact like these ones are the hardest contact-induced effects to model computationally. Building a general model structure that can take into account language ideologies is extremely difficult and moreover requires very detailed socio-linguistic data.^[5] We can say for certain that Gaussian processes are likely the wrong tool for this type of linguistic situation, but we do not know what would be an appropriate tool. We remain excited about new developments in this area.

2.6.2 Euclidean versus other types of distances

In their commentary, Shcherbakova et al. criticize our approach for using Euclidean distances. They argue that Euclidean distances are only (poor) approximations of the real separation between communities because they assume a flat and deformed earth. Regarding Euclidean distances versus other types of distances, Guzmán Naranjo and Jäger (2023) introduce and compare topographic and walking distances and show that, very often (although not always), topographic and walking distances do indeed perform better than Euclidean and Geodesic distances. These types of distance metrics can be used in Gaussian process models, but they require the model to be written in Stan. As was highlighted by Miestamo & Sinnemäki, Haspelmath and Di Garbo, the modeling choices presented here may already present a steep learning curve. We agree with that, and we wanted to encourage others to use our models and therefore avoided writing in Stan directly, as it requires familiarity with another programming language in addition to R.

Furthermore, there are some additional, but minor complications when using more realistic distance measures. Topographic and walking distances require some additional work to get them into shape to be included in a Gaussian process. Computationally these distances are also much more costly to compute, and thus, they will necessarily represent an additional layer of complexity for researchers.

Another relevant point that we do not discuss explicitly in the paper is that exact Gaussian processes are computationally very costly. This means that they scale poorly with the number of languages in the sample. While fitting a Gaussian process with 100 observations is very fast, it becomes slower once a sample has more than 500, and anything above 1,000 observations will be very slow. A practical solution is the use of approximate Gaussian processes (Riutort-Mayol et al. 2023), but these are only available with Euclidean distances. The consequence is that if one wants to work with very large datasets like Grambank, exact Gaussian processes with more complex distance metrics become impractical to work with.

2.6.3 Macro-areal effects versus local contact effects

Both Dryer and Seržant note the apparent contradiction between the assumptions of a Gaussian process that contact quickly decays to 0 after a few hundred kilometers, and the potential existence of large macro-areal effects in the typological distribution of linguistic features. We agree with Dryer’s interpretation of the situation, namely that large contact areas emerge as the result of serial, (relatively) short-distance contact events. What he calls snowball effects, Guzmán Naranjo et al. (2024) have also referred to as water bucket effects.^[6] The point is that a series of local contact events can produce very large linguistic areas. This situation does not represent an issue for Gaussian processes, and water bucket effects can be captured without problem: Gaussian processes can build large contact zones from local contact situations.

2.6.4 Modeling spatial relations versus real contact relations

Both Hammarström and Seržant make an important conceptual distinction between the spatial effects that we model and the (many different types of) contact situations that have occurred and occur in the real world. This leads to the important overall question which subsumes most other comments discussed in this section, namely how the spatial relations that we model actually relate to real contact phenomena between languages.^[7] In short, we do not model language contact directly, but use spatial relations between languages as a proxy. It is extremely difficult, if not impossible at this point, to build a statistical model that captures (direct) language contact on a global scale in an adequate way. We have pointed out some of these complexities in our replies above, but we feel that this fact is still under-appreciated in the typological community, and that some linguists (not the authors of this discussion) right out dismiss approaches like ours because certain details about language contact are not properly captured. We sympathize with the frustration from an expert’s perspective on language contact, but we do not think that the consequence should be to stop trying to capture contact in quantitative models, even if it is via spatial relations. What can help to advance modeling techniques of contact effects in quantitative typology is to start out with smaller areas and more controllable (socio-)linguistic realities, where we can try to find modeling solutions that fit the reality better. This is what we have tried to do, for instance, for asymmetric language contact and expansion effects for Polynesian languages in Guzmán Naranjo et al. (2025) and for the Americas in Urban and Guzmán Naranjo (2025).

To sum up, we fully agree that our approach to space in the present paper cannot capture but a fraction of the complexities of real language contact. At the same time, we believe that the methods we employ in this paper are a reasonable compromise for most studies, and that they offer an improvement over other quantitative approaches that include no spatial component at all. We will continue to work on developing and testing new techniques that take the complexities of real language contact situations more seriously, but it remains challenging to adapt and use such techniques in large-scale typological studies.

3.1.1 Methodological pluralism

In her commentary, Di Garbo provides a wider perspective to our study by distinguishing four types of researcher profiles in typology, from documentary linguists to statisticians. She shows how different researcher profiles and perspectives lead to different types of typological studies that make up an incremental workflow from primary data collection to exploratory comparative work to statistical modeling of distributions in space and time. Di Garbo highlights that each of these perspectives has and requires different methodologies, and she rightly points out that “not all typological investigations are amenable to statistical modeling, it is important to underscore that inferential statistics may not be the only way to validate research results in our field”. We fully agree with this statement; in our paper, we focused on quantitative typological studies only, for which we do think that statistical modeling is relevant and important. For other types of typological research, especially qualitative and more exploratory work, statistical modeling and statistical bias control are irrelevant. Di Garbo’s commentary thus emphasizes the importance of the plurality of approaches and perspectives in typology and makes a call for an inclusive view of the field, where none of the approaches “should be conceived of as scoping above the others.” We completely agree with this point. In particular, we second the importance of descriptive and qualitative exploratory linguistic work as particularly relevant. Without a solid understanding of the linguistic realities, quantitative and statistical approaches cannot contribute much to a better understanding of why linguistic properties are distributed the way they are.

3.1.2 Sampling and statistical testing versus statistical bias control

Another critical point raised by Di Garbo is that sampling plus statistical testing is a useful method to make hypotheses and discover new potential paths of research. She writes that “[s]tatistical testing can thus be a valid way to (start) investigating small to moderately large typological datasets.” In principle, we agree with this point in that we do not want to promote methodological gatekeeping for sample-based typological studies and suggest that they should only be carried out and published using advanced statistical modeling. As mentioned above, we also agree that much of typological work does not need to be quantitative at all.

What we would argue, though, is that if a primarily qualitative typological study is supposed to include a first quantitative overview of the cross-linguistic distributions, showing the raw counts and proportions is more useful than additional statistical tests. As we mentioned in Section 7.2.2 of our paper, others have shown in detail why statistical tests such as Chi-Square tests or t-tests are often not very useful. Their assumptions are not met in most cases, which makes their results irrelevant. Therefore, in cases of primarily qualitative studies, we see no issues with simply reporting the raw distributions, which can also serve to formulate hypotheses to be tested in a further study.

In cases of typological studies with a clear quantitative focus, we are less certain about how suitable some form of probabilistic sampling with statistical tests may be, and we think that there is evidence for statistical modeling with statistical bias control to be advantageous. We base this on several previous studies that have shown how clear effects found in studies using no or less detailed statistical bias controls are considerably weakened or disappear once such controls are included (e.g. Becker et al. 2023; Guzmán Naranjo et al. 2025; Hartmann 2022; Hartmann et al. 2024; Jaeger et al. 2011; Roberts et al. 2015; Van Tuyl and Pereltsvaig 2012). Not all of these studies were based on probability samples to begin with, but some were, which has two possible consequences. We can either use statistical bias controls to model the remaining dependencies between the languages in the sample, which often represent contact or areal relations between languages. The alternative is to incorporate the contact and spatial component more seriously in the sampling method used, where the focus has traditionally been more on phylogenetic relations between languages. Dryer’s (2018) study that we tested (and much of his previous work using macro-areas) is a case in point.

3.3.1 Alternative analyses and annotations

As we understand it, the issue of robustness in the annotation and analysis of linguistic data across studies is one of the most important aspects for Haspelmath in terms of replicability in typology. He writes: “This is perhaps the greatest stumbling block for reproducibility in comparative linguistics, because the comparative concepts used for cross-linguistic comparison are not standardized in the field, and traditional terms are often understood and used in diverse ways across scholars and subcommunities.” We agree with his assessment of the status quo in that concepts and terminology are used in many different ways across studies, scholars and research areas. Yet, we do not think that this is an insurmountable issue; as suggested by Mauri & Sansò, it requires more transparent documentation of the annotation process and more explicit definitions of the concepts and terms used. We understand this point as a reminder to us all to be as explicit as possible in our work when defining linguistic terms, even if (or especially if) the terminology seems well established. We also think that including linguistic examples to illustrate each of the annotation decisions can help to remedy unclear terminology, as it allows the reader to check and follow the annotation decisions for themselves.

As for the robustness of findings based on alternative analyses and annotations, we think that if two different operationalizations of a comparative concept produce very similar results, it provides evidence for the general classifications to be linguistically meaningful. If the two operationalizations lead to different results, we should try to understand which of the theoretical decisions could have led to these differences.

3.3.2 Protection against cognitive biases

Haspelmath rightly points out that cognitive biases can have an impact on the results of a study. This aspect of robustness relates to what Smith discusses in his commentary as researcher flexibility or researcher degrees of freedom. Haspelmath notes that while in experimental research, blinding is a possible shield against this type of bias, it is less so in comparative grammar studies. He concludes that “[t]his particular problem may not be so acute with large-scale studies of the type discussed by B&GN, but otherwise it is unfortunately quite typical of the field of (non-quantitative) linguistics.” We partially disagree with this last point in that we think that cognitive biases are also highly relevant for large-scale quantitative typological studies as the ones that we focused on in our paper. There is a garden of forking paths when a researcher builds a statistical model, and every decision has an effect on the results of the model. Some of the many choices include: the statistical software used, choice of likelihood, pre-processing of predictors, linear versus non-linear effects, types and number of controls, multiple models versus multiple regression, Bayesian versus frequentist framework, prior selection, model checks, etc. Every single one of these choices has an impact on the final results of the model, and we do not have an agreed upon setup in typology (and likely never will). Researcher bias could easily influence these choices to guide the model toward the desired result. The degree of such biases, especially concerning modeling choices, is exactly what we wanted to assess in our study by testing previous studies for robustness using the original data but different statistical methods. As we mentioned before, methodological pluralism is desirable, and paired with replication studies of various kinds, it can help us to understand better the impact of cognitive biases in (quantitative) typology.

3.3.3 Encouraging team science

Regarding team science, Haspelmath mentions the “Many Labs” project, and discusses whether something similar could be implemented in typology “[…] where dozens of linguists working in different locations collaborate to improve the methodology of comparative grammar”. This is a very interesting and innovative take on robustness testing and it would make much sense to combine efforts this way.^[8] We see different ways how a typological “Many Labs” project could play out. On the one hand, it would be fairly easy to do something like a shared task in quantitative typology. One would need to provide a typological dataset and have several teams build different models to analyze the same data. On the other hand, we think that some existing typological work sort of fits Haspelmath’s “Many Languages” description, e.g. collaborative projects with a common theoretical framework such as the Leipzig Valency Classes Project (Malchukov and Comrie 2015a, b) or the Mainz Grammaticalization Project (Bisang and Malchukov 2020a, b). What we take away from this for the future is that a potential “Many Languages” collaboration is something to keep in mind and certainly worth a try to organize.

3.5.1 Researcher degrees of freedom

Smith correctly points out that our robustness analyses relate to a well-known phenomenon from experimental psychology, namely that researchers are faced with a series of choices when analyzing data, and that these choices can have a large impact on the results of the analysis (Gelman and Loken 2014). Two researchers analyzing the same dataset can reach very different conclusions if they make slightly different (conscious but also unconscious) choices, each of which can lead to more analytical differences further down the line. The difficulty lies in the fact that it is often not obvious that there is a single set of correct choices. We appreciate the wider perspective from experimental psychology that Smith offers, and the reminder that researcher degrees of freedom are well known in other disciplines. Our impression is, though, that this has not been discussed much in general linguistics, one notable exception being the “Many Speech Analyses” project, which examined analytical flexibility in phonetics (Coretta et al. 2023). Smith is correct in pointing out that, against this background, it is not surprising that we can replicate some but not all results in our study despite using the original data.

While experimental fields have found pre-registration to be a useful mechanism to protect against researcher bias, it is less clear that pre-registration would be useful for more methodological robustness in typological research. In qualitative or more exploratory typological work, many of the decisions regarding how to categorize the data and which phenomena to include and to exclude are made and updated after having seen and analyzed the linguistic data. To an extent, this also holds for quantitative typological studies that build their own sample with grammars.

3.5.2 Triangulation with experimental linguistics

Smith brings up another important aspect, i.e. the relation between typology and experimental linguistics, the latter of which can help “to test cognitive and interactional mechanisms hypothesized to be responsible for potential universals.” This was not a main point in our paper, also because we have only limited experience with experimental approaches, but we welcome the insights from this broader and interdisciplinary perspective.

Smith suggests that experimental data can help us better understand the causal mechanisms behind typological generalizations, something observational typological studies cannot do. We generally agree that some research setups are more adequate for investigating certain types of questions, and a division of labor, or triangulation, makes sense from this perspective. The difficulty emerges, again, with cases of disagreeing results between experimental and typological studies. Smith provides two very insightful examples of such cases. We will react to the first example, as it concerns a topic that we also explored in previous work, namely the relation between sociolinguistic factors and linguistic complexity (cf. Becker et al. 2023; Guzmán Naranjo et al. 2025). In both cases, we failed to find clear, convincing evidence for sociolinguistic correlates of linguistic complexity. In contrast, Smith (2024) reports on an artificial language learning experiment that supports the presence of mechanisms proposed in the typological literature to account for an association between sociolinguistic factors and linguistic complexity. In such a situation, the important question arises: how can we understand the discrepancy between the results? Smith mentions two hypotheses: (i) the factors identified in the experiments are outweighed by other factors in the wild, and (ii) natural language data cannot show the correlation with sufficient confidence. We agree, and we can think of a number of other potential explanations that can lead to the situation of finding an effect of, e.g., socio-linguistic factors on linguistic complexity in experimental studies but not in typological ones. We think that all these issues should be explored and subsequently discarded in order to understand diverging results:

1. experimental studies:

   1. the experimental design may not be suitable

   2. the experimental study may not reflect natural language learning

   3. the data analysis of the experimental study may have issues

2. typological studies:

   1. the study may not operationalize the actual socio-linguistic hypotheses well

   2. the data collection and annotation may contain too many mistakes

   3. the language sample may be too small to detect the (potentially weak) effects

   4. the language sample may be wrong in just the right way, hiding the effects

   5. the data analysis of the typological study may have issues

These issues all highlight the possibility that either the experimental or typological studies could lead to fundamentally incorrect results. This goes back to our main point: we can only increase our confidence about our findings with more transparency about the work process, with robustness tests and with replication. If at some point we reach high confidence about results from both experimental and typological studies, and these still diverge, we can then start to think about how and why they diverge. Currently, we do not believe that we can have high certainty about our typological results regarding sociolinguistic effects on linguistic complexity to begin with. Therefore, we should be cautious when trying to interpret differences between the typological and experimental results.

Finally, we agree with the conclusion of Smith who suggests to “view typology as part of a joint endeavor to uncover mechanisms shaping languages, where other methods (specifically, controlled experiments) are anyway required to test the critical mechanisms held to be responsible for candidate linguistic universals.”

3.6.1 Robustness across alternative linguistic analyses and annotations

Amongst other things, we argue in our paper that robust typological results should replicate across different datasets, different statistical methods, and different linguistic analyses and annotations. We gave Nichols et al. (2006) as an example for replication across different linguistic analyses. Miestamo & Sinnemäki point out that Nichols et al. (2006) is not a clear example of replication across different analyses, but rather “testing a theoretical claim through the lens of different linguistic phenomena using different types of data”. As another example we can think of the classical topic of word order typology. Traditional approaches to word order universals (e.g. Dryer 1992, 2013a, b, Greenberg 1963, Hawkins 1983) operationalize dominant word order as the most common word order in simple, declarative sentences. Put differently, word order is represented in a categorical way, each language being assigned a single, dominant or basic word order (although some approaches allow for the value “no dominant order”). More recent, corpus-based approaches to word order typology have argued for a gradient representation of word order preferences within languages (e.g. Futrell et al. 2015; Gerdes et al. 2019, 2021; Guzmán Naranjo and Becker 2018; Levshina 2019; Levshina et al. 2023; Östling 2015; Talamo and Verkerk 2022). Yet another recent take on word order typology, namely Grambank (Skirgård et al. 2023), uses a third strategy to operationalize word order preferences. Word orders are represented in a categorical way, e.g. as in Feature GB131 (Is a pragmatically unmarked constituent order verb-initial for transitive clauses?). However, different orders are split up into different features, which allows for single languages to be annotated as having more than one unmarked word order.^[9] If we find that a cross-linguistic word order preference holds across all three types of operationalizations, our certainty about the underlying fact should be much higher than if this preference is only detected using one type of operationalization.

3.6.2 Full data transparency

We appreciate that Miestamo & Sinnemäki highlight in more detail than we did in our study that also qualitative, exploratory typological studies based on a variety or convenience sample should provide a comprehensive list of the languages as an appendix. At the very least, it would be useful to include the language name, its glottocode and the reference(s) used in such a list.

Furthermore, Miestamo & Sinnemäki are absolutely correct in noting that we missed important typological work that includes exhaustive examples for their annotation decisions; we should have been more careful in our phrasing. We thank the authors for bringing to our attention Stassen (1997, 2009), Miestamo (2003, 2005), as well several WALS chapters. We are probably still missing other work that should be mentioned here. Our main argument remains, though, namely that many sample-based typological studies do not provide such an exhaustive list of examples. Doing so may not be feasible in all cases, but we should strive to provide all relevant examples whenever possible to contribute to more transparency in typological work.

3.6.3 Sampling in quantitative typology

Miestamo & Sinnemäki’s main point of criticism concerns the role that sampling should play in (quantitative) typological studies. We admit that we could have presented a clearer and less strong perspective in our study, and we thank the authors for addressing this issue in detail, giving us the opportunity to clarify our position.

Miestamo & Sinnemäki distinguish between probability samples, variety samples and convenience samples, the latter of which effectively corresponds to our proposal of using no manual sampling method. They argue that also in quantitative typology, variety sampling is a more sound approach than convenience sampling, and they show why this is the case discussing a theoretical example: if we systematically leave out phylogenetic or areal groupings of languages, statistical bias control cannot help to make up for missing representations of cross-linguistic variety. We fully agree with this argument and with Miestamo & Sinnemäki’s position, and we should have expressed our position in the paper more clearly and more carefully. We only referred to probability sampling in our study, and did not want to make any statement about variety sampling. We fully agree that variety sampling is advantageous over convenience sampling in quantitative typology, since statistical techniques cannot fix fundamental flaws that occur during the data gathering stage such as missing variety in heavily skewed samples. Miestamo & Sinnemäki make a strong argument for variety sampling when building linguistic datasets, as they have in the past (Miestamo 2005; Miestamo et al. 2016). In fact, in Guzmán Naranjo and Becker (2022: 605) we come to a very similar conclusion: “[…] we also show that strict probability sampling is not required with the statistical controls that we propose, as long as the sample is a variety sample large enough to cover different areas and families.”

3.6.4 Modeling decisions

Miestamo & Sinnemäki address two issues regarding our model: the tree used for the phylogenetic term and the underlying assumptions of how space affects contact. We already discussed the latter comment in Section 2.6 and focus on the first comment here. The authors are correct in pointing out that including a phylogenetic tree in the model comes with additional conceptual and technical questions, and we cannot address them fully in this reply. What we will say is the following. The implications of using different trees in quantitative modeling in typology are not yet well understood. There are several ways of inducing tree branch lengths (e.g. Bouckaert et al. 2012; Guzmán Naranjo and Becker 2022; Wichmann and Rama 2021), but we do not yet know how much such differences in tree structures affect model results. Preliminary tests (not yet published) suggest that in practice, the impact is minimal to negligible because phylogenetic terms are flexible enough that they can accommodate variations in relative branch lengths. Of course, this should ideally be tested properly with data across different types of linguistic phenomena. Our second answer is that whether we use branch lengths in thousands of years (based on some sort of imputation) or relative branch lengths in terms of number of shared nodes (what we do) cannot have an impact, provided that the relative relations are the same. What the model uses is the covariance between leaves. This covariance is calculated as Brownian motion based on the distance matrix.

The choice of the phylogenetic tree has no consequence for the interpretation of the results as synchronic or diachronic. The phylogenetic term is inherently diachronic, just as the Gaussian process term built on geographic distance is inherently spatial. The phylogenetic process captures the feature evolution along the tree. The intercepts of the model are then equivalent to a sort of steady state that expresses universal tendencies.

To conclude, these are valid concerns and questions, and we are currently preparing a study that goes into more detail regarding the conceptual basis for different technical ways of implementing phylogeny in quantitative typology.

3.7.1 Weak versus strong theories

Tallman proposes to focus more on the relationship between the linguistic theories and our statistical models. Following Fried (2020), he calls for a distinction between strong and weak theories. Strong theories are explicit about underlying assumptions and the causal relations to be tested, while weak theories are not. Tallman relates this distinction to the broader topic of replicability and our study by arguing that “the effects that lack robustness might relate as much to the fact that B&GN are attempting to assess weak theories.” We agree that this is a very important issue not only regarding the replicability of (typological) studies but for a transparent scientific workflow in our studies more generally.

In our study, we wanted to highlight how complex the modeling decisions are when analyzing typological data and how much the results can depend on these decisions even when the data used is identical. We fully agree, though, that this only addresses one puzzle piece of robustness and replicability in typology, and that the question of how strong the theoretical underpinnings of our statistical analysis really are is in fact an equally if not more important question. We are therefore thankful that this aspect was brought up by Tallman, and we fully agree with his conclusion that statistical modeling can help us to formulate stronger hypotheses by being explicit about our assumptions and by making them testable.

3.7.2 Implicit assumptions

Related to the previous issue of strong versus weak theories that build the conceptual foundation for the statistical analysis, Tallman discusses implicit assumptions in Seržant (2021) and Shcherbakova et al. (2023), as well as in our robustness tests.

Regarding Seržant (2021), we do not fully agree. Tallman claims that Seržant has a (latent) theory about how the particular contact situation of Slavic languages has led to its skewed positioning in an East-West cline of person-number index decay, and that we could not have tested this theory properly with the type of data used. The goal of our robustness test was different, though. We primarily wanted to test the hypothesis about the East-West cline, i.e. whether we also find evidence for this cline when using statistical modeling. We tested for the East-West cline by explicitly comparing a model with a contact component versus a model with a contact and a cline component. It is a secondary question how such a cline could have originated, and what the concrete mechanisms were, that resulted in Slavic showing less paradigm decay than Germanic or Romance, and we agree with Tallman that this is something that we could not test with the type of data at hand.

As for Shcherbakova et al. (2023) and our robustness analysis, we fully agree with Tallman that both studies fell short on spelling out “explicit reference to causal theories about the diachronic relationship between grammatical coding” and that “the analysis could (or should) be regarded as mostly exploratory.” Our operationalization of the replication tried to follow, as closely as we could, the hypothesis that the original study aimed to test, namely “[t]o investigate correlations between the amounts of coding on nominal words and verbs” and to “test whether a change in one domain implies a change in another domain” (Shcherbakova et al. 2023: 157). We agree that a stronger hypothesis or theory could be linguistically more informative and should take into account more complex relations between the linguistic features analyzed. This would also result in a more complex model structure that our setup could, in principle, be extended to.

3.8.1 Better and poorer models

Hammarström correctly points out that there are two perspectives to our study: (i) robustness testing, which only makes sense if multiple methods are equally suitable, and (ii) trying to understand which models are better. We agree that we know for some techniques that they are better or poorer for modeling typological questions, e.g. we know that multiple regression is fundamentally better than fitting many disjoint regression models. However, we do not think that we have a complete understanding of which methods are objectively better in all situations, also because it depends on the phenomenon and data at hand (as Hammarström also notes). To give a concrete example, we have argued before that phylogenetic effects and Gaussian processes are better than having effects by family and macro area (Guzmán Naranjo and Becker 2022). However, these results are only partial. In ongoing work, we find that phylogenetic regression can be suboptimal and perform considerably worse than family effects under certain conditions. This happens, for example, when a sample contains many isolates and small families. Similarly, while Gaussian processes are incredibly powerful, approximate Gaussian processes or splines can be more appropriate for large datasets because sampling exact Gaussian processes becomes exceedingly difficult with more than 1,000 observations.

For these reasons, in practice, we take (and need to take) both perspectives at the same time, i.e. checking for methodological robustness but also trying to understand which methods may be better for a given study. If the results are robust across methods, we learn that both methods are likely to work comparably well for the given data and question. If the results are not robust across methods, then we need to try to understand where the differences come from (as was also noted by Dryer). This can help to learn which method may be better for capturing a given linguistic phenomenon.

3.8.2 Standards for replicability

Hammarström claims that our suggestions for more transparency and better replicability are not ideal because we call for full code sharing. His argument is that code is neither sufficient nor needed for replication, and that in other disciplines like computer science, the standard is to share pseudocode specifications of the algorithms used. Pseudocode, as the name suggests, is a description of the steps in an algorithm without being executable code as such. There is no single convention for pseudocode, but the idea is that it is easily understandable without additional explanation for scientists who are familiar with programming, while being independent of a particular programming language. This may seem like a useful alternative to sharing code in a particular language, likely making it more sustainable in the long run (e.g. stable over decades and hundreds of years). Despite these potential advantages, we think there are several strong arguments against using pseudocode in linguistics.^[10] Converting math and pseudocode into working code is not straightforward, as there is no single standard that can quickly be adopted, and it is yet another layer of the quantitative workflow that linguists would need to familiarize themselves with. It is also an additional step where mistakes can happen that are likely to go undetected by the original author(s) because pseudocode cannot be executed and checked (Lamport 2009: 25). The fact that it cannot be checked or run is equally an issue for review, replicability/reproducibility and re-useability. Linguists may want to adopt methods from a previous study; they would then have to convert pseudocode back into actual code, which would likely lead to questions and additional choices that could be avoided without the conversion of code into pseudocode in the first place. Also, sharing pseudocode instead of the code used seems to add another skill to the (already long) list of skills needed to do quantitative typology.

There is an argument to be made about sharing model specification in statistical notation (we have done this in the past when we introduced new models, e.g. Guzmán Naranjo and Mertner 2023). While this is certainly important from a theoretical perspective, it may not be very useful for many linguists who would need to re-implement it to use the same model. Code, on the other hand, can be re-used and adapted easily for new projects.

If the concern is about whether code will run in the future, there are some solutions for that. The most robust one, but also the most costly in terms of time and effort, is to use docker (Merkel 2014) or similar containers.^[11] Containers guarantee that the environment, from the operating system to the linear algebra libraries and to packages are the same every time. A less robust but still very robust alternative is to use a versioning software like renv or conda.^[12] Conda environments are the standard in many fields. We provide both renv and conda environments for our code. The least robust but still useful solution is to simply share the session information (sessionInfo() in R or session_info.show() in python). For these various reasons, we disagree with Hammarström that pseudocode is necessary and sufficient, and argue that sharing the actual code is more helpful and transparent, at least for the foreseeable future in quantitative typology.

3.8.3 Some details on our models and method

Hammarström makes two statements about our model and methods that are not entirely correct and that we want to clarify here. He writes: “However, as employed in the replication study, the same hyperparameters apply across the entire sample, which means that the influence decays the same way per kilometer (or, actually, per lat/lon degree) over the whole world.” This is not completely correct, at least not for all studies. We use grouped Gaussian processes by macro-area for the robustness test of Dryer (2018), which means that the parameter is shared by all languages in the same macro-area, but it allows for variation across areas. In our experience, however, this does not matter much in practice in most situations because Gaussian processes are very flexible and can adapt well to different areas even if one builds one single Gaussian process. Having multiple Gaussian processes is mostly useful for performance reasons. It is also relatively straightforward to build more complex model structures with several Gaussian processes that separate long-distance and short-distance contact.

The other point brought up by Hammarström is that we do not use AIC. This is a minor but important detail: AIC as well as Bayes factor have been shown to be suboptimal methods for comparison and evaluation of Bayesian models (and likely non-Bayesian as well). We provide full model comparisons using ELPD under cross-validation. According to Piironen and Vehtari (2017a), this is currently the standard for Bayesian statistics.

3.10.1 Model assumptions and model validation

Coupé discusses two important points in his commentary regarding model sanity checks, namely if the model assumptions are met and if the model does what we want it to do. The first one relates to something we did not mention explicitly, but which is always a crucial aspect of any quantitative analysis, namely that model likelihood choice must be justified by the data, and that models can give us incorrect answers if their assumptions are not met. There are many more ways in which model assumptions could be violated, leading to incorrect results. We fully agree with Coupé, who states: “It is not common at all to see authors report that they have verified that the assumptions of their model(s) were met. This should become a standard requirement, as one among different sanity checks.”

Coupé also suggests that researchers should present model diagnostics. We agree, with the caveat that there is no consensus yet as to what those diagnostics should be. Q-Q plots, as shown in Coupé’s commentary, appear to be rarely used in Bayesian statistics, with posterior predictive checks being seen as more insightful. Following mostly Vehtari et al. (2017), we have argued that cross-validation results are essential in evaluating model performance, and this is what we have used in this study as well as in other previous work. The practice of using cross-validation, however, seems to remain unpopular in linguistics, and not all models are easy to cross-validate (e.g. BayesTraits).

3.10.2 Better alternatives

Another point made by Coupé is that there might be better alternatives to phylogenetic regression and to the exponential quadratic kernel we used for the Gaussian process. This comment directly ties to our previous discussion on the complications of modeling space in Section 2.6, but also to Cathcart’s commentary on different models of feature evolution. Regarding kernel choices, we actually believe that Matérn family of kernels are a better choice.^[13] We have also explored non-stationary Gaussian processes to model language expansion, with mixed results (Guzmán Naranjo et al. 2025; Urban and Guzmán Naranjo 2025).

As for phylogenetic effects, we strongly agree that they make simplified assumptions about feature evolution, which might not match very well how languages evolve. We are currently working on testing some alternatives. At the same time, we remain slightly skeptical of the following, specific comment by Coupé: “[…] the phylogenetic component in B&GN’s models assumes that the rate of linguistic evolution is always the same along the different branches of the language tree. It is possible to relax this constraint/assumption (Davies et al. 2019), and therefore build a model that better matches the history of languages.” We see two issues with this statement, one conceptual and one practical. The conceptual issue is that we do not know, for the most part, what a correct model, matching the history of language, would or should look like. We would be pleased to include more complex phylogenetic structures in our models, but we need to be cautious about those models not only being more complex but potentially also introducing additional, wrong assumptions. In our models, we could implement different evolution rates by using independent phylogenetic terms for each family. This is straightforward from a modeling perspective, but the question is whether it makes sense from a data perspective, depending on how much we know about each of the language families included in the sample.

From a practical modeling perspective, we already find that phylogenetic regression can be too sensitive and too flexible in some cases. Prior choice has a considerable impact on the estimates (this actually relates to Coupé’s comment on the number of parameters vs. the number of data points), and wide priors lead to very poor performance under cross-validation. We also already mentioned that phylogenetic regression can fail, namely if the language sample contains many isolates or a high number of very small families with two to three languages. We are, of course, happy to be proven wrong on both points.

3.12.1 Explaining concrete decay factors

One of the main points mentioned by Seržant is that the model in our study does not explain the specific decay factors of Slavic languages. We agree to some extent. We do not have a complete, linguistic explanation of why the decay factors for Slavic are exactly what they are. Rather, our argument is that based on the data as presented by Seržant (2021) for the quantitative analysis, we cannot conclude that the specific values of Slavic are a result of contact with Turkic and Uralic, and that we do not find strong support for an East-West cline when adding phylogenetic and spatial components to our model. Seržant (2021) and Seržant’s commentary offer explanations for the decay factors of Slavic languages based on additional knowledge and evidence about the particular contact situation of Slavic languages in the past. This is very insightful from a theoretical perspective, but our methodological point is that we do not find evidence for the cline and the need for contact to account for the position of Slavic to begin with when analyzing the data.

3.12.2 Distinguishing between pressures and signals

Seržant makes a useful distinction between areal and phylogenetic pressures as mechanisms in the real world shaping language structures, and signals as found in statistical models. He points out that it is not always clear whether the relevant signals identified through modeling can necessarily be taken as evidence for the respective pressures to be relevant in the real world. Related to that, Seržant asks: “should a strong genealogical signal always be taken literally to mean that inheritance alone is sufficient, i.e., that there would be something specific to a family?” This is an excellent question and, unfortunately, a very difficult one to resolve. In models like the one we use, when there is complete overlap in the variance explained by two predictors, it is fundamentally impossible to decide categorically whether both predictors play a role in the real world or not. In this case, our result shows that a model which only includes the phylogenetic structure performs equally well as the model with a phylogenetic structure and areal structure. Does this mean we can discard any type of areal pattern? Not really, but it does mean that our certainty about the areal dynamics needs to be set much lower than if we found that both predictors together perform better than either predictor on its own.

3.12.3 Genealogical pressure and innovations

Related to the last point is the question of how we should interpret a phylogenetic bias in regard to decay factors. Seržant argues that “genealogical pressure is logically incompatible with innovations.” Addressing this question properly would require a paper of its own, but we think that Seržant’s claim is potentially too strong. We can imagine that genealogy could bias a family branch towards innovation if a common change at some point in the past caused the system to be in an unstable or communicatively less-preferred state. Then, the languages belonging to that branch would have a greater than chance probability to undergo a change, leading to an innovation compared to the proto-language. The changes that individual languages would undergo could be different, but they could also be similar if there are universal pressures towards certain structures. In our opinion, it would therefore be possible that, at some point in the past, Romance and Germanic languages innovated some unstable configuration that resulted in more decay in their paradigms than what we observe in Slavic.

3.13.1 Inter-correlated evolution

One of the technical points mentioned by Shcherbakova et al. is that their model captures correlated evolution of two features, while ours “model[s] correlation between the variables that ignores the interdependence between the changes in these variables throughout time” and “[i]s a synchronic correlational analysis”. We do not think that this opposition between the two approaches is entirely accurate because our phylogenetic term also captures correlated feature evolution, and we do not completely agree with the characterization of our approach to be entirely synchronic. Our model explicitly samples two phylogenetic terms (which capture diachronic evolution based on a specific assumption of diachronic development), but samples them as correlated, with an explicit correlation parameter in the model. The correlation is not an a posteriori metric, and it is not dependent on how well the model can predict the dependent variables. We admit that we could have made this point more explicit in our study.

One can, of course, argue that phylogenetic terms are not a good representation of linguistic evolution (see Cathcart’s commentary for some discussion), or that BayesTraits is fundamentally better. However, as far as we are aware, these are currently unknowns. We have no solid data about which types of models work best with what types of situations. The fact that BayesTraits or our model yield some result does not mean that the result is correct. We did try to evaluate our model against synthetic data for which we knew the real parameter values, and we could recover the parameter values very well. In their commentary, Shcherbakova et al. argue that our simulated data is not a good representation of real linguistic data. They may be correct, but this is yet another open question that needs to be tested empirically. One of the points of our paper was precisely to show how much uncertainty there currently is regarding methods in quantitative typology including computational models, and how this calls for more caution with any single result.

3.13.2 Contact versus no contact

Shcherbakova et al. note that one of the differences between their and our approach is that we include spatial dependencies as a proxy for contact between the languages in the model. The authors argue that this “further widens the divide between the two studies’ theoretical assumptions” and makes it less of a replication study. As we mentioned in Section 2.5, we gladly follow Shcherbakova et al.’s suggestion to classify our study as a robustness analysis. As such, we do not understand why it is an issue that we include spatial effects. We think that it should be desirable in this case to include some form of control for contact in our models, whether evolutionary or regression based, as we can assume that contact between languages will have played a role in at least some of the morphosyntactic changes analyzed. If a model does not include any control for potential contact effects, then we cannot possibly know whether the results really represent language-internal evolutionary dynamics alone, or whether contact dynamics could have also impacted the changes. In other words, we need to control for confounding factors.^[14]

3.13.3 Multicollinearity

Shcherbakova et al. are concerned about multicollinearity between our predictors, as we predict the outcome of the 13 feature groups simultaneously instead of making pair-wise comparisons like Shcherbakova et al. (2023). Multicollinearity is an issue which appears in a (generalized [additive]) linear model of the form y ∼ F(α + β ₁ x ₁ + β ₂ x ₂, … ) where F is some likelihood function, and x ₂ and x ₂ are correlated predictors. This leads to a situation where estimating β ₁ and β ₂ becomes difficult because there is a wide range of values which work equally well, and thus we cannot reach a good estimate for the real values of the coefficients. There are two points to our reply regarding this concern. Our first point is that under a Bayesian framework, multicollinearity – if it does exist – can be mitigated considerably and even overcome by careful prior choice (Al-Essa et al. 2024). The reason is that if we provide the model with a good guess about likely values of β ₁ and β ₂, then the model has an easier time finding reasonable estimates for both. Of course, there could still be issues of high uncertainty, but it is a much less pronounced problem than with maximum likelihood estimation. Other alternatives like Dirichlet processes or Horseshoe priors can also be used in the presence of multicollinearity (cf. Piironen and Vehtari 2017b).

Second, and more importantly regarding our study, there is no such risk in our model setup because we do not use the individual variables as predictors but as dependent variables in the model. Hence, there is no question of collinearity. The correlation comes from the fact that we sample the different phylogenetic components from a matrix normal distribution, which means that we sample them as correlated (there is an explicit correlation coefficient in the model).

3.13.4 Interpretation of the results

In their commentary, Shcherbakova et al. also discuss differences between our and their interpretation of how results compare. The authors state that they do not think that the results of Shcherbakova et al. (2023) and those of our study are as different as we make them out to be. They note that they find a null result and we find a positive correlation between nominal and verbal complexity, which are both in agreement that there is no trade-off (which would imply a negative correlation). We do not think that we actually disagree that much, as we wrote the following: “While we could not fully replicate Shcherbakova et al.’s results, we did not find entirely opposite trends and patterns either” (Becker and Guzmán Naranjo 2025). At the same time, the models produce different effect sizes and effect directions (no effect vs. positive correlation) in some cases, which is what we highlighted as interesting. We think that it is worth trying to understand where these differences come from, but we do not have a good answer in this case. Perhaps Tallman is correct in pointing out that a stronger theory, i.e. a less exploratory conceptual setup could help in designing more specific and less complex statistical models whose results and comparisons could be easier to interpret.