Replication, robustness and the angst of false positives: a timely target article and its multifaceted comments

As scientists, we^[1] love what we are doing and we hope that our findings will outlive us, and we dread being shown wrong (probably for many of us, nightmares of discovering we’re naked in public are far outranked by being shown wrong in public). This is a foundational paradox, as any first year undergrad will breathlessly and smugly tell you that all theories are false and that what sets science apart from everything else is its falsifiability (Godfrey-Smith 2021; Newton-Smith 2001; Psillos and Curd 2008). Which means, by extension, that everything we do is ultimately wrong.

But surely there are degrees of wrongness: how many of us would not wish to be wrong the way Newton was, given that most of what we do, build, and even launch in space is still based on his theory, spectacularly falsified by a certain Albert more than a century ago. Or the way Darwin got it wrong with inheritance. Some kinds of wrongness are to be desired at all costs, as they make us advance, question fundamental assumptions, and find new, previously invisible paths. Other kinds of wrongness, such as fundamental overlooked problems in data analysis, are deeply dreaded. Another foundational paradox is the following: in general we dread being wrong but there can be no advancement without being wrong: discovery is, despite what various voices keep claiming, messy, costly, wasteful and resists “optimisation”, “rationing” and “ideology”. Discovery is intrinsically built on getting it wrong again, and again, and again.

Simplifying in the extreme, we can get it wrong in two ways (Peterson 2009): claim that something is when it ain’t (a so-called false positive or type I error) or the other way around, that there’s nothing interesting to see when, in fact, there is (false negative or type II error). For lots of reasons, it is the false positives that seem to keep most people awake at night, and the fear of those is inculcated in every student that ever sat in a statistics class where hypothesis testing and p-values were repeatedly drilled into them.^[2] And this is why a study needs to be repeated as well, as there are lots of ways null effects can still masquerade as “significant” findings^[3] – and this is precisely where our target article, Replication and methodological robustness in quantitative typology,^[4] comes in.

We will not summarise it, nor the many very relevant comments it has generated, as we would rather leave the pleasure of drawing their own conclusions to our diverse readership, but instead, briefly discuss why we decided to host this debate in Linguistic Typology. First, our endeavour is eminently empirical and interested in the real, messy, complex, always surprising world of language and languages, which means that most of our claims are potentially false, and not in the ways of Newton or Darwin, but in the much more boring and common ways of false claims that have generated the various “replicability crises” (Ioannidis 2005; Vasishth et al. 2018). We are unwilling to add to those crises a “typological” one, but, given the traditional small samples, lack of methodological agreement and tendency to draw grand conclusions, we suspect that there might, indeed, be one, despite the increased attention given to these issues over the years, in particular, in Linguistic Typology (e.g., Jaeger et al. 2011; Janssen et al. 2007; Editorial Board 2016). Second, our field is fast becoming heavily quantitative, methodologically very sophisticated, and with access to large databases, which makes it ready to embrace more formal ways of buttressing its claims. Third, whether we like it or not, our field is important, in that the patterns of linguistic diversity, their causes and effects affect many human enterprises, generate genuine interest and even have actual consequences in the real world of money, power and justice, forcing us to be extra careful with our claims. Finally, the submitted paper was, frankly, very good and inciting, irrespective of whether one agrees or not with its claims, and obviously, in need of further replication.

Before ending, we want to briefly return to the false claims nobody seems to care about: the negative ones. They matter, too, and guide research along with positive ones, although they may not be as visible in practice. Overlooking an intriguing fact because it did not reach “significance”, passing by a tiny, overgrown path in the forest, or ignoring a furtive look in the subway, could be missed opportunities to cure cancer, miss a breathtaking view, or fail to find the love of your life.^[5] More to the point: because of the perverse way the world works, we cannot simultaneously decrease the chances of a false negative and of a false positive (Kim 2015; Nakagawa et al. 2024): the more afraid we are of making false discoveries, the more we will miss genuine ones (Peterson 2009). It’s a choice we must make as a scientific field and as a society. What hurts more (and when): misleading and directing scarce resources into avenues that don’t exist, or trudging on well-worn motorways and missing the little diverging paths? Given the inevitability of having to make this choice over and over again under complex, explicit and implicit, conscious and unconscious, economic, ethical, scientific, ideological, etc., etc., etc. constraints, we are looking forward to how our scientific field develops amidst this tug of war.