The wompom

2.1 Resources and computational infrastructure

Looking back over the past decades, there have been enormous developments in computational infrastructure and the written and spoken resources that have become available. In the eighties, a megabyte was an enormous amount of memory. When I was working on my PhD thesis in the eighties, I was advised to travel to Nijmegen, where the celex lexical database was being developed, in order to extract morphologically analysed word frequency lists from this database. At that time, I was investigating whether the Good-Turing probability for unseen types (Good 1953) would be a reasonable statistic for evaluating morphological productivity. Since this probability is estimated by the ratio of hapax legomena and the corpus size, I was particularly interested in the words with frequency 1 in celex, which typically constitute roughly half of the types. To my disappointment, it turned out that celex only included a handful of hapax legomena. One of the reasons turned out to be that adding hapax legomena to the database would have led to a doubling of the size of the database, and to doubling the size of external memory required. As in the early eighties a hard disk storing 1 megabyte had a price tag of around 1 million US$, adding hapax legomena to the database was prohibitively expensive. Fortunately, the costs of storing devices have plummeted while the amounts of data that can be stored have increased dramatically. In parallel, the resources available for corpus linguistic research have also increased exponentially.

Given the abundant resources we now have, the work done by early pioneers of corpus linguistics such as Zipf, Yule, and Ellegåard, which was done by hand (Ellegård 1953; Yule 1944; Zipf 1949), is all the more impressive. My earliest experiences with corpus linguistics involved the study of aspect in Biblical Hebrew (Baayen 1997). The instrument that I used to collect the data for this study was a keyword in context concordance in book format (Mandelkern 1896), a monument of Jewish corpus linguistics long before the advent of computers. (Now, I would use a different tool, the computerized and meticulously linguistically annotated electronic version of the Hebrew Bible https://shebanq.ancient-data.org/hebrew/text.) When I started working on my PhD thesis in the mid-eighties, my initial insights were gleaned from a perusal of a book with word frequencies compiled by computer (probably using punch cards) from a Dutch corpus of some 600,000 words (Uit den Boogaart 1975). This corpus was modeled after the Brown corpus, which had 1 million words, and for which a book with word frequencies had been published almost a decade earlier (Kučera and Francis 1967). Fortunately, the frequency tables of Uit den Boogaart (1975) included information on all the hapax legomena. Soon after, I found myself learning UNIX, programming in C, and scripting in awk, on a small VAX computer that was in constant use by some 15 people simultaneously. But this computer gave me access to the online version of the Dutch corpus.

Since then, developments have been rapid. In the nineties, the British National Corpus (Burnard 1995) was created, comprising 100 million words. A decade later, the WaCky corpora (Baroni et al. 2009) extended the size of written corpora to more than 1 billion words. The Common Crawl February/March 2024 archive brings together no less than 3.16 billion web pages.^[2]

Over the years, resources for spoken language have also been increasing rapidly. The switchboard telephone conversation corpus comprises some 260 h of recorded and transcribed speech (Godfrey et al. 1992). The Buckeye corpus (Pitt et al. 2005) comprises 40 one-hour face-to-face interviews with a stratified sample of speakers from Columbus, Ohio, with audio, orthographic transcriptions, and broad phonetic transcriptions. Inspired by the British National Corpus, a corpus of spoken Dutch was constructed and completed in 2004, with some 900 h of speech, transcribed at various levels http://hdl.handle.net/10032/tm-a2-k6 (Oostdijk et al. 2002). A much smaller corpus of spontaneous speech in German was created by Arnold and Tomaschek (2016), but this corpus provides, for 6.5 h of speech from 13 speakers, not only the audio and transcriptions, but also recordings of tongue movements traced with electromagnetic articulography. A truly vast multimedia corpus is being compiled by the Little Red Hen Consortium (https://www.redhenlab.org/home), which provides videos, audio, and aligned transcription for TV news broadcasts, mainly from the USA, but with increasing contributions from other languages as well. The NewsScape infrastructure developed by Uhrig (2018) provides access to some 350,000 h of broadcasts, comprising some 2 billion words. Pipelines are currently in development for adding detailed information of movements of the head, torso, arms, and hands in the videos, enabling large scale study of co-speech gesture, and the consequences of gesture for the realization of speech in natural settings (see, e.g., Alcaraz Carrión et al. 2020).

2.2 Methods

The development of tools for the analysis of corpora has also been impressive. With respect to specifically statistical methods, the logistic regression modeling framework for variable rule analysis introduced in the early seventies (see, e.g., Cedergren and Sankoff 1974) gave sociolinguistics an impressive head start compared to other subdisciplines of linguistics. Extensions within the general approach of linear mixed models (Pinheiro and Bates 2000) and the generalized additive (mixed) model (GAM, Wood 2017) provide the analyst not only with tools for studying count data, but also with tools for evaluating measurements made on corpora, such as the durations in speech of segments or words (see, e.g., Gahl 2008; Plag et al. 2017). GAMs provide an especially attractive analytical tool for the analysis of non-linear trends in time, or non-linear properties of linguistic units in spoken corpora, such as F ₀ contours for stress (Koesling et al. 2012) and tone (Chuang et al. 2021, 2024).

Multivariate methods such as principal components analysis, multidimensional scaling, linear discriminant analysis, and more recently t-distributed stochastic neighbor embedding (tSNE, Maaten and Hinton 2008) have become part of the corpus linguist’s toolkit. My first encounter with this group of methods goes back to an ACH-ALLC conference at which John Burrows presented his stylometric analyses of authorial hands and language change (Burrows 1986, 1992, 1993). As a young post-doc, I was stunned by the analytical power of the relative frequencies of the most frequent function words (just 40 function words will already get you a long way), when analysed using principal components analysis.

About a decade later, researchers started moving away from monocausal explanations to multifactorial explanations. In cognitive linguistics and corpus linguistics, Stefan Gries and Dagmar Divjak developed behavioral profiling (Arppe 2008; Arppe and Järvikivi 2007; Divjak 2004; Divjak and Gries 2006; Gries 2000; Gries 2006; Gries 2010; Gries and Divjak 2009; Levshina 2022). But also in subsequent more formal approaches to language spearheaded by Joan Bresnan (Bresnan 2006; Bresnan et al. 2007), multifactorial explanations gained traction. Across sociolinguistics, cognitive linguistics, and corpus linguistics, a variety of methods for count data is now available to predict which of a possible set of alternative language variants is most likely to be used in a given context. These methods range from statistical methods such as mixed effects logistic regression and support vector machines to methods from machine learning such as random forests, gradient boosting machines, and memory-based learning (Daelemans and Van den Bosch 2005).

At the time that logistic models and variable rule analysis were making their way into linguistics, psychologists were studying learning rules for very simple neural networks (Rescorla and Wagner 1972). The Parallel Distributed Processing books (McClelland and Rumelhart 1986; Rumelhart and McClelland 1986) were published at the same time that the idea of network models was introduced to morphology by Bybee (1985, 1988. In the eighties, seminal algorithms and network architectures such as back-propagation Rumelhart, Hinton, and Williams (1986) and recurrent networks (Jordan 1986) were already on the table. Recurrent networks were developed further by Elman (1990), and not much later enriched with other kinds of neurons (Hochreiter and Schmidhuber 1997) that made these networks even more powerful. More recent transformer technology (Vaswani et al. 2017) has been applied to huge datasets such as the Common Crawl, resulting in large language models (LLM) such as the current GPT series initiated by Open-AI (Radford et al. 2018). There is hardly a natural language processing task left that an LLM, supplemented with some fine-tuning, cannot do. According to Binz and Schulz (2023a, 2023b), with some fine-tuning pre-trained LLMs can also be turned into generalist cognitive models. It seems that there is nothing that an LLM cannot be made to do.

2.3 Linguistic theory

Looking back at the development of linguistic theory over the last decades, one constant, across widely different linguistic theories, has been that explanations have been relying on symbolic descriptors that partition complex continuous and multifaceted phenomena into discrete subsets with superordinate labels such as phoneme, morpheme, word category, aspect, tense, and voice. According to early proposals, both in post-Bloomfieldian American structuralism and in generative grammar, these symbolic descriptors are part of a calculus that governs how symbolic units are arranged (Blevins 2016). However, over time, it was recognized that the constructions in a language provide islands of systematicity in a variegated whole (Croft 2001; Goldberg 2005). Monofactorial explanations were replaced by multifactorial explanations, and researchers in NLP started replacing deterministic grammars by probabilistic grammars.

Making sense of language as a variegated, probabilistic system that is subject to many simultaneous pressures and constraints is not straightforward. It is all fine to have models predicting the probabilities of alternative realizations of a given feature. Social stratification of language features can be dealt with relatively straightforwardly by positing different probabilistic grammars for dialects, sociolects and idiolects. But variability within one language variety remains enigmatic. If a speaker selects a less-probable variant, is this the inevitable consequence of the randomness in the firing patterns of biological neurons? Or, more along the lines of current next-word prediction based large language models, is every choice made ultimately predictable from a deterministic chain of previous experiences?

Exemplar models and analogical models provide a different kind of explanation. These models propose that generalization and selection in language is driven by analogical reasoning over exemplars stored in memory. Examplars can range from structured chunks of words (Bod 1998, 2006; Borensztajn et al. 2009) to structured sequences of phones (Daelemans et al. 1995; Skousen 1989). Analogical reasoning based on the most similar structured exemplars in memory then accounts for generalization. As more frequent phenomena will have more exemplars in memory, effects of frequency and probability are straightforwardly explained by selecting that class that is supported by most exemplars. The changing rates at which phenomena are encountered in individual language use, and hence the changing composition of analogical sets of exemplars, then explain within-individual fluctuations in choice behavior.

But exemplar models face two challenges. One challenge is that feature engineering is required. Some metric is required on the basis of which it can be decided how similar exemplars are. Often, exemplars are described by means of arrays of discrete feature-value pairs. This may not be optimal, a point to which I return below.^[3]

The other challenge is the overwhelming numbers of exemplars. The challenges that come with storing and evaluating huge numbers of exemplars is especially prominent for spoken language, and becomes visible already at the word level. Although an exemplar model for auditory word comprehension was developed in the late nineties (Johnson 1997), network-based models for auditory comprehension were considered already at the same time (Gaskell and Marslen-Wilson 1998), and have been explored subsequently from many different theoretical perspectives (Magnuson et al. 2020; Scharenborg 2008; Shafaei-Bajestan et al. 2023). Simple network models operating on discrete featural representations, using the learning rule of Rescorla and Wagner (1972) are also available for exploring both lexical processing (Baayen et al. 2011; Milin et al. 2016) as well as higher levels of language structure (see, e.g., Romain et al. 2022).

For many years, I believed that network models avoid the problems faced by exemplar models. But with the advent of large language models, I’m not so sure anymore. These models work with billions of parameters (175 billion for GPT-3, more than a trillion for more recent AI systems). Some engineers now describe language modelling as a form of compression (Delétang et al. 2023). From their perspective, LLMs are compressed re-representations of the data they were trained on. Importantly, compression results in a memory that is productive and that makes accurate predictions for unseen data. From this perspective, Chat-GPT3 is a corpus that talks back.

But this suggests to me that there is considerable continuity with exemplar-based models. Daelemans et al. (1999) observed, using memory-based learning, that forgetting is harmful: performance drops when low-frequency words are not taken into consideration. Furthermore, in order to optimize algorithms for efficient exemplar-driven generalization such as TiMBL (Daelemans et al. 2007), they also found that it is essential to make use of data compression.

The analogical sets of nearest neighbors on which the models of Skousen (1989) and Daelemans et al. (1995) base generalization offer the analyst considerable insight. Analogical modeling has successfully and insightfully been applied to a range of phonomena, ranging from the study of stress (Arndt-Lappe 2011; Eddington 2000) and final devoicing (Ernestus and Baayen 2003) to the study of derivational semantics (Plag et al. 2023).

By contrast, a current LLM is more like a black box. In fact, the LLMs put the connectionist debate that raged for two decades in a new light. This debate came to an inconclusive end in 2002 with a series of exchanges in Trends in Cognitive Sciences (McClelland and Patterson 2002a, 2002b, 2002c; Pinker and Ullman 2002). This debate left many researchers with the feeling that connectionist networks are just not good enough. However, history has proven the connectionists right. Neural network models can be made to work, and a well-designed connectionist memory is productive. Unfortunately, LLMs are not straightforwardly interpretable. In this respect, history has proven the connectionists wrong. For instance, Seidenberg and Gonnerman (2000) viewed the hidden layer of their networks as a layer of morphological representation, where orthographical, semantic, and phonological systematicities (in their terminology, ‘codes’) converge. According to this proposal, just as there are distributed representations of form and meaning, there is a distributed representation of morphology which captures regularities attributed to morphemes without having to represent morphemes with discrete units. However, the many layers of units and the internals of attention heads in current transformer-based LLMs are opaque to any such interpretation.

3.1 Oak leaves

In standard approaches in theoretical linguistics to the formation of noun plurals in English, an operation on form (the addition of a dental fricative with phonological adjustments) has a parallel operation on meaning: the change from singular number to plural number. Typically, the semantics of pluralization is described as involving the change in value of an abstract feature for number, either by adding a feature specification for plurality, or replacing a singular feature value by a plural feature value. Although the semantics of pluralization in English is well known to be more interesting than this (see, e.g., Quirk et al. 1985), in distributional semantics, pluralization has been conceptualized as a unitary semantic operation. For instance, taking pluralization to involve an equipollent opposition, we can set up proportional analogies such as

and rewrite this in vector notation as follows:

We next express the plurals as a function of their singulars,

and it now follows that

Here, I specified separate plural vectors for apple and cow, one reason being that word embeddings inevitably contain measurement noise. In what follows, I will refer to the vectors PLURAL → apple and PLURAL → cow as the shift vectors for apple and cow. Given the measurement noise in the embeddings, we average over all shift vectors to obtain an average shift vector calculated over all n pairs of singulars and their corresponding plurals in a given dataset (see, e.g., Drozd et al. 2016):

However, as pointed out by Shafaei-Bajestan et al. (2022, 2024, upon closer inspection, the empirical shift vector for English is not very informative. To see this, consider the left panel of Figure 1, which plots the shift vectors (calculated from word2vec embeddings, Mikolov et al. 2013) for 11,665 English nouns in a two-dimensional plane, obtained by applying a tSNE dimension reduction (Maaten and Hinton 2008). The average shift vector (not shown) is located near the origin of the scatterplot. Shift vectors show considerable clustering by high-level WordNet-based semantic class, four of which are highlighted (artifacts in gray, animals in brown, plants in green, and group nouns in black). A straightforward linear discriminant analysis with leave-one-out cross-validation is able to predict the high-level semantic class (N = 26) for a shift vector with 63 % accuracy (majority baseline: 23 %).

Figure 1:

Shift vectors for 11,665 English nouns, using word2vec embeddings. Left panel: selected high-level WordNet categories. Right panel: WordNet-based subcategories of the higher-level category of artifacts. Individual shift vectors are represented by tiny dots, shift vectors of selected categories are represented by larger symbols.

The right panel of Figure 1 shows that within a large category such as that of artifacts, words belonging to further subclasses show additional clustering. For more fine-grained WordNet-based semantic classes, classification accuracy increases even further (see Shafaei-Bajestan et al. 2024, for detailed discussion). The emergence of clusters in just a two-dimensional t-SNE plane indicates that English shift vectors are well-separated in a relatively low-dimensional subspace of the embedding space (cf. Stupak and Baayen 2022).^[5]

What are the theoretical implications of the clustering of English shift vectors by semantic class? It might be argued that we are observing performance noise, and that English pluralization is at its core a symbolic operation – the adding or exchanging of a feature value. However, as pointed out by Shafaei-Bajestan et al. (2024), languages such as Swahili (Polomé 1967) and Kiowa (Harbour 2008) have elaborate gender systems with exponents that vary with a set of high-level semantic classes. Given the present observations for English, it is conceivable that Swahili and Kiowa have grammaticalized parts of the differential semantics of nominal pluralization, whereas English has not.

Of course, one’s response to this observation could be: so what? On the other hand, what if the observed clustering of shift vectors is actually part of English speakers’ knowledge of their language? If so, there are non-trivial implications for our understanding of inflectional productivity. Rules depend on simplicity. It is this simplicity that enables generalization. If English nouns have plurals that are to some extent class-specific and even item-specific, then the implication is that predicting the precise meaning of a noun plural that has not been encountered before, given a singular that has been encountered before, is not possible.

Let me illustrate this simple point with an example. Consider how listeners might interpret the plural compound oak leaves. Listeners who do not know what an oak tree is will think of the leaves of some plant. Listeners who know that an oak is some kind of tree, will understand that the compound references the leaves of some tree that goes by the name of oak. Those listeners who can distinguish oak trees from willow trees and ash trees, will recognize that leaves such as depicted in Figure 2 are the leaves of oak trees. But European listeners will primarily think of the leaves of the European oak (left panel), whereas listeners in North America will first think of the leaves of the American oak (right panel). What the precise meaning of an inflected word is varies across speakers and listeners, and depends on their knowledge of the world.

Figure 2:

Leaves of the European oak (left) and the American oak (right). Source: Pixabay.com.

One might argue that this variability in understanding does nothing more than clarify that all that a grammatical description can reasonably accomplish is to associate plurality with a single general operation, such as an average shift vector in distributional semantics, or a change in a symbolic feature an in theoretical morphology. Undoubtedly, this keeps things simple, but it has an important downside. If embeddings capture important aspects of meaning, then linguistic theories that work with a single fixed featural change for nominal plurality give up completely on making precise predictions for the actual meanings of plural nouns.

Alternatively, we can embrace embeddings, and accept the challenge to think of how we might predict words’ meanings from their forms as well as possible. Before addressing this question, a cautionary note on embeddings is in order. The knowledge base underlying embeddings is super-human in many ways, both in terms of volumes of text and the wide coverage of topics. Embeddings cannot be expected to be precise for any single language user. Nevertheless, they are useful, especially as similarity scores of embeddings have been found to be precise predictors for aggregated human perceived semantic similarity (Boleda 2020). I look at them as crutches to walk with.

If we accept embeddings as legitimate and useful approximators of words’ meanings, then the question arises how close a listener can get to a word’s embedding given its form as input. This question can be addressed with a computational model of the mental lexicon that we have been developing in Tübingen, the discriminative lexicon model (DLM, Baayen et al. 2019; Chuang and Baayen 2021; Heitmeier et al. 2024).

The DLM is a simple model for studying the relation between words’ forms and their meanings. Both forms and meanings are represented by high-dimensional numeric vectors.^[6] The task that the model is designed for is to predict meanings from forms (comprehension), and forms from meanings (production). This is accomplished with the help of networks that map form and meaning onto each other. The model’s simpler and computationally light mappings make use of the linear algebra that underlies multivariate multiple regression, its more complex and computationally intensive mappings make use of deep learning networks. In what follows, I illustrate what can be achieved with the computationally light linear mappings.

An implementation of a DLM model provides a high-level computational approximation of the lexicon of a single language user. Although ideally a DLM model would be exposed to the language use and language exposure of a given speaker at a given point in their life, the data required for implementing such a model are not available (and for ethical reasons, I am hoping such data will never become available). As a consequence, a DLM model is usually trained on community-based form representations (in what follows, words’ standard written forms) and community-based semantic representations (in what follows, word2vec embeddings).

A linear mapping F that transforms a form vector c into its corresponding embedding s ,

can be estimated in three ways, given a corpus-based lists of word forms, the corresponding word frequencies, and their embeddings. First, one can estimate an optimal mapping for the set of word types without taking the frequencies with which these types are used into account. Such mappings estimate what we have called the ‘endstate of learning’ (henceforth EOL), which provides the regression solution for F . This regression solution is mathematically equivalent to the asymptotic solution that would be obtained with infinite step-by-step learning experience with the set of word types.

Second, one can let the model incrementally learn a mapping, step by step and token by token. This method comes into its own for training data that are explicitly ordered in time. Heitmeier et al. (2023b) used it to study within-experiment lexical learning, and also applied it to chronologically ordered data sets from childes (MacWhinney 2000). Differences in frequency of use in the training data are taken into account automatically, as more frequent words will be encountered more often during learning. However, this method is computationally intensive and relatively slow.

When no information on the temporal order of learning events is available, or when order information is not essential, then one can implement a form of frequency-weighted regression, henceforth FIL (frequency-informed learning, see Heitmeier et al. 2023a, for detailed discussion). FIL is as computationally light as EOL, and its predictions are similar to those of incremental learning, except for words with tokens that appear primarily either early in training, or late in training.

In what follows, we use the DLM to assess the pros and cons of working with regularized plural vectors, obtained by taking the empirical vectors of singulars and adding the average plural shift vector. Figure 3 presents the accuracies of EOL and FIL mappings constructed for the 11,500 singular and plural noun pairs the embeddings of which were previously visualized in Figure 1. The blue dots in Figure 3 represent the case in which a prediction is counted as correct when the predicted semantic vector has its targeted gold-standard vector as closest neighbor (accuracy@1). Orange dots represent the case in which the target vector is among the closest 10 neighbors of the predicted embedding (accuracy@10). Models using FIL are summarized in the upper panels, and models using EOL are visualized in the lower panels. The left panels present type-wise accuracy, the right panels present accuracy when token frequency is taken into account.

Figure 3:

Accuracy@1 (blue) and Accuracy@10 (yellow) for mappings trained with FIL (upper panels) and EOL (lower panels) for plurals with empirical shift vectors and regularized plurals obtained with average shift vectors. Accuracies are evaluated over types (left panels) and over tokens (right panels).

Figure 3 clarifies that when we regularize plurals by deriving them from their singulars with the average shift vector, learning accuracy always increases. The learning task is simpler, the plurals are more predictable. Second, when frequency of use is not taken into consideration (EOL learning), the advantage of working with average plural shift vectors is greatest. Higher-frequency words are more difficult to learn, they have more form neighbors. Regularizing high-frequency plurals alleviates this problem especially for EOL mappings. Third, with FIL, higher-frequency words are learned well, in contrast to lower-frequency words. However, with lenient evaluation (accuracy@10), performance is just about as good for empirical plural vectors as for regularized plural vectors.

From a usage-based perspective, a model trained with FIL is the most attractive. Heitmeier et al. (2023a) report that FIL outperforms EOL when predicting visual lexical decision latencies. FIL provides a good window on what words are more difficult to learn, and where speakers and listeners are likely to have learned words less well. Evaluation on a token-basis is attractive because it assesses how well learning serves actual use: knowing words that are encountered frequently is more important than knowing words that are hardly ever used. Evaluating with Accuracy@10 is also attractive, for two reasons. First, we are dealing with a classification task with 11,500 possible outcomes. Getting close is already an achievement. Second, from a cognitive perspective, it is highly unlikely that a listener would arrive at exactly the same conceptualization as the speaker, as language users have different experiences with the language and different life histories. Provided interlocutors succeed in communicating meanings that are similar enough, communication can proceed unhindered. The fact that token-wise Accuracy@10 is very high for empirical plural vectors, and almost as good as Accuracy@10 with regularized plural vectors, therefore suggests that from a usage-based perspective, not much is to be gained by regularizing plural vectors.

At the same time, regularizing plural vectors comes with a clear loss. Regularized plurals are a far cry from the true meanings of plural forms as gauged with the empirical embeddings. The averaging process used to generate regularized plurals filters out all world knowledge. The different ways in which multiple objects of the same type configure in the world (see Figure 4 for examples) is not part of what grammars with general shift vectors (or symbolic features) can predict. If the task of a grammar is to model and predict as well as possible what speakers really understand given text or speech input, and what speakers say when they have an idea or a message in mind, then simplified plural vectors simply don’t do the job. Prediction of exactly what a speaker had in mind is not possible, and not necessary, but this modeling example shows we can get the gist right. For instance, in the hypothetical case that the plural apples has not been encountered, the similarities in shape and function of apples and oranges may well be picked up from the embedding space, allowing the model to predict for apples configurations in the world similar to those of oranges and other large fruits.

Figure 4:

Multiple cows (left) and multiple oranges (right). A decent computational model should be able to get the gist of plural configurations right. Source: Pixabay.com.

3.2 Determining determiners

Behavioral profiling, analogical modeling of language (Skousen 1989) and memory-based learning (Daelemans and Van den Bosch 2005) typically work with categorical predictors that are carefully curated by the analyst. For any algorithm, decisions about representations are unavoidable. For behavioral profiling, compiling well-motivated and replicable feature values for large numbers of sentences, when done by hand, can be prohibitively expensive. More important for the present discussion than this pratical issue is that categorical features may not offer the precision that we really need.

By way of example, consider the problem of predicting the English determiner. Divjak et al. (2023) studied determiner use in the British National Corpus. To my mind, it is an exemplary study of what can be accomplished with behavioral profiling. And yet, have these authors really solved the problem?

Divjak et al. (2023) coded 2,200 tokens of nouns by hand for the number and countability of the noun, for whether the noun is part of a set phrase, for the presence of pre- or post-modifiers, for hearer knowledge, and for specificity. They report very high accuracies (close to perfect) using recursive partitioning trees and random forests. At first sight, this might suggest that with carefully selected and theoretically well-motivated discrete features, the problem of determiner selection in English is now solved.

However, the features provided to the tree and forest classifiers are themselves based on considerable human interpretation. For instance, ‘Hearer Knowledge’ is reported to be “based on whether it appears that the speaker assumes that the hearer knows what they are referring to” (p. 12). The authors admit that applying these kinds of concepts to actual data is not always straightforward. In fact, in the case of ‘Hearer Knowledge’, a problem of form (predicting the determiner) is replaced by a problem of meaning: figuring out when ‘Hearer Knowledge’ is present or absent. Although I can get some intuitive meaning out of this term, my mind shuts down on me when I try to think through what it actually means. I can imagine that for non-native speakers of English, developing a sense of what in the English speaking world counts as ‘Hearer Knowledge’ is a formidable task. These considerations suggest to me that what we need is a model for predicting the value of ‘Hearer Knowledge’ for a given utterance token. This predicted value can inform a feature-driven symbolic classifier. Unfortunately, this would imply a two-step process, and the lesson from end-to-end modeling in NLP and AI is that in-between steps tend to be suboptimal for modeling accuracy.

Large language models also predict English determiners with high accuracy. Figure 5 illustrates this for some 15,000 nouns in the Buckeye corpus (Pitt et al. 2005), using contextualized embeddings obtained with GPT-2. Two context windows were used: a wide window with 100 sublexical units (about 50 words) to the left of the article, and a narrower window with 20 sublexical units (about 10 words).^[7] For both windows, prediction accuracy of a linear discriminant analysis (LDA) with leave-one-out crossvalidation is shown for the contextualized embeddings of the articles themselves (distance 0) as well as for the words preceding the article, up to a distance equal to 3. Baseline accuracy (using the majority choice) is at 0.558. Except for words at distance 3, GPT-2 accuracies are significantly higher than the baseline accuracy.^[8] The somewhat better performance of contextualized embeddings obtained with a wider context window suggests that information at quite some distance from the target noun is already providing information on what determiner to use. It is also interesting that without any knowledge of the upcoming noun, and hence without any direct information about the upcoming noun’s number or countability, the LDA models are already extremely successful. In other words, the contextualized embeddings of the GPT-2 model capture micro-level variation that may motivate some features (e.g., ‘Hearer knowledge’). They also may render others unnecessary. In the case of grammatical number, for instance, the GPT-2 model may already be anticipating the upcoming grammatical number, and hence may not need to be provided explicitly with information about grammatical number that is realized on the noun. Contextualized embeddings appear to be looking into the future. The challenge for linguistics is to explain why.

Figure 5:

Contextualized embeddings are looking into the future: prediction accuracy for English determiners (the, a, an) in the Buckeye corpus using GPT-2 derived contextualized embeddings of the words at distances 3–0 from the determiner. The gray horizontal lines denotes majority baseline accuracy. Accuracy for a wide preceding context window is shown in red, and accuracy for a narrow preceding context window is shown in blue.

Of course, the question of whether actual human language users can make use of the information that LLMs extract from planet-wide global usage can only be addressed by experiments with real speakers, and with computational models trained on human-scale input using cognitively plausible learning mechanisms. LLMs are models constructed at a superhuman scale, using non-human learning algorithms. The way in which LLMs predict upcoming determiners is a far cry from what is currently known about human language processing. But my hunch is that they are powerful statistical tools for probing dependencies in language use.

3.3 Models and models

Thus far, I argued that abstract features such as ‘plural’ and ‘Hearer Knowledge’ can be suboptimal because abstraction may render useful fine within-category detail invisible, because abstraction may filter out essential knowledge of the world, and because an abstract feature may itself stand in need of explanation. What then is the role of abstraction, and of symbols, in the scientific method?

I can only offer a tentative line of thought that takes as point of departure the distinction made by Breiman et al. (2001) between machine learning and statistical modeling. Breiman and colleagues describe machine learning as goal oriented, it not being important to understand how a model works as long as it does what it is designed to do. By contrast, in statistics one sets up a mathematical model with predictors and response variables that are represented symbolically, in the hope is that the model can predict and even generate the data, apart from measurement noise, and in this way provide the analyst with an explanation of why we observe what we observe.

LLMs are machine learning models. Just as a smartly compressed version of the collected works of Plato does not provide a model of Plato’s language or Plato’s thought that humanist scholars would find useful, LLMs (conceptualized as smartly compressed corpora that are productive) do not provide a model of human thought or even human language. I think of them as Large Productive Compressed Corpora, rather than as language models.

When we build statistical models, we always work with abstraction and with symbols, but, paraphrasing (Box 1976, p. 176), we should guard against falling in love with our own models. The cautionary note in Box and Draper (1987) that all models are approximations, and thus, essentially wrong (p. 424), also applies to linguistic theories using abstract features for quantitative prediction. The practical question is how wrong a linguistic model has to be to not be useful (cf. Box and Draper 1987, p. 74). Of course, this raises the question of what counts as ‘useful’.

Symbolic quantitative models can be useful for language teaching. Here, linguistics has a long history. Assuming that in the near future we will not be wearing automatic translation earplugs, and that there still will be people who would like to learn another language themselves, being able to provide explanations for main trends in language use will continue to be very helpful. Symbolic quantitative models also help us to dig deeper, to engage in the iterative process of refining features so that prediction become increasingly more accurate. In principle, when applied to the same problem, with the same goal, machine learning and iterated linguistic modeling should converge. But I fear that as linguistic models become increasingly refined, the pleasure of understanding will decrease exponentially. Anyone who has tried to make sense of an anova model with a five-way interaction, or a recursive partitioning tree generating hundreds of splits, will know what I mean.

4.1 Why fund research in corpus linguistics and linguistic theory?

In the coming years, I anticipate that funding for the humanities, including linguistics, will decrease as the economic costs of the unfolding climate catastrophe for our societies skyrocket. The current global political situation adds to this with increasing defense spending. For linguistics, and corpus linguistics, to have a chance of survival in the academic marketplace, it is imperative that we can clarify to the tax-payer what the benefits of our discipline for society are.

As a humanities scholar, I strongly believe in the added value of better understanding how the human mind works within societies and their cultures. But as countries have to take measures to deal with, for instance, rapidly rising sea levels, increasing frequency and magnitude of storms, as well as raging forest fires, reduced amounts of funding across academia will flow primarily to applied sciences. Funding agencies in many countries are already requesting grant proposals to clarify their practical value for society, and I expect the pressure on submitting proposals outlining applicable research to increase over the coming years. At the moment, we can get away with paying lip service to applicability. However, as a field, we will have to provide clear answers to the question why linguistics, and corpus linguistics, should be funded. After all, with LLMs, society now has unprecedented highly effective language technology at its fingertips – it seems that there is nothing that this wompom cannot do. Why would citizens be required to fund academic research in corpus linguistics and linguistic theory now that engineers “have figured it all out”? This is a harsh question, and I don’t have any good answers. But here is one thought.

People without linguistic training tend to think that there is correct language use, as opposed to incorrect language use. This rigid perspective on language reinforces ingroup versus outgroup behavior and is easily harnassed to serve the interests of demagogues. However, the accumulated results of corpus linguistics over the past decades indicate that language is not deterministic and fixed, but stochastic, a patchwork of constructions, idioms and collocations, with writing conventions that diverge considerably from actual conversational speech. But this is not common knowledge. Although teachers at secondary schools must have some knowledge of the key findings of corpus linguistics, sociolinguistics, and dialectology, little of this knowledge is passed on to their students. These students, some of whom will later in life be policy makers, need to be much better informed about the variability that is inherent in language.

I’m hoping that a better understanding of the true nature of language will reduce the risk of policy makers to overvalue the standard language or their own sociolect. To achieve this, it will be essential that we better communicate the core insights of our field to the general public. An example from sociolinguistics that I find impressive is the documentary entitled “Talking black in America” (https://www.youtube.com/watch?v=8QFpVgPl9tQ), an excellent response of American sociolinguistics to widespread negative sentiments and misconceptions in the USA about English as spoken by black Americans. For educational purposes, having similar documentaries about key results from corpus linguistics need to be developed. Many outstanding videos explaining central concepts from mathematics and AI are available on the web. Similar materials, perhaps gamified, need to be developed to enable the general public to get a better understanding of language and language use, the variability of language, and the social and cultural importance of language diversity.

4.2 Environmentally sustainable corpus linguistics

In 1950, Sir Ronald Fisher pointed out the following:

For the future, so far as we can see it, it appears to be unquestionable that the activity of the human race will provide the major factor in the environment of almost every evolving organism. Whether they act consciously or unconsciously human initiative and human choice have become the major channels of creative activity on this planet. Inadequately prepared we unquestionably are for the new responsibilities, which with the rapid extension of human control over the productive resources of the world have been, as it were, suddenly thrust upon us. (Fisher 1950, p. 20)

More than 70 years later, human-induced climate change is unfolding at a catastrophic pace. The planet is seriously ill: it has intelligence. Corpus linguistics and corpus based natural language processing are contributing to climate change, just as all other sectors of modern society. Figure 6 plots cumulative carbon emissions as a function of time from 1930 to 2022 (data from Friedlingstein et al. 2023). This is what humanity has been pumping into the planet’s atmosphere. There is no sign that this behavior will stop anytime soon.

Figure 6:

Cumulative fossil fuel and cement emissions (in GtC) for 1930–2022, annotated for the emergence of selected resources for corpus linguistics.

The date of publication of a selection of resources for corpus linguistics has been added to the plot, beginning with A teacher’s word book of 30,000 words by Thorndike and Lorge (1944) and the word frequency lists published by Kučera and Francis (1967), and ending with the 2022 release of Common Crawl. The magnitude of resources now available to corpus linguistics has increased along with the ever growing global carbon footprint of humanity. The contribution of increasingly large corpora to humanity’s global carbon emissions is likely to be small, and I know, correlation is not causation. But it is not unlikely that even though storage of electronic data is becoming cheaper, the rate at which storage devices are used is increasing even faster, leading to a net increase in construction costs and of costs of use, across society,^[9] including corpus linguistics.

Environmental costs are not only increasing with storage space, but also with computational resources. Data centers housing LLMs and AI systems currently are responsible for 2 % of global electricity consumption, and this figure will have doubled in the near future. These systems also need enormous amounts of fresh water to cool processors. Within years, large AI systems may be consuming as much energy as entire nations (Crawford 2024). The planet not only has intelligence, but now it has artificial intelligence as well. It is important to realize that the carbon footprint of LLMs is not restricted to development, but also extends to use (think of applications such as Meta-Llama 3). A search using generative AI is estimated to use four to five times the energy required by a conventional web search (Crawford 2024).^[10]

Even though computations and storage have become increasingly energy efficient, the rate at which our use of the new tools and resources increases has been increasing even more. AI and machine learning are no exception (see, for AI and NLP Wu et al. 2022). Although efforts are under way to mitigate the environmental costs of AI and LLMs (Zhu et al. 2023), what we really need is green AI, defined by Schwartz et al. (2020) as “AI research that yields novel results without increasing computational cost, and ideally reducing it”. As long as a novel AI application is not carbon neutral, for the sake of future generations, it should be banned from large-scale public use.^[11]

The challenge to become carbon neutral is not only urgent for AI and machine learning, it extends also to corpus linguistics (and any other scientific discipline). Simple measures at the personal level are to minimize air travel, to drastically reduce meat consumption, and to switch off your modem when not in use. But collectively, we also need to become much better aware of the carbon costs that come with the use of computer programs and applications. What is the carbon footprint of downloading English fasttext embeddings, or of training a deep learning model for 2 weeks, or fitting a linear mixed model with maximal random effects that takes 3 h in R and then reports it didn’t converge?^[12]

At present, we act without thinking about environmental costs. I confess guilty. We have ethics boards that assess whether our experiments could be harmful to participants, but the harm of carbon-intensive research for upcoming generations seems to be outside the radar of academic ethical awareness. Funding agencies should be encouraging us to explain how we plan to keep emissions low or how we will offset them. Perhaps journals, including Corpus Linguistics and Linguistic Theory, should ask researchers to publish estimates of the amount of electricity that was required for the research reported. As researchers, we will need to learn how to balance the higher carbon costs of energy-intensive methods, as compared to carbon-leaner alternatives, against the benefits of their use (see also Dhar 2020; Schwartz et al. 2020; Strubell et al. 2019).

Let me provide two examples of the kind of issues that I am running up against in my own research practice. The first example concerns fitting statistical models to empirical data. A complex model may require hours of computation time. A simple model may require only a few minutes. Is 3 h of computation time justified by the additional precision that is supposedly gained? This question is exacerbated by experiences indicating that for reasonably sized datasets, statistical models that take many hours to fit are likely overfitting the data.

My second example concerns computational modeling within the discriminative lexicon framework. We have been developing software facilitating the implementation of DLM models (Heitmeier et al. 2024). This software is written in the Julia language, following recommendations of Douglas Bates and Reinhold Klieg, a language that is highly optimized for numerical computations. Most of our models have been implemented with linear mappings between meaning and form. It is surprising how well these simple mappings perform. Recently, however, we have been exploring deep networks. Deep mappings tend to outperform linear mappings with respect to comprehension and production accuracy.^[13] The problem that we are now running into is whether to recommend using the simple linear mappings, which for standard datasets can be estimated on a normal CPU within a few seconds, or should we recommend to use deep networks, which take several hours to train on a GPU. Does the greater precision of deep networks justify a carbon footprint that is a thousand times greater?

Perhaps a reasonable environmentally-aware ‘code of conduct’, both for statistical modeling and for computational modeling, is to conduct exploration with models that are simple,^[14] and only build complex models when doing so is truly crucial for theory development or practical application. If so, a complex model should be run once only.

4.3 From butterfly collection to model building

Unfortunately, to put it bluntly, a lot of research on language looks disquietingly like butterfly collecting. An individual study can be well designed, competently executed, and highly informative. Across many such studies, a general picture is emerging of how language works. This is all great. Unfortunately, a body of loosely connected research questions, analysed with a wide variety of research methods (however well-motivated) now has to compete with LLMs. Ideas about how language might work are no match for a technology that actually works, that works extremely well, and that is going to be integrated into widely used application software.

I believe that as a field we should be more ambitious, and aim for comprehensive computational models that are cognitively motivated, trained on human-scale data, with algorithms that respect important properties of human learning. Without implementation, theories of language are at best food for thought, and in the worst case belief systems similar to the many other belief systems that permeate human culture (see Horton 1967, for discussion of the rationality of pre-scientific thought). Humanistic cognitive computational models will be theories of language that enable us to predict much more effectively the details of language acquisition, language change over the lifespan, the interaction of language and social identity, and the interplay of language and cognition. What (corpus) linguistics needs is something along the lines of what Anderson’s ACT-R model (see, e.g., Taatgen et al. 2006) is in psychology: an integrated model that generates precise quantitative predictions for vast arrays of empirical findings.

I am hoping that humanistic cognitive modeling in an interdisciplinary context will also contribute to greening AI. As pointed out by Schwartz et al. (2020), “Ironically, deep learning was inspired by the human brain, which is remarkably energy efficient.” The long-term goal of humanist cognitive computational modeling of language can actually be energy-lean models that take inspiration from on the one hand the fact that we now know for sure that probability in language can be dealt with effectively (as demonstrated by LLMs), and on the other hand by the fact that we also know a green solution exists as well (as demonstrated by the human brain).