Not so SUBTLE(X): word frequency estimates and their fit to sentential reading times in interaction with predictability

2.1 Frequency

One of the most important psycholinguistic variables to explain processing times of words is frequency, which is normally measured as the frequency of occurrence of a word given a particular corpus. The usual observation made across many, many studies is that high-frequency words are easier to process than low-frequency words (for review, see Brysbaert et al. 2018; Staub 2015, 2024), and this facilitatory effect of word frequency has been found for a variety of research methods, such as lexical decision (e.g., Brysbaert and New 2009; Brysbaert et al. 2011a; Chen et al. 2018; Heister and Kliegl 2012), word recognition and naming (Schilling et al. 1998), eye-tracking during reading where frequency influences even the earliest stages of word processing such as lexical access (e.g., Inhoff and Rayner 1986; Kliegl et al. 2004; Rayner et al. 2004; Whitford and Titone 2014), and self-paced reading, as in the present study (e.g., Kapteijns and Hintz 2021; Tremblay et al. 2011). For the German language, a variety of frequency measures have been published so far (for review, see Brysbaert et al. 2011a; Heister and Kliegl 2012); three that are critical in the present study are the CELEX, dlexDB, and SUBTLEX corpora. CELEX is a corpus dating back to the mid-1990s that consists of 5.4 million word tokens derived from written text (i.e., newspapers and books) and 600 thousand word tokens derived from transcribed speech (Baayen et al. 1995). Despite its age, CELEX continues to be used by state-of-the-art research (see e.g., Crossley et al. 2014; Mickan et al. 2024; Petilli and Marelli 2024), which is likely a result of the long-standing prominence of this corpus (see Heister et al. 2011, for discussion). DlexDB, in turn (see Heister et al. 2011), is a more recent compilation and relies on the corpus compiled by DWDS (i.e., Digitales Wörterbuch der deutschen Sprache; see Geyken 2007). DlexDB consists of 122,816,010 tokens and is, in comparison to CELEX, somewhat less restricted in terms of genre as it comprises texts from books, newspaper articles, transcribed speech from all over the 20th century. Finally, the SUBTLEX database is a corpus of 25.4 million words that was derived from the subtitles of 4,610 movies and TV series (i.e., German and non-German TV material; see Brysbaert et al. 2011a; also see Heister and Kliegl 2012, for discussion).

Given the importance of word frequency in language processing, an obvious question is which one of the three corpora – i.e., CELEX, dlexDB, or SUBTLEX – psycholinguists should rely on. To date, several studies have obtained independent evidence that the SUBTLEX frequencies tend to outperform other corpora when it comes to accounting for word processing times (e.g., Brysbaert et al. 2011a, 2018; Chen et al. 2018; Heister and Kliegl 2012). For example, Brysbaert et al. (2011a) compared the goodness of fit of the SUBTLEX and the CELEX corpora (among others) in predicting lexical decision times of German words presented in isolation. They relied on previously published data from three lexical decision experiments where participants were mainly psychology students from various German universities. Goodness of fit was operationalized as the magnitude of the correlation coefficient between the various frequency measures and lexical decision times, such that higher correlation coefficients were deemed better than lower ones, as they reflect a greater amount of explained variance. According to the results, SUBTLEX outperformed the other frequency databases across all three experiments. According to the authors, the superiority of the SUBTLEX corpus compared to other corpora might be related to the type of register that is normally used in movies and TV shows, compared to written sources such as newspapers or books. Psychology students may have had less exposure to this latter type of register, by watching more television and spending less time reading (also see Brysbaert et al. 2011b). Similar findings were obtained in a subsequent study by Heister and Kliegl (2012), who compared the goodness of fit of several corpora in accounting for lexical decision times, by additionally taking into account the emotional valence of words. The authors found that corpora that include large amount of words high in emotional valence (such as movie subtitle corpora or corpora derived from tabloid newspapers such as the German newspaper Bild) outperformed others in accounting for lexical decision times. The authors explain this result by arguing that emotional words are processed faster than neutral ones, yielding larger correlation coefficients with response times. However, another intuitive reason for the superiority of the subtitle corpus could simply lie in the fact that both studies relied on previously published lexical decision data that, at least partially, consisted of word lists high in emotional valence (see Brysbaert et al. 2011a: 417). It seems somewhat intuitive that a frequency corpus based on emotional and dramatic content (as in movies and TV shows) would explain more variance in emotional rather than neutral word lists (see Baayen et al. 2016; Heister and Kliegl 2012, for discussion). As a case in point, Baayen et al. (2016) showed that subtitle frequencies perform worse in predicting outcomes of tasks other than lexical decision (in this case, eye-tracking during reading), simply because lexical decision promotes the impact of variables highlighted in subtitle frequencies.

An open question is therefore not only how various frequency databases (here CELEX, dlexDB, and SUBTLEX) perform when using neutral words as opposed to emotional ones, but also, how they perform in psycholinguistic paradigms other than lexical decision that likely promote different processing strategies. For example, lexical decision tasks present words in isolation, which may promote rather wholistic, one-on-one, recognition of words, where frequency might have a strong impact (e.g., Barton et al. 2014). Sentential reading paradigms such as used in the present study emphasize serial processing, due to the requirement of recognizing words in a broader linguistic context, which may down-regulate the impact of frequency effects (e.g., Kretzschmar et al. 2015).

2.2 Predictability

Predictability is a second prominent measure that explains the efficiency of language processing. Predictability is often assessed by means of the cloze procedure in which participants are asked to complete sentence frames that are truncated before the final word (e.g., Taylor 1953). The cloze probability of a word then reflects the proportion of participants who completed the sentence with that word. A robust body of research has shown that high-predictability words are processed more quickly and effortlessly than low-predictability words, and across a variety of research methods (for reviews, see Kuperberg and Jaeger 2016; Kutas and Federmeier 2011; Staub 2015; Van Petten and Luka 2012, among others).

An empirical question in psycholinguistic research is whether frequency and predictability affect word processing independently from one other or jointly (e.g., Kretzschmar et al. 2015; Shain 2019; Staub 2015; Whitford and Titone 2014). This question is not only relevant for models of eye-movement control during reading (e.g., Engbert et al. 2005; Reichle et al. 2009), which differ from one another in how strongly they advocate additive versus interactive effects of frequency and predictability. This question is also relevant to psycholinguistic models of word recognition, which emphasize either context-independent lexical access mechanisms (e.g., Coltheart et al. 2001; Seidenberg and McClelland 1989), advocating separable effects of frequency and predictability, or unified comprehension that depends entirely on the incremental probability of a linguistic structure, subsuming frequency and predictability in a single processing stage (e.g., Hale 2001; Levy 2008; Norris 2006; for dissemination, see Shain 2024; Staub 2024).

If predictability and frequency interact, an intuitive hypothesis could be that predictability effects should be more pronounced for low-frequency words, as high-frequency words should have strong levels of activation in mental lexicon anyways (e.g., Kretzschmar et al. 2015; Shain 2024; Staub 2024). Crucially, this hypothesis is not supported by empirical results obtained in previous eye-tracking studies (e.g., Altarriba et al. 1996; Ashby et al. 2005; Bélanger and Rayner 2013; Kliegl et al. 2004; Rayner et al. 2004; Whitford and Titone 2014; for review, see Staub 2015). For example, Rayner et al. (2004) investigated fixations made on high- and low-frequency words presented in sentential contexts that either rendered the words predictable or unpredictable. Across eye-movement measures, the authors found no statistical evidence for interactive effects of frequency and predictability: even though fixations were shorter for high-compared to low-frequency words, predictability reduced fixation durations to a similar extent for high- and low-frequency words. Even though the absence of a statistical effect does not constitute evidence for the null hypothesis (see Wagenmakers et al. 2007, for dissemination), it is striking to observe that, out of the numerous eye-tracking studies that have examined joint effects of predictability and frequency over the years, none have observed interaction effects between predictability and frequency, which would be expected based on Type I error probability alone (see Kretzschmar et al. 2015). Nevertheless, many of the eye-tracking studies that factorially manipulated predictability and frequency were likely underpowered: for conventional ANOVA analyses, a 2 × 2 design requires a total of 84 subjects to detect a statistical interaction with 80 % power. Few if none of the previous reading time studies included such a sample size.

4.1 Participants

We pooled the data from five self-paced reading studies conducted in our lab. The final sample of participants included 329 native German-speaking adults (mean age = 26 years, range = 18–41; 184 female, 139 male, 3 non-binary), who participated for financial compensation (participants recruited through Prolific, n = 171) or course credit (participants recruited through the university’s study recruitment website, n = 155). Participants recruited through the university’s study recruitment website were mostly first- and second-year psychology undergraduates. The majority of participants recruited over Prolific were not students but worked in part-time or full-time jobs (n = 87). Sixteen Prolific participants were unemployed/job-seeking; 46 Prolific participants had not provided information on employment status. Hence, our participants were not limited to psychology students, and occupational background varied among participants. All participants had normal or corrected-to-normal vision and reported no neuropsychiatric medication and/or a history of language impairments at the time of testing.

4.2 Materials

Materials consisted of 46 constraining German sentence frames (e.g., Unser freundlicher Nachbar mähte für uns kürzlich den…, English translation: ‘Our friendly neighbor mowed for us recently the…’), which were continued with a predictable or unpredictable noun (e.g., Rasen, ‘lawn’; vs. Hof, ‘courtyard’). Each sentence was concluded by a sentence continuation (identical for predictable and unpredictable versions), which was inserted to allow for spill-over effects after the noun, common to self-paced reading (Keating and Jegerski 2015; Witzel et al. 2012; e.g., auf dem Grundstück nebenan, ‘of the property next door’). Examples of experimental sentences and critical regions are presented in Table 1.

Noun predictability was assessed by means of a cloze pre-test in which 40 psychology students who did not participate in the main experiment (age range: 18–26, 26 female, 14 male), were asked to complete each sentence frame with the first word that came to mind. According to the results of the cloze test, predictable nouns had a cloze probability of 0.78 (Range = 0.29–1.00). Unpredictable nouns had a cloze probability of zero throughout (i.e., no unpredictable noun was ever produced as a response in the cloze test). All sentential materials are accessible under this paper’s OSF project using the link, https://osf.io/zn4am/?view_only=1e69f68799e143be9b40dc1f2f83b719.

4.2.1 Frequency estimates

Frequency estimates for each word in the critical region were taken from three corpora: SUBTLEX-DE, CELEX, and dlexDB.^[1] The critical region consisted of the predictable/unpredictable noun, as well as the two words following the noun (i.e., the spill-over region; see Table 1). For each word, we added to the data the word form frequency estimates from all three corpora. Common problems when comparing frequency estimates from different corpora are differences in corpus size and missing values. In order to address these, we converted the per-million frequency estimates to Zipf values (e.g., Brysbaert and Diependale 2013; Carroll 1970; Chen et al. 2018; Heuven et al. 2014; Zipf 1949; see). Zipf values are useful because they quantify frequency estimates on a common scale (i.e., 1–7, with 1 being very highly infrequent and 7 being extremely frequent), while also taking into account missing observations in the corpus. To convert to Zipf values, we used the formula, log10[ (f + 1)/N] + 3, where f refers to the raw frequency count of a word in a corpus and N to the size of the corpus (in millions).^[2] Figure 1 shows histograms of Zipf frequencies for each word in the critical region, split out by SUBTLEX-DE, CELEX, and dlexDB. As can be gleaned from the figure, the frequency estimates in the three corpora were similar, but not identical, which underscores the relevance of our research question.

Figure 1:

Histograms of Zipf frequency values in SUBTLEX-DE, CELEX, and dlexDB, split out by each word in the critical region.

4.3 Procedure

We pooled the data from five online self-paced reading experiments which we ran in the past; all experiments were run using the experimental presentation software LabVanced (Finger et al. 2017). The main task was a word-by-word self-paced reading task (no moving-window reading; no mask). Comprehension questions that followed after one third of the sentences ensured that participants were reading for comprehension. Participants saw one word appear in the center of the screen at a time and pressed the space bar to reveal the next word. They were instructed to read all sentences as quickly and accurately as possible, and to answer all true/false comprehension questions as accurately as possible by pushing the “J” (Yes, correct) and “N” (No, incorrect) keys on the keyboard. Sentences were separated by a 500 ms fixation cross. Each experiment also included a minimum of 30 moderately predictable filler sentences (taken from the Potsdam sentence corpus or earlier studies in our lab). The filler sentences were added to make sure that participants continued to generate predictions in the course of the experiment, despite having predictions disconfirmed multiple times (e.g., Delaney-Busch et al. 2019; Fine et al. 2013; Kukona and Hasshim 2024).

5.1 Research Question 1: Which frequency corpus performs best in predicting reading times in context?

5.2 Research Question 2: Is there evidence for interactive effects of frequency and predictability?

6.1 Estimating frequency effects in psycholinguistic research

The results of the present study could not substantiate the superiority of subtitle frequencies that previous research has highlighted (e.g., Brysbaert et al. 2011a, 2018; Chen et al. 2018; Heister and Kliegl 2012). Instead, in the majority of our analyses on critical-region words (i.e., the noun+1 and noun+2 analyses, but not the noun analysis, see below), models that included the dlexDB frequencies had the best fit to the data. We believe that the primary reason for these discrepant findings lies in the methodology that was used here. The present study relied on self-paced reading of words in context, whereas most previous investigations had used lexical decision tasks. We know that lexical decision likely promotes wholistic rather than serial processing of words (e.g., Barton et al. 2014), and that wholistic processing is more likely to be impacted by form-related variables such as lexical frequency, but also valence, a variable the present study could not include (see Method section). It seems trivial that frequency databases such as SUBTLEX, which are known for overusing sad and happy words rich in such form-related variables (e.g., Baayen et al. 2016), would have the best fit to lexical decision data. Ultimately, the results from our study call into question the current practice of many, many psycholinguistic studies that use subtitle frequencies as their de facto default.

In the same vein, the superior performance of the SUBTLEX frequencies for specifically the noun region could be an artifact of the way the SUBTLEX corpus is built. The SUBTLEX corpus is built on large amounts of emotionally valenced words, and the frequency estimates that pertain to these words may simply work best for nouns, as most nouns are concrete, refer to objects in the world, and give the language user some kind of visual or tactile experience. Instead, most words in the N+1 and N+2 region in the present study were prepositions or pronouns – i.e., words for which such haptic experiences do not apply. Indeed, when we ran follow-up models on a subset of our data that only included normal nouns (irrespective of region), we found that SUBTLEX frequencies had the best fit.

The fact that, across models, CELEX performed worst as frequency predictor could be a consequence of the sheer age of this corpus, but it could also be an artifact of its size: CELEX is the smallest corpus in our investigation and, despite the Zipf-normalization approach used here (which attempts to minimize differences in corpus size), the CELEX frequencies may have the noisiest estimates.

That being said, our results also show that, in the majority of our statistical models, word frequency did not account for a substantial amount of variance in the data. In contrast, model predictors that made much more sizable contributions to reading time were contextual predictability or non-parametric variables such as trial number. This aspect of our findings is noteworthy as it mirrors previous work using event-related potentials (ERPs)which has often shown that, even though frequency explains a sizable portion of N400 amplitude when words are presented in isolation (e.g., Grainger et al. 2012), these frequency effects tend to become smaller or even disappear for words processed in sentential context (e.g., Osterhout et al. 1997; Van Petten and Kutas 1990). This shows that a linguistic context can diminish or even override frequency effects during language processing (see Kretzschmar et al. 2015; Staub 2024).

A question that remains open in the context of our statistical analyses is why frequency effects seemed to differ depending on population (i.e., Prolific workers vs. psychology students), at least numerically. This question was not of primary interest to the present study, but some of the models in our analysis showed clear improvements in data fit once the interaction between frequency and population was included. Nevertheless, the results are somewhat puzzling, as they are inconsistent: for nouns, psychology students showed numerically smaller frequency effects than Prolific workers, but on the spill-over region (N+1 word), the reversed pattern surfaced, such that psychology students showed numerically larger frequency effects. We would like to refrain from over-interpreting these interactions, as most of them remained statistically non-significant. However, there is now a growing body of research showing that individual experience and expertise shape frequency effects in lexical processing (e.g., Baayen et al. 1996; Halteren et al. 2005; for discussion, see Baayen et al. 2016). This could potentially indicate that some of the interactions that surfaced in our analysis were, nevertheless, meaningful and could be driven by genuine individual differences between these two populations (e.g., educational attainment, socio-economic status, political views, but also internet usage or social media activity and many more). It will take more research with more consistently counterbalanced subject groups to further substantiate these findings. Nevertheless, this aspect of our results underscores the necessity of conducting research on target populations other than psychology students.

6.2 Additive versus interactive effects of frequency and predictability

Results for our second research question showed that frequency and predictability likely make independent contributions to language processing. Our statistical models yielded no improvement in model fit when specifying the interaction between frequency and predictability over and above taking into account both main effects. In the present study, this result is unlikely to be driven by low power, since we used a sufficient amount of both subjects and items (i.e., single words) in order to find an interaction between the two variables. However, our frequency estimates from all three corpora showed relatively low rates of variability relative to the full range of the frequency metric (see Figure 1). Therefore, we cannot fully exclude the possibility that different results would obtain with a more widespread set of frequency distributions. Nevertheless, the additive effects of predictability and frequency found here do match with a large body of previous research manipulating frequency and predictability factorially to show that there are no joint effects of predictability and frequency in eye-tracking during reading; rather, frequency and predictability have additive effects (e.g., Rayner et al. 2004). Additive effects of these variables support models of language processing that assume a staged processing architecture in which lexical properties of incoming input (such as frequency) are evaluated independently of prior context (such as predictability). An avenue for future research will be to pin down the exact time course with which these processes unfold. Classic psycholinguistic models of word processing such as the DRC (dual route cascaded) or the Logogen models (e.g., Coltheart et al. 2001; Morton 1969) assume a strictly serial architecture, in which lexical access precedes contextual or semantic integration, with the latter only happening during later stages of processing. Strictly speaking, this would imply that frequency and predictability effects cannot occur together, i.e., at a single processing stage, in line with the present results. In contrast to this, recent findings from eye-tracking challenge this strictly serial processing view by showing context effects even in very early measures of eye-movement control during reading (e.g., Staub 2015; Staub and Goddard 2019). In the same vein, large language models challenge the modularity view of classic psycholinguistic research as they no longer have a clear distinction between lexicon and syntax. It is difficult to put these results together and draw firm conclusions as to the underlying processing architecture of the cognitive system, but tentatively, they are consistent with a view in which frequency and predictability affect processing in potentially encapsulated modules, but they do so at overlapping time points (i.e., the seriality assumption would need to be revised). This is no doubt an exciting avenue for future research.

6.3 Limitations and conclusions

In the introduction of this paper, we asked a simple question: which frequency database should psycholinguistic research in German predominantly rely on? Given that our results largely show statistically invariant effects of word frequency, and since other predictors were nearly ten times as potent in explaining variability, one may feel tempted to argue that, for word processing in context, the choice of frequency database does not matter much. However, that is a stronger statement than we wish to make.

First, as we argued above, our results may reflect a lack of variance in frequency estimates rather than a genuine indifference between the frequency databases, which means that it will take more studies to answer this question definitively. Second, even if that conclusion is valid, our results are strictly limited to the research method used here, i.e., sentential self-paced reading. Above, we have reviewed evidence suggesting that different research methodologies (i.e., ERPs, eye-tracking) promote diverging processing strategies, i.e., wholistic versus serial processing, which emphasize and down-regulate effects of frequency (e.g., Barton et al. 2014). The self-paced reading paradigm used here likely reflects a hybrid between these two, as it presented single words to subjects at a time, but nevertheless required some degree of serial processing, as words needed to be comprehended within a sentential context. A fruitful avenue for further investigation will therefore be to investigate frequency effects on the same materials using research methodologies that promote more wholistic versus more serial processing (e.g., lexical decision vs. eye-tracking during reading, respectively). Only then will we be able to isolate genuine effects of frequency from those of the research paradigm at hand.

Regarding the choice of frequency database, our results are not in line with previous investigations showing that subtitle frequencies perform better than text-based frequency estimates (Brysbaert et al. 2011a; Chen et al. 2018). In fact, in the majority of our analyses, the dlexDB – and not subtitle – frequencies had the best fit to the data. Above, we have suggested possible explanations for these discrepant findings, e.g., differences in methodology. Ultimately, the findings reported here highlight that recommendations as to the use of one frequency database over another are only warranted when they are supported by a multitude of research methods. Hence, we concur with previous studies (e.g., Baayen et al. 2016; Heister and Kliegl 2012): issuing a blind recommendation to use subtitle frequencies as a default is, indeed, premature.