LexCH: a quick and reliable receptive vocabulary size test for Chinese Learners

2.1 Measuring vocabulary size

When assessing vocabulary knowledge, an important distinction concerns the size or breadth of vocabulary knowledge and the depth or quality of this knowledge (Anderson and Freebody 1981). Despite some debate over the conceptual difference between breadth and depth (e.g., Vermeer 2001), empirical studies have shown that the two are unique—at least for lower-frequency words and larger vocabulary sizes (Schmitt 2014). Most vocabulary size tests in this field assess the breadth of vocabulary knowledge, i.e., how many words are known, rather than the depth of vocabulary knowledge (Read 2000).

The Vocabulary Size Test (VST) (Nation and Beglar 2007), designed to measure L1 and L2 learners’ written receptive vocabulary size in English, is one of the most popular vocabulary size tests in applied linguistics. The sampling of test items is based on the frequency levels of word families occurring in the British National Corpus up to Level 6 (Bauer and Nation 1993). With the aim of capturing total vocabulary size, the VST “represents the various frequency levels without a bias towards any particular frequency levels” (Nation 2012, p.2). The test contains 140 items; they are presented in a decontextualized multiple-choice format, in which learners are instructed to choose the best definition of each word from four choices. Studies suggest that the VST has good validity (e.g., Beglar 2010). Even so, it has received criticism for its multiple-choice format because the construct of meaning recognition does not approximate real language activities such as reading and listening. Instead, the construct of meaning-recall, where learners must remember a word’s meaning upon seeing the term, is more closely related to everyday language use (Gyllstad et al. 2015; Kremmel and Schmitt 2016; Stewart 2014). A second alleged limitation of this format is that individuals’ scores can be inflated due to test strategies such as guessing (Nation and Coxhead 2021). Finally, the full VST typically takes between 40 and 60 min to complete, which is inefficient for research in which participants take multiple tests.

Another influential vocabulary size test is the Eurocentres Vocabulary Size Test (EVST) (Meara and Jones 1987, 1990). This test adopts a Yes/No format. The test was originally used to measure L1 vocabulary size (e.g., Anderson and Freebody 1983; Zimmerman et al. 1977) before later being adopted by second language researchers. In the EVST, items are presented as a list of words in isolation, and test takers are asked to respond “Yes” to the words they know and “No” to words they do not recognize. To avoid inflated scores, half of the test items are pseudowords. Points are deducted if test takers respond affirmatively to pseudowords. This kind of lexical judgement task is thought to represent a special type of meaning-recall task that is more similar to the mental processes accompanying reading and listening (Zhang et al. 2019). Moreover, test construction and administration are easier compared with other formats, making it possible to sample a high number of words and to collect data from large participant groups. Although the Yes/No format has been questioned in several aspects, further investigation has indicated that these issues are not truly problematic (Amenta et al. 2021). Some researchers have pointed out that in responding to a Yes/No item, learners can draw on partial knowledge of the word that is not strong enough to enable them to tackle the word when reading or listening (e.g., Nation and Coxhead 2021). However, others have reported that the results of Yes/No tests are closely correlated with other test types, such as multiple-choice, translation, and interview tasks (Nakata et al. 2020; Stubbe 2012; Zhang et al. 2019). Meanwhile, some scholars have critiqued the test’s isolated and decontextualized way of presenting items. However, we agree with Read (2000) that it is unnecessary to present words in context if an instrument is not intended to measure test takers’ ability to use contextual information when responding.

2.2 LexTALE

The Lexical Test for Advanced Learners of English (LexTALE) developed by Lemhöfer and Broersma (2012) is a vocabulary size test targeted at advanced English adult learners. It contains 60 items, including 40 words and 20 nonwords selected from the 240 items of an unpublished vocabulary size test called “10 K” by Meara (1996). Test takers are instructed to respond “Yes” to an existing English word and “No” to a nonexistent English word. LexTALE test results have been shown to correlate highly with general English proficiency (Lemhöfer and Broersma 2012; Nakata et al. 2020; Zhang et al. 2019). Given the sound validity and convenience of administering the LexTALE (which takes only 3–5 min to complete), researchers have adapted the instrument into Dutch and German (Lemhöfer and Broersma 2012), French (Lextale_FR, Brysbaert 2013), Spanish (Lextale_Esp, Izura et al. 2014), Chinese (LEXTALE_CH, Chan and Chang 2018), and Italian (LexITA, Amenta et al. 2021). The Chinese version LEXTALE_CH is a character-based vocabulary size test; that is, it evaluates how many characters test takers know. However, most Chinese words are combinations of multiple characters, so the number of known characters should not equal the number of words known. This consideration is detailed in the ensuing section (see Section 2.3).

All existing LexTALE versions adopt the same format and take word-frequency as the critical criterion. The actual number of items varies. LexTALE contains 40 words and 20 nonwords; Lextale_FR consists of 56 words and 28 nonwords; Lextale_Esp, LexITA, and LEXTALE_CH have 60 words and 30 nonwords each. Raising the number of words is meant to increase the test’s reliability and make the test applicable to both L1 and L2 speakers (Brysbaert 2013). Some researchers have contended that the test might not include enough difficult words for L1 speakers. For example, a study of LexITA reported a ceiling effect for the L1 group, with nearly perfect responses (mean score = 57.9/60) and a small standard deviation (SD = 2.15, range = 50–60 in Phase 1) (Amenta et al. 2021). The authors stated that the test was appropriate for L2 learners but should have increased the number of difficult words for L1 Italian speakers. Additionally, although Izura et al. (2014) argued in a study of Lextale_Esp that there were no signs of a ceiling effect for L1 speakers, the average score for this group was 53.8 out of 60 (approximately 90%). Raising the number of difficult words could possibly lead to better discrimination among L1 speakers.

2.3 Measuring lexical knowledge in Chinese

Chinese characters are the basic unit of Chinese writing. Each character represents a syllable. In modern Chinese, some words are composed of only one character, such as 天 [sky], 火 [fire], and 跑 [run]. These words are called single-syllable simple words (Fu 2020) and constitute a small proportion of all Chinese words. Other Chinese words, including double- or multisyllable simple words, derivative words, and compound words, consist of more than one character. In many cases, the meaning of these words is not simply the combined meaning of each character. For instance, the double-syllable simple word 蝴蝶 [butterfly] is composed of 蝴 and 蝶, but neither character alone holds meaning (Fu 2020). Another example would be the compound word 大家 [everybody]. 大 means big and 家 means family, but 大家 means everybody rather than big family (Fu 2020). Therefore, the number of characters known in Chinese is not necessarily analogous to the number of words the person knows.

Yan et al. (2020) compared the predictive power of character-based vocabulary size and word-based vocabulary size in children’s reading development in L1 Chinese. The importance of character-based vocabulary size in reading was found to decline with age, whereas the significance of word-based vocabulary size increased as children grew up. The development of a reliable word-based vocabulary size test is hence urgently needed. To date, researchers have mainly adopted two methods to measure L1 Chinese speakers’ word-based receptive vocabulary size. The first is the oral definition task, where participants are required to orally define the given word (e.g., Chen et al. 2018; Chen et al. 2019; Cheng and Wu 2017; Li et al. 2012; Yan et al. 2020). However, native speakers sometimes struggle to explain the meanings of words despite knowing what these words mean. Test scoring can also be troublesome in this case (Nation and Coxhead 2021). Furthermore, the oral definition task usually occurs during one-on-one interviews. Collecting data from large samples is accordingly time-consuming. Another common assessment tool is the Chinese version of the Peabody Picture Vocabulary Test (PPVT) (Siu et al. 2015; Zhang 2017), in which test takers listen to several words and choose corresponding pictures. Despite the good reliability and validity of this test, it has been normed with children between ages 2.5 and 7.5 and may be too easy for older learners (Guo et al. 2019).

In the absence of a widely accepted receptive vocabulary size test for Chinese learners, the present study aims to develop and trial a word-based receptive vocabulary size test called LexCH. This paper attempts to answer the following research question:

Does the LexCH produce valid and reliable results?

3.1 Materials

Word items for LexCH were sampled from the Lexicon of Common Words in Contemporary Chinese (LCWCC, 现代汉语常用词表) released by the Department of Language Information Management of Ministry of Education of the PRC (Project Group of Lexicon Common Words in Contemporary Chinese 2008). LCWCC provides 56,008 “commonly used words in Putonghua” (p.1), referring to modern Mandarin vocabulary that Chinese L1 speakers with secondary education frequently encounter and use in daily life. The majority of these words are double-syllable words. Some common abbreviations, idioms, and other fixed phrases that express integral concepts are collected in the LCWCC,^[1] but proper nouns and terms are not included.^[2] All words are sorted by frequency.

We divided the words in LCWCC into 25 frequency levels, with each level containing approximately 2,200 words. Following Nation and Beglar (2007), the sampling of the word items for the initial version of LexCH represented the various frequency levels of the words without bias towards any particular level. In this way, the test items would better represent all words in LCWCC. Ten words were randomly chosen from each level. In total, 250 words were extracted from LCWCC, constituting test items for the initial version of LexCH. In addition, 75 nonwords were made by randomly selecting two characters from LCWCC to form a nonword in Chinese. We did not include nonwords that looked or sounded like real Chinese words to avoid confusing test takers (Amenta et al. 2021). We then searched for these words in the Chinese National Corpus and Google search engine to ensure the words’ nonexistence. We also asked five Ph.D. candidates majoring in applied linguistics to confirm that these nonwords did not make sense. The initial version of LexCH featured 250 words and 75 nonwords. These terms were then mixed, listed in a random order, and presented on paper. The final version of LexCH was presented in Appendix.

3.2 Participants

Two groups of participants took part in the present study. The first group consisted of 251 students (137 men, 107 women, and 7 who chose not to identify by sex) from a junior high school and a high school in Guangdong, China. Both were middle-ranking schools in the same city. The participants were from five intact classes between grades 7 and 11. This group of learners took the initial version of LexCH. The second group, who took the final version of LexCH, comprised 229 students from another 5 intact classes at the same schools (133 men, 82 women, and 14 who chose not to identify by sex). In total, 480 students participated in the study. Their mean age was 15 years (SD = 1.467; range = 12–19; mode = 15). All participants spoke Chinese as their first language.

3.3 Procedure

The test was presented in a paper-and-pencil format. Signed consent forms were obtained from students’ parents before the participants took the test. Additionally, students were asked to provide their personal information (e.g., age, grade, and sex) before taking the test. Their final grades in Chinese language classes were obtained from their school records. Next, students were instructed to review the list of words in LexCH and put a tick “✓” in the box if they knew the word. Participants were asked not to put a tick if they were unsure of the word’s meaning. They were also told that the test results would not affect their course grades and that the test items included some nonwords (i.e., it would be pointless to put a tick for a word they did not know).

The first version of LexCH was piloted with a small group of learners similar to the target participants. The test instructions were changed to appear clear to the test takers. Then, the test papers were given to the teachers of the five classes who then distributed the test in class. To ensure that participants paid attention to the test instructions, the teachers read the directions aloud in front of the class before the test commenced. Furthermore, teachers ensured that all participants finished the test independently without referring to a dictionary. The test had no time limit, but all test takers (n = 251) completed the initial version of LexCH (containing 325 items) within 15 min.

Based on data from the first version of LexCH, a two-parameter logistic model was run to determine the difficulty and discriminative power of each item. Items covering a range of difficulty and with strong discriminative power were chosen as test items in the final version of LexCH (144 items), which was then given to a new group of test takers (n = 229).

3.4 Scoring

Following the Italian, Spanish, and French versions of the LexTALE test, the test scores were computed as follows:

This formula ensures that guessing behaviour is penalized. A test taker could obtain full points only if they answered “Yes” to all word items and “No” to all nonword items. If they responded “Yes” to all test items, they received zero points. Although there are more complicated ways to calculate scores, such as %Correct_av, Δm, and I_SDT (Lemhöfer and Broersma 2012), Brysbaert’s (2013) study showed that these approaches produced similar results. The above formula represents the simplest calculation, and is convenient for researchers and teachers in classroom settings.

3.5 Data analysis

After collecting data for the first version of LexCH, we analysed the data using point-biserial correlation and item response theory (IRT). We opted for IRT because it accounts for test takers’ performance level and items’ level of difficulty. A two-parameter model of IRT can provide information about each item’s difficulty and discriminative power; as such, we were able to select items of varying difficulty and strong discriminative power to constitute the final version of LexCH. After that, following the validity framework proposed by Messick (1989, 1995, we provided initial validation of LexCH based on data from the second group of participants, who took the final version of the measure.

4.1 Selecting items for LexCH

To evaluate the quality of each item in the initial LexCH, a point-biserial correlation between the response to each item and test takers’ overall accuracy was run separately on the 250 words and 75 nonwords. For a good-quality item, the point-biserial correlation coefficient should be positive, indicating that participants who claim to know this word are more likely to have higher overall accuracy. We first ran the correlation between the mean accuracy of each word item and the overall word item accuracy and then computed the correlation between the mean accuracy of each nonword item and the total accuracy on nonword items. The correlation was positive for all words, which met our expectations.

In addition, we performed IRT analysis on all test items because we wanted to select items that covered a range of difficulty levels and had strong discriminative power. IRT is a probabilistic model of how test takers respond to items based on the level of the underlying latent trait (i.e., vocabulary size in this study) (Desjardins and Bulut 2018). Within the IRT framework, items are characterized individually, and test characteristics are derived from the items. Characteristics in the two-parameter model include difficulty and discrimination; IRT assumes that these attributes are invariant across subgroups of examinees and across test administrations (Desjardins and Bulut 2018). Following Amenta and her colleagues (2021), we ran the two-parameter model on words and nonwords separately using the mirt R package (Chalmers 2012). First, we used the item fit function to examine the item fit statistics for item parameters from the two-parameter model. Ten words did not fit this model (p < 0.05) and were removed from further analysis. Next, we computed the item characteristics for each item.

Figure 1 demonstrates an example of the item characteristic curves for four word items. In this graph, the difficulty value refers to the point on the x-axis where the item response curve crosses the 0.5 probability value on the y-axis. The item discrimination parameter is represented by the steepness of the response curve in its middle section: the steeper the curve is, the better the item’s discriminative power. As seen in Figure 1, 再说 [besides] is easier than 疤痕 [scar] and 仿行 [replicate], and 溽热 [muggy] is most difficult. Additionally, 疤痕 has stronger discriminative power than 再说 and 仿行 is more discriminative than 溽热.

Figure 1:

Item response curves for four sample word items.

Based on each item’s difficulty parameter, we ordered the word items and then chose them based on their discriminative power by extracting those with higher values at intervals of roughly one-twenty-fourth of the difficulty range. We therefore ensured that the items covered a range of difficulty levels and had strong discriminative power. Next, we repeated the same procedure for nonword items. As previous LexTALE tests were shown to be too easy for native or highly proficient speakers (Amenta et al. 2021), as suggested by Brysbaert (2013), we increased the number of items. The final version of LexCH included 96 words and 48 nonwords. We then assessed the test’s reliability by computing the KR-20 coefficient, which reached 0.98 for all word items and 0.95 for all nonword items, reflecting strong reliability.

4.2 Validation

In this section, we aim to provide initial validity evidence for the final version of LexCH. We referred to Messick’s (1989, 1995 framework of validity, which includes six aspects of construct validity that “… function as general validity criteria or standards for all educational and psychological measurement” (Messick 1995, p.6): content, substantive, structural, generalizability, external, and consequential validity. The first five aspects of construct validity are addressed below.

4.2.3 Structural aspect of construct validity

Regarding the structural aspect of construct validity, Messick (1989) argued that “… scoring models should be rationally consistent with what is known about the structural relations inherent in behavioural manifestations of the construct in question” and that “… the degree of homogeneity in the test should be commensurate with this characteristic degree of homogeneity associated with the construct” (p.43). It was hypothesized that LexCH would show a high degree of psychometric unidimensionality for words and nonwords. Dimensionality can be tested via a scree plot (Desjardins and Bulut 2018). Plots were accordingly constructed for word and nonword items in this study. If only one major factor equal to the eigenvalues occurs immediately before the plot’s elbow, then the unidimensionality is good (Desjardins and Bulut 2018). Figure 2 indicates one major factor and a second potential factor, as the eigenvalues began to level off thereafter. Because the eigenvalues of the first factor far outweighed those of the second factor, only one major factor appeared before the elbow. Word items displayed good unidimensionality as a result. Figure 3 depicts only one major factor before the elbow; the nonwords in LexCH therefore demonstrated good unidimensionality as well.

Figure 2:

Scree plot of word item data (final version).

Figure 3:

Scree plot of nonword item data (final version).

4.2.5 External aspects of construct validity

Messick (1989) pointed out that the external aspect refers to “the extent to which the test’s relationship with other tests and nontest behaviours reflects the expected high, low, and interactive relations implied in the theory of the construct being assessed” (p.45). Although examining participants’ performance on LexCH in relation to other tests measuring written receptive vocabulary knowledge would be useful, no widely accepted written receptive vocabulary size test is presently available for L1 Chinese speakers. Therefore, the external aspect of LexCH’s construct validity was examined in relation to participants’ age and Chinese literacy.

Age is a significant predictor of vocabulary size. In a massive study of native Dutch speakers’ vocabulary size, Keuleers and colleagues (2015) identified age as the most important factor affecting vocabulary size among the examined variables. Similarly, Coxhead et al. (2015) explored factors contributing to the vocabulary size of native speakers of English in New Zealand secondary schools. They found age to be the only factor that entered their regression model (p < 0.0005, R ² = 0.146). As people age, they encounter more opportunities for language exposure, which creates productive conditions for vocabulary learning (Nation and Coxhead 2021). Therefore, test takers’ scores on LexCH were anticipated to rise with age.

As seen in Figure 4 and Table 2, the average LexCH scores (final version) rose with participants’ age,^[3] and the growth rate seemed to decelerate as age increased. This phenomenon is in line with Keuleers et al.’s (2015) study, which observed a slowdown in native vocabulary growth across age groups. We also performed a regression analysis to reveal the extent to which age could explain the variance in LexCH scores. The results highlighted that age was a significant predictor of scores (p < 0.001, F = 18.515, R ² = 0.264), explaining 26.4% of the total variance, higher than the figure (R ² = 0.146) identified by Coxhead et al. (2015). This discrepancy is probably due to the difference in the unit of measurement: LexCH measured Chinese vocabulary size based on Chinese words, whereas Coxhead et al. (2015) assessed vocabulary size by the number of word families known. Thus, it makes sense that a gap would emerge between the total variances in vocabulary sizes in the two studies and that the explanatory power of age would differ.

Figure 4:

Mean scores on LexCH by age.

Table 2:

Participants’ LexCH scores by age.

Age	M.	N	SD	Minimum	Maximum
13	52.82	39	14.440	23	84
14	61.05	42	14.461	22	84
15	69.06	48	12.274	24	85
16	71.53	43	10.534	36	87
17	72.90	39	9.577	36	89
Total	65.68	211	14.331	22	89

Another aspect of interest in this study was the relationship between LexCH scores and students’ Chinese literacy. We expected receptive vocabulary knowledge to correlate significantly with test takers’ Chinese literacy, especially reading comprehension, as suggested in prior work (e.g., Li et al. 2012; Ouellette 2006; Peng et al. 2020; Yan et al. 2020). Given the current lack of an available standardized language proficiency test for L1 Chinese speakers, we obtained students’ final scores in the Chinese subject from school records as a measure of their Chinese literacy. A Pearson correlation analysis was conducted between LexCH scores and final scores in the Chinese subject by grade, as Chinese examination papers varied by grade level. Table 3 shows that participants in all grades had positive correlations between LexCH scores and Chinese literacy.

Table 3:

Correlation coefficients between participants’ LexCH scores and Chinese literacy by grade.

Grade	N	Sig.	r
Seventh	40	0.006	0.431b
Eighth	54	0.000	0.677a
Ninth	34	0.012	0.424c
Tenth	48	0.000	0.575a
Eleventh	52	0.005	0.387b

1. ^a p < 0.001; ^b p < 0.01; ^a p < 0.05.

In sum, the relationships between LexCH scores and age and between LexCH scores and Chinese literacy reflected the expected relations in previous research. These patterns further substantiate the external aspect of construct validity.