Modelling Differences in Multiword Expression Use Across Proficiency Levels in a Test of Spoken English

2.1.1 Language Processing and Language Tests

The main reason why MWEs are so important is that they aid language processing, which, in turn, means that they are related to multiple aspects of language, not just vocabulary. MWEs are thought to be processed as single chunks, which reduces memory load (e.g., Martinez and Schmitt 2012), and therefore more knowledge of MWEs correlates with faster processing speed for both L1 and L2 language users (e.g., Wray 2002). Use of MWEs not only affects general language processing speed (e.g., Pawley and Syder 1983) and the encoding and decoding of texts (e.g., Poulsen 2005), but also affects fluency in reading (e.g., Wray 2002) and in speaking (e.g., Xu 2018).

MWE use is especially crucial in speaking (and writing to a lesser extent), where the use of unexpected word combinations may throw off listeners (and readers). This is because when a listener hears (or a reader reads) the first word of a MWE, they use their vocabulary (and grammar) knowledge to start narrowing down the range of expected upcoming words. For example, when hearing the word rancid, the expected category of nouns that it modifies is some foods (e.g., butter, oil, milk), so it would be unexpected if the speaker produced paint instead. This is related to fluency and cohesion/coherence aspect of language often used in assessing speaking tasks; not only has MWE use been found to increase speaking fluency (e.g., Xu 2018), but it also covers some cohesive features (Hyland and Tse 2004).

MWE use can be linked to other aspects of language assessed in speaking tasks, with the most obvious being the vocabulary aspect, often referred to as lexical resource or vocabulary/lexical range and accuracy, where MWE use is related to precise and idiomatic vocabulary use and depth of vocabulary knowledge (e.g., Qian 2002; Read 1998). MWEs are also related to grammatical range and accuracy, as correct use of MWEs requires knowledge of the relationship between words, including the typical grammatical patterns or categories that go with them (i.e., colligation, Xiao and McEnery 2006; Scrivener 1994). MWE use can also increase overall comprehensibility and have a positive effect on multiple suprasegmental pronunciation features like pausing at logical junctions and speech rate (Isaacs et al. 2015). Note that these pronunciation features can also be considered relevant to fluency and coherence/cohesion; this is in line with Seedhouse et al.’s (2014) observation that there is a possible halo effect of accurate and appropriate use of MWEs.

2.2.1 Model of Phrase Density

The proposed model of phrase density assumes that level of MWE group acquisition tends to be positively correlated with language proficiency. Note that MWE group acquisition refers to the acquisition of a particular type of MWE (e.g., verb-noun collocation) in general, rather than specific/individual MWEs, which have their own properties that highly affect their ease of acquisition (e.g., the more frequent and more concrete play game is usually acquired before the less frequent and abstract engender interest), meaning that in this paper, the model reflects phrase density of particular phrase groups overall, rather than that of individual MWEs, though it is likely that acquisition of individual MWEs would follow a similar phrase density pattern.

It was hypothesized that differences in phrase density produced by speakers and writers would be roughly in an upside-down ‘U’ shape with increasing level of MWE group acquisition, tapering off steadily, as shown in Figure 1. This is because when proficiency is too low, speakers do not have enough word or MWE knowledge to produce many MWEs. As proficiency increases, there is a corresponding increase in MWE acquisition and therefore MWE use, though this is not straightforward. While some of the less proficient learners may avoid using MWEs altogether (Koya 2003), it is more common that (intermediate level) learners overuse highly frequent MWEs but underuse less frequent ones that strongly cohere where compared to an L1 baseline (e.g., Durrant and Schmitt 2009; Granger and Bestgen 2014); this tends to disappear for highly proficient learners, who start to use less common MWEs that strongly cohere (Granger and Bestgen 2014).

Figure 1:

Hypothesized phrase density pattern by level of MWE acquisition in general. Note. Figure is not to scale.

Figure 2 shows how the hypothesized model can be applied in predicting patterns of phrase density use across proficiency levels in any particular speaking or writing test (or potentially even task[s]). The dashed lines show the range of amount of MWE group acquisition for a hypothetical test-taker group, with the left line representing the lowest level of acquisition, corresponding to lower language proficiency, and the right line representing the highest level of acquisition, corresponding to higher language proficiency. In other words, it is assumed that the lowest proficiency group of target test takers would fall on the left of the range and the highest proficiency group would fall on the right.

Figure 2:

Hypothesized phrase density pattern across language proficiency for phrase groups of varying difficulty levels for a hypothetical test-taker group.

For a phrase group that is difficult to acquire for the target test-taker group (left-most model), phrase density increases as acquisition/proficiency levels increase; while the middle model shows an increase followed by a decrease, phrase difficulty can vary and produce slightly different patterns. For instance, a medium-difficult phrase group may largely show an increase across acquisition/proficiency levels but show smaller increases at the higher acquisition/proficiency levels (imagine the dashed lines falling from the left of the peak to the middle of the peak), or a medium-difficult phrase group may show very little variation across proficiency levels (imagine the dashed lines being much closer together and centred around the peak).

Note that if the test were to include extremely low proficiency levels, the left dashed line would sit at or very close to the left-most part of the model, no matter the difficulty level of the phrase group, as test takers with extremely low proficiency levels would not be able to produce many instances, if at all, of any phrase group. On the other hand, if the test were to only include extremely highly proficient speakers, it is expected that there would be little change across proficiency levels due to high levels of MWE group acquisition (i.e., dashed lines would fall along the rightmost part of the model), though it is highly unlikely that there would be such a test, as it is arguably not very useful to try to differentiate between those at the highest proficiency levels.

Figure 3 shows how this model can be applied to the mixed and contradictory findings from three recent longitudinal corpus-based studies on MWE production. Garner and Crossley (2018) looked at bigram frequency in spoken production (i.e., all bigrams considered together, with no specific MWE group[s] considered) of L2 learners of English from various L1 backgrounds, either enrolled in undergraduate and graduate-level courses at a US university or attending an intensive English program at the same university. They found an increase in bigram frequency over time, with this increase being larger for less proficient speakers. The dashed lines on the left model in Figure 3 thus covers a range where there is greater increase in phrase density as proficiency increases at the lower proficiency/acquisition end, and less increase at the higher proficiency/acquisition end.

Figure 3:

Estimated MWE acquisition ranges of participants matching observed phrase density patterns from three recent longitudinal corpus-based studies of MWE production.

Siyanova-Chanturia (2015) and Siyanova-Chanturia and Spina (2020) were methodologically comparable with similar test-taker groups, so findings were placed on a single model (Figure 3, right model). Siyanova-Chanturia (2015) explored noun-adjective collocations in beginner level L1 Chinese learners of Italian and found that written phrase density was comparable across three time periods (across 21 weeks), though the average strength of association of the collocations produced increased. Siyanova-Chanturia and Spina (2020) similarly explored noun-adjective collocations in L1 Chinese learners of Italian, with participants at three different proficiency levels (CEFR level A1, A2, and B1). They found that phrase density decreased across time (six months), that phrase density was higher for A1 learners than for A2 and B1 learners, and that the phrase density decrease was stronger for A1 than for the other two proficiency groups. The dashed lines on the right model in Figure 3 therefore show a narrower range of test-taker proficiency levels for Siyanova-Chanturia’s (2015) study, and centres around the peak of the model, reflecting the finding that phrase density did not significantly change. The lines for Siyanova-Chanturia and Spina’s (2020) study are wider apart, showing that it covered a wider range of proficiency/acquisition levels, and somewhat reflects the decrease in phrase density over time, with less decrease at the higher proficiency levels. Note that the hypothesized model of phrase density is currently not to scale, and further improvements can be made to tweak its shape to better fit further research findings.

2.2.2 Model of MWE Group Difficulty

Turning back to the issue of determining the relative positions of each phrase group on the hypothesized model, there are several factors to consider. This paper focused on five variables related to the part of speech (PoS) of the words making up the MWEs:

1. Word class: lexical/open/form class (nouns, verbs, adjectives, adverbs) versus function/closed/structure class (pronouns, modal verbs, determiners, conjunctions, prepositions, interjection). Function words were expected to be more difficult due to them carrying less meaning, noting that light verbs (vs lexical verbs) can be argued to be closed class. It has been found that language learners acquire vocabulary more quickly if a word is more concrete (vs abstract), imagable (i.e., ease of constructing a mental image), and meaningful (i.e., level of association with other words) (e.g., Salsbury, Crossley, and McNamara 2011), which applies more to words from a lexical word class.

2. Simplicity of use:

   1. Position: either having a set position (e.g., adjectives modify nouns) or having multiple possible positions (e.g., adverbs modify verbs, adjectives, other adverbs, whole sentences) such that having a set position is easier, with those that have a set position but are used optionally (e.g., the conjunction that) being in the middle. An example is that the multiple positions for adverbs in English has been pointed out as a reason for their difficulty (e.g., Rutledge and Fitton 2015).

   2. Functions: either one function or multiple functions (e.g., modal verbs do not only indicate ability, necessity, possibility, permission, but are also related to the pragmatic principle of politeness and serve as the principal means of expressing hedging in English academic discourse), such that having one function is easier. For instance, Bensaid (2015) noted that although English learners can master declarative of modals, they find the pragmatic uses of modals very difficult, especially since these uses are underrepresented in English language textbooks.

3. Frequency (e.g., nouns are more frequently used than adverbs), especially as they appear in the corpus in question, where higher frequency reflects more ease. While there are several frequency effects (token vs type, absolute vs relative, e.g., Ambridge et al. 2015), the consensus is that with all else being equal, the more often a learner hears/reads a word, the more likely they are to learn it (e.g., Read 2000).

4. L1 comparability: same or similar use in the L1 versus quite different use or does not exist in the L1, such that more similarity between the L1 and the target language leads to easier acquisition. L1-L2 congruency (i.e., word-for-word overlap in L1-L2 form-meaning connection) of specific MWEs has been found to be a good predictor of MWE knowledge (e.g., Nguyen and Webb 2017) and non-congruency has been found to be an obstacle to learning (e.g., reviewed in Boers and Webb 2018); a similar effect of L1-L2 differences can be expected for parts of speech.

In other words, the model consisted of two functions, with larger numbers corresponding to easier acquisition: (1) Predicted word ease (for any given PoS) = word class + position + function + frequency (in the corpus of test performances) + L1 comparability, and (2) Predicted MWE group ease = predicted ease of word 1 × predicted ease of word 2. The two functions were kept as simple as possible; as there was no clear reason to weigh any of the variables differently, they were given equal weighting, with values ranging from 0 to 1. Note that the values for each variable should be mostly self-explanatory and differences in rating were made only in clearly more difficult cases. This means that maximum predicted word ease for constituent words is 5, and maximum MWE group ease is 5 × 5 = 25.

3.1.1 Impetus for the Test

In Japan, English communicative competence is lacking, as are speaking skills in general. According to the English Proficiency Survey to Improve English Education conducted by the Ministry of Education, Culture, Sports, Science and Technology (MEXT) in 2017, 33.6 % of Japanese high school seniors were at CEFR A2 level or above in listening, 33.5 % in reading, and 19.7 % in writing, but only 12.9 % in speaking. Therefore, one of the two compulsory English courses for first-year undergraduates at the Kyoto Institute of Technology focuses on communication and output.

For language tests to be at their most effective, they should be reflective of local needs, student populations, and be connected to the language curriculum (Bernhardt, Rivera, and Kamil 2004); hence locally developed tests are likely to be more effective than existing products (Dimova, Yan, and Ginther 2020; O’Sullivan 2019). Therefore, the decision was made to develop an original university-wide speaking test to assess the communication course’s efficacy, rather than use existing commercialized tests such as TOEFL, IELTS or Versant, which were determined to be expensive, assess language skills too advanced for Japanese students, and not necessarily match the university’s curriculum and educational content. The aim was also to motivate students towards studying by providing an end goal: doing well on the KIT Speaking Test.

3.1.2 Speaking Construct

One major reason why existing commercialized tests were inappropriate for the KIT Speaking Test is that their speaking constructs were all oriented towards native speaker norms, while English education at the Kyoto Institute of Technology incorporates the concept of English as a lingua franca (ELF; Jenkins 2007; Seidlhofer 2013). Instead of adhering to these norms, various skills and knowledge are taught so that students can creatively use their linguistic resources to achieve tasks and convey opinions as an ELF user.

Consequently, the KIT Speaking Test focuses on ELF communication, with the scoring focusing on task achievement (80 %) and task delivery (i.e., fluency, 20 %) rather than on the proximity of pronunciation and grammar to native speakers. This emphasis on ELF and heavy weighting on fluency means that MWE use is especially pertinent to the KIT Speaking Test; as reviewed above, accurate and appropriate MWE use is linked to faster processing speed and increased fluency. Note that the ELF-focus of the KIT Speaking Test, with the assumption that interlocutors familiar with ELF would be more accepting of less conventional MWEs, meant that a very small number of unconventional MWEs were identified and accepted in this study, such as blue light instead of green light when it came to traffic lights.

3.1.3 KIT Speaking Test Corpus

The KIT Speaking Test corpus (KISTEC, n.d.) was created because analysis of learner corpora can be of great assistance in building on and improving language teaching and assessment methods (Granger 1998) and there are currently very few spoken English learner corpora with L1 Japanese learners. Although the corpus is still in need of minor corrections, it is now freely available on https://kitstcorpus.jp/. The KISTEC complements the NICT Japanese Learners of English (NICT JLE) Corpus (Izumi, Uchimoto, and Isahara 2004) and the International Corpus Network of Asian Learners of English (ICNALE) (Ishikawa 2023).

Corpus construction was supervised by one of the co-authors and his colleagues. The audio responses were automatically transcribed by Microsoft’s Video Inder, a speech-to-text system, manually corrected, and given 16 different tags to represent their speech features, such as fillers, repetitions, and self-corrections. A work manual was created, and the transcription process was monitored to maximize consistency. Metadata such as examinee attributes, test scores, and the test questions that elicited responses are provided.

The KISTEC contained contained data from 574 examinees, comprising a total of 290,454 words (M = 506.02 words per test taker). Approximately 97 % of the examinees were Japanese and 3 % were international students, including students from China, Korea, and Malaysia. A total of 72 % of the examinees stated that they had studied English in Japan because they had no experience of living abroad.

A full score on the KIT Speaking Test is 100; test takers recorded a mean score of 48.22 (SD = 10.45, Min = 21, Max = 90). Their mean Test of English for International Communication (TOEIC) listening and reading score was 563.54 (SD = 133.15, Min = 195, Max = 985). The correlation coefficients between the TOEIC and KIT Speaking Test were 0.59 for correlation with the total score, 0.56 with the listening score, and 0.54 with the reading score.

The KIT Speaking Test scores were well distributed, with most learners considered basic users of English, around the A1 and A2 levels of the Common European Framework of Reference for Languages (CEFR). Therefore, while the four proficiency levels referred to in this paper are labelled low (score ≤ 39), intermediate (score 40–49), high-intermediate (score 50–59) and advanced (score ≥ 60), these labels are relative.

3.3.1 Predicted Word Ease

Table 2 shows how the five variables were summed to calculate the predicted ease score for the PoS of constituent words of the MWE groups included in this paper. To calculate predicted verb-noun collocation ease for the KIT Speaking Test, for instance, the predicted ease score of verbs (5) is multiplied by the predicted ease score of nouns (5) to give 25.

Remember that the any lower rating (denoting increased difficulty) for functions and L1 comparability were made only in clearly more difficult cases, while ratings are straightforward for word class, position, and KISTEC frequency. This is why for L1 comparability, it was determined that English prepositions were not too different from Japanese postpositions (which have a wider range of uses than in English, such as subject marker が), as they are similar in that they link (pro)nouns to other words. On the other hand, the optional that conjunction was also given a rating of 0, as there is no similar word in Japanese. For instance, the equivalent of the man (that) Tom saw is トムが見た男 (see Example 1). Also note that relative clauses come before the noun (phrase) in question in Japanese, rather than after, as is the case for English.

Similarly, only modal verbs were considered different enough in Japanese to warrant a function rating of 0. While some modal verbs are used similarly to indicate ability, necessity, possibility, and permission, they are usually encoded in the verb itself rather than as an auxiliary verb, such as the equivalent of can (in the sense of ability to do something), where 話す (dictionary form to speak) becomes 話せる when can is encoded in the verb:

At other times, very different constructions are used. For example, one way of constructing the equivalent of must is a double negative conditional, such as must speak 話さなければならない (see Example 3), where the negative form of 話す (hanas+u _{dictionary form}) is 話さない (hanas+anai _negation) and negative form of なる(nar+u _{dictionary form}; to be(come)/get) is ならない (nar+anai _negation), resulting in the literal meaning ‘if not speak, not be(come)’. Of course, this construction is learned as/considered a single chunk, +なければならない (+nakerebanaranai), rather than reconstructed every time it is used. The other ways to construct must are also more complex than a simple encoding of the verb.

3.3.2 Part 1: Creation of Prediction Model

Part 1 of the study focused on the prediction model for the KIT Speaking Test, which entailed a comparison of the predicted MWE group ease scores with the observed phrase density patterns for a set of MWE groups, to determine cut scores for different patterns. Table 3 lists the MWE groups used in Part 1, with examples of each.

Note that the MWE groups selected here were the most frequent ones that also usually appear as bigrams, to ensure a more accurate representation of MWE use patterns. For instance, noun-preposition collocations (used in Part 1) rarely have words in between the node word and the collocate, while verb-noun collocations (used in Part 2) often have at least one word in between (e.g., for solve + problem, ‘solve the/some/her problem[s]’). Four were grammatical MWEs (ModVV, AdjPrep, NPrep, PhV/VPrep) and four were lexical (AdvAdj, VAdv, CompW, AdjN), to cover a range of difficulty levels, as grammatical ones tend to be more difficult.

3.3.3 Part 2: Application of Prediction Model

The second part of the study served as a check of the cut scores set in Part 1, to see if the predicted phrase density patterns reflected observed patterns for a further set of MWE groups (see Table 4). Similarly, they were balanced in terms of having two grammatical MWEs (Vthat, PrepN) and two lexical MWEs (VAdj, VN).