Home Validation of a multiple-choice implicature test: insights from Chinese EFL learners’ cognitive processes
Article
Licensed
Unlicensed Requires Authentication

Validation of a multiple-choice implicature test: insights from Chinese EFL learners’ cognitive processes

  • Hsuan-Yu Tai , Chin-Ting Liu ORCID logo and Yuan-Shan Chen EMAIL logo
Published/Copyright: April 13, 2022
International Review of Applied Linguistics in Language Teaching
From the journal International Review of Applied Linguistics in Language Teaching Volume 61 Issue 4

Abstract

In the past decades, implicature has been recognized as an indispensable topic in L2 pragmatics research. While various instruments have been used to test implicature knowledge, scant research has verified their construct validity of implicature items by examining test takers’ cognitive processes while responding to them. In response to this research gap, Roever, Carsten. 2005. Testing ESL pragmatics: Development and validation of a web-based assessment battery. Frankfurt am Main, Germany: Peter Lang implicature test was scrutinized. Twelve L2 test takers with high and low proficiency levels were asked to verbalize their thoughts retrospectively. Content analyses were employed to examine whether their cognitive processes were (ir)relevant to what the test intended to elicit and how well their processes aligned with the test results. The findings revealed that relevant processes were associated with idiosyncratic implicature questions, while irrelevant processes were related to formulaic implicature ones. The high-proficiency test takers generally reported engaging in relevant processes, whereas the low-proficiency test takers reported using for the most part irrelevant processes. Moreover, test takers’ cognitive processes aligned with the test results on the whole. In other words, when the respondents’ cognitive processes corresponded to those the test designed to elicit, the test results were normally correct. One the other hand, when the cognitive processes deviated from the expected ones, the test results were generally incorrect. However, fourteen misalignments were identified and classified as irrelevant conceptual blending, contextual judgment, insufficient effort responding, intuition, and wild guessing. These misalignments were seen as construct irrelevant variance and pose potential threats to the construct validity.

Keywords: cognitive processes; multiple-choice implicature test; validation; validity
Corresponding author: Yuan-Shan Chen, Department of Applied English, National Chin-Yi University of Technology, Taichung, Taiwan, E-mail:

Funding source: The Ministry of Science and Technology, Taiwan, R.O.C

Award Identifier / Grant number: MOST 108-2410-H-167-005

Acknowledgments

We would like to thank Michael Rau for his kind assistance in this study.

  1. Research funding: This study was funded by the corresponding author’s research grants from the Ministry of Science and Technology, Taiwan, ROC (MOST 108-2410-H-167-005).

References

Anderson, Neil J., Lyle Bachman, Kyle Perkins & Andrew Cohen. 1991. An exploratory study into the construct validity of a reading comprehension test: Triangulation of data sources. Language Testing 8(1). 41–66. https://doi.org/10.1177/026553229100800104.Search in Google Scholar

Bachman, Lyle F. 1990. Fundamental considerations in language testing. Oxford: Oxford University Press.Search in Google Scholar

Bachman, Lyle F. & Adrian S. Palmer. 1996. Language testing in practice: Designing and developing useful language tests. Oxford: Oxford University Press.Search in Google Scholar

Bax, Stephen. 2013. The cognitive processing of candidates during reading tests: Evidence from eye-tracking. Language Testing 30(4). 441–465. https://doi.org/10.1177/0265532212473244.Search in Google Scholar

Bax, Stephen & Sathena Chan. 2019. Using eye-tracking research to investigate language test validity and design. System 83. 64–78. https://doi.org/10.1016/j.system.2019.01.007.Search in Google Scholar

Bouton, Lawrence F. 1988. A cross-cultural study of ability to interpret implicatures in English. World Englishes 7(2). 183–196. https://doi.org/10.1111/j.1467-971x.1988.tb00230.x.Search in Google Scholar

Bouton, Lawrence F. 1992. The interpretation of implicature in English by NNS: Does it come automatically—Without being explicitly taught? Pragmatics and Language Learning 3. 53–65.Search in Google Scholar

Bouton, Lawrence F. 1994. Can NNS skill in interpreting implicature in American English be improved through explicit instruction? – A pilot study. In Lawrence Bouton & Yamuna Kachru (eds.), Pragmatics and language learning, vol. 5, 88–109. Urbana-Champaign: University of Illinois, Division of English as an International Language.Search in Google Scholar

Brown, Penelope & Stephen C. Levinson. 1987. Politeness: Some universals in language usage. Cambridge: Cambridge University Press.10.1017/CBO9780511813085Search in Google Scholar

Chen, Yuan Shan & Ming-Fang Lin. 2021. Effects of peer collaboration on EFL learners’ comprehension of conversational implicatures. System 97. 102114. https://doi.org/10.1016/j.system.2020.102441.Search in Google Scholar

Conway, Morgan A., Richard A. Klein Brian O’Shea & Kate A. Ratliff. 2019. Can carelessness be captured? Assessing careless responding in attitudes toward novel stimuli. Social Cognition 37(5). 468–498. https://doi.org/10.1521/soco.2019.37.5.468.Search in Google Scholar

Crystal, David (ed.). 1997. The Cambridge encyclopedia of language, 2nd edn. New York: Cambridge University Press.Search in Google Scholar

Currie, Michael A. & Thanyapa Chiramanee. 2010. The effect of the multiple-choice item format on the measurement of knowledge of language structure. Language Testing 27(4). 471–491. https://doi.org/10.1177/0265532209356790.Search in Google Scholar

Davies, Wayne A. 2019. Implicature. The Stanford encyclopedia of philosophy. Stanford: Stanford University. https://plato.stanford.edu/entries/implicature/ (accessed 31 October 2021).Search in Google Scholar

DeKeyser, Robert M. 2000. The robustness of critical period effects in second language acquisition. Studies in Second Language Acquisition 22. 499–533. https://doi.org/10.1017/s0272263100004022.Search in Google Scholar

Ellis, Nick C. 2002. Frequency effects in language processing. A review with implications for theories of implicit and explicit language acquisition. Studies in Second Language Acquisition 24. 143–188. https://doi.org/10.1017/s0272263102002024.Search in Google Scholar

Elo, Satu & Helvi Kyngäs. 2008. The qualitative content analysis process. Journal of Advanced Nursing 62. 107–115. https://doi.org/10.1111/j.1365-2648.2007.04569.x.Search in Google Scholar

Farr, Roger, Robert Pritchard & Brian Smitten. 1990. A description of what happens when an examinee takes a multiple- choice reading comprehension test. Journal of Educational Measurement 27(3). 209–226. https://doi.org/10.1111/j.1745-3984.1990.tb00744.x.Search in Google Scholar

Fauconnier, Gilles & Mark Turner. 2002. The way we think: Conceptual blending and the mind’s hidden complexities. NY: Basic Books.Search in Google Scholar

Garcia, Paula. 2004. Pragmatic comprehension of high and low level language learners. The Electronic Journal for English as a Second Language 8(2). 1–15.Search in Google Scholar

Grice, Herbert Paul. 1975. Logic and conversation. In Peter Cole & Jerry L. Morgan (eds.), Syntax and semantics Vol. 3: Speech acts, 41–58. New York: Academic.10.1163/9789004368811_003Search in Google Scholar

Huang, Jason L., Mengqiao Liu & Nathan A. Bowling. 2014. Insufficient effort responding: Examining an insidious confound in survey data. Journal of Applied Psychology 100(3). 828–845. https://doi.org/10.1037/a0038510.Search in Google Scholar

Hudson, Thom, James Dean Brown & Emily Detmer. 1995. Developing prototypic measures of cross-cultural pragmatics. Honolulu: University of Hawai’i, Second Language Teaching and Curriculum Center.Search in Google Scholar

Hymes, Dell. 1967. Models of the interaction of language and social setting. Journal of Social Issues 23(2). 8–28. https://doi.org/10.1111/j.1540-4560.1967.tb00572.x.Search in Google Scholar

Hymes, Dell. 1972. On communicative competence. In John Bernard Pride & Janet Holmes (eds.), Sociolinguistics, 269–293. Middlesex, England: Penguin.Search in Google Scholar

Kane, Michael. 2013. The argument-based approach to validation. School Psychology Review 42(4). 448–457. https://doi.org/10.1080/02796015.2013.12087465.Search in Google Scholar

Kane, Michael. 2016. Explicating validity. Assessment in Education: Principles, Policy & Practice 23(2). 198–211. https://doi.org/10.1080/0969594x.2015.1060192.Search in Google Scholar

Kane, Michael T. 2006. Validation. In Robert L. Brennan (ed.), Educational measurement, 4th edn., 65–110. Westport, CT: American Council on Education and Praeger.Search in Google Scholar

Labben, Afef. 2016. Reconsidering the development of the discourse completion test in interlanguage pragmatics. Pragmatics 26(1). 69–91. https://doi.org/10.1075/prag.26.1.04lab.Search in Google Scholar

Leech, Geoffrey. 1983. Principles of pragmatics. London, England: Longman.Search in Google Scholar

Liu, Chin-Ting & Yuan-Shan Chen. 2021. On the relationship between intrinsic saliency and implicit learning of apologetic strategies: The case of Taiwanese EFL learners. European Journal of Investigation in Health, Psychology and Education 11(4). 1310–1319. https://doi.org/10.3390/ejihpe11040095.Search in Google Scholar

Liu, Jianda. 2007. Developing a pragmatic test for Chinese EFL learners. Language Testing 24(3). 391–415.10.1177/0265532207077206Search in Google Scholar

Lowewen, Shawn, Rosemary Erlam & Rod Ellis. 2009. The incidental acquisition of third person -s as implicit and explicit knowledge. In Rod Ellis, Shawn Loewen, Catherine Elder, Rosemary Erlam, Jenefer Philp & Hayo Reinders (eds.), Implicit and explicit knowledge in second language learning, testing and teaching, 262–280. Bristol, UK: Multilingual Matters.10.21832/9781847691767-013Search in Google Scholar

MacDonald, Maryellen C. 2013. How language production shapes language form and comprehension. Frontier in Psychology 4. 1–16. https://doi.org/10.3389/fpsyg.2013.00226.Search in Google Scholar

Meade, Adam W. & Bartholomew S. Craig. 2012. Identifying careless responses in survey data. Psychological Methods 17(3). 437–455. https://doi.org/10.1037/a0028085.Search in Google Scholar

Messick, Samuel. 1988. The once and future issues of validity: Assessing the meaning and consequences of measurement. In Howard Wainer & Henry I. Braun (eds.), Test validity, 33–45. Hillsdale, N. J.: L. Erlbaum Associates.Search in Google Scholar

Messick, Samuel. 1989. Validity. In Robert L. Linn (ed.), Educational measurement, 3rd edn., 13–103. New York: American Council on Education & Macmillan.Search in Google Scholar

Nevo, Nava. 1989. Test-taking strategies on a multiple-choice test of reading comprehension. Language Testing 6(2). 199–215. https://doi.org/10.1177/026553228900600206.Search in Google Scholar

Roever, Carsten. 2004. Difficulty and practicality in tests of interlanguage pragmatics. In Diana, Boxer and Andrew, D. Cohen (eds.), Studying speaking to inform second language learning, 283–301. Clevedon: Multilingual Matters.Search in Google Scholar

Roever, Carsten. 2005. Testing ESL pragmatics: Development and validation of a web-based assessment battery. Frankfurt am Main, Germany: Peter Lang.Search in Google Scholar

Roever, Carsten. 2006. Validation of a web-based test of ESL pragmalinguistics. Language Testing 23(2). 229–256. https://doi.org/10.1191/0265532206lt329oa.Search in Google Scholar

Roever, Carsten. 2007. DIF in the assessment of second language pragmatics. Language Assessment Quarterly 4(2). 165–189. https://doi.org/10.1080/15434300701375733.Search in Google Scholar

Roever, Carsten. 2009. Teaching and testing pragmatics. In Michael H. Long & Catherine J. Doughty (eds.), The handbook of language teaching, 560–577. Malden, MA: Wiley-Blackwell.10.1002/9781444315783.ch29Search in Google Scholar

Roever, Carsten. 2013. Testing implicature under operational conditions. In Steven J. Ross & Gabriele Kasper (eds.), Assessing second language pragmatics, 43–64. NY: Palgrave Macmillan.10.1057/9781137003522_2Search in Google Scholar

Roever, Carsten. 2014. Assessing pragmatics. In Antony John Kunnan (ed.), The companion to language assessment, 125–139. Malden, MA: John Wiley & Sons.10.1002/9781118411360.wbcla057Search in Google Scholar

Roever, Carsten. 2020. Assessment of pragmatics. In Carol A. Chapelle (ed.), The concise encyclopedia of applied linguistics, 87–95. Hoboken, NJ: Wiley Blackwell.Search in Google Scholar

Smith, Mark, Joel Breakstone & Wineburg Sam. 2019. History assessments of thinking: A validity study. Cognition and Instruction 37(1). 118–144. https://doi.org/10.1080/07370008.2018.1499646.Search in Google Scholar

Storey, Peter. 1997. Examining the test-taking process: A cognitive perspective on the discourse cloze test. Language Testing 14. 214–231. https://doi.org/10.1177/026553229701400205.Search in Google Scholar

Tada, Masao. 2005. Assessment of EFL pragmatic production and perception using video prompts. Philadelphia: Temple University Dissertation.Search in Google Scholar

Taguchi, Naoko. 2002. An application of relevance theory to the analysis of L2 interpretation processes: The comprehension of indirect replies. IRAL-International Review of Applied Linguistics in Language Teaching 40(2). 151–176. https://doi.org/10.1515/iral.2002.006.Search in Google Scholar

Taguchi, Naoko. 2005. Comprehending implied meaning in English as a foreign language. The Modern Language Journal 89(iv). 543–562. https://doi.org/10.1111/j.1540-4781.2005.00329.x.Search in Google Scholar

Taguchi, Naoko. 2011. The effect of L2 proficiency and study abroad experience on pragmatic comprehension. Language Learning 61(3). 904–939. https://doi.org/10.1111/j.1467-9922.2011.00633.x.Search in Google Scholar

Tai, Hsuan Yu & Yuan-Shan Chen. 2021. The effects of L2 proficiency on pragmatic comprehension and learner strategies. Education Sciences 11(4). 174. https://doi.org/10.3390/educsci11040174.Search in Google Scholar

Thomas, Jenny. 1983. Meaning in interaction: An introduction to pragmatics. London: Longman.Search in Google Scholar

Weir, Cyril J. 2005. Language testing and validation: An evidence-based approach. Basingstoke: Palgrave Macmillan.Search in Google Scholar

Winke, Paula, Shinhye Lee, Jieun Irene Ahn, Ina Choi, Yaqiong Cui & Hyung JoYoon. 2018. The cognitive validity of child English language tests: What young language learners and their native-speaking peers can reveal. TESOL Quarterly 52(2). 274–303. https://doi.org/10.1002/tesq.396.Search in Google Scholar

Wu, Yi’an. 1998. What do tests of listening comprehension test?- A retrospection study of EFL test-takers performing a multiple-choice task. Language Testing 15(1). 21–44.10.1177/026553229801500102Search in Google Scholar

Yamanaka, Janice E. 2003. Effects of proficiency and length of residence on the pragmatic comprehension of Japanese ESL learners. Second Language Studies 22(1). 107–175.Search in Google Scholar

Youn, Soo Jung & Nick Zhiwei Bi. 2019. Investigating test-takers’ strategy use in task-based L2 pragmatic speaking assessment. Intercultural Pragmatics 16(2). 185–218. https://doi.org/10.1515/ip-2019-0009.Search in Google Scholar
Received: 2020-09-12
Accepted: 2022-03-16
Published Online: 2022-04-13
Published in Print: 2023-11-27

© 2022 Walter de Gruyter GmbH, Berlin/Boston

Articles in the same Issue

  1. Frontmatter
  2. Research Articles
  3. Investigating the impact of task complexity on uptake and noticing of corrective feedback recasts
  4. Consequences of the comparative fallacy for the acquisition of grammatical aspect in Spanish
  5. Incorporating peer feedback in writing instruction: examining its effects on Chinese English-as-a-foreign-language (EFL) learners’ writing performance
  6. Listener engagement: the missing link in research on accented speech
  7. Enhancing English spatial prepositions acquisition among Spanish learners of English as L2 through an embodied approach
  8. Lexical and grammatical collocations in beginning and intermediate L2 argumentative essays: a bigram study
  9. When concept-based language instruction meets cognitive linguistics: teaching English phrasal verbs with up and out
  10. Validation of a multiple-choice implicature test: insights from Chinese EFL learners’ cognitive processes
  11. A longitudinal study of topic continuity in Chinese EFL learners’ written narratives
  12. Miscommunicated referent tracking in L2 English: a case-by-case analysis
  13. Rule-based or efficiency-driven processing of expletive there in English as a foreign language
  14. When are performance-approach goals more adaptive for Chinese EFL learners? It depends on their underlying reasons
  15. Teaching L2 Spanish idioms with semantic motivation: should this be done proactively or retroactively?
  16. Role of individual differences in incidental L2 vocabulary acquisition through listening to stories: metacognitive awareness and motivation
  17. Measuring and profiling Chinese secondary school English teachers’ language mindsets: an exploratory study of non-native teachers’ perceived L2 proficiency loss
  18. The role of working memory in the effects of models as a written corrective strategy
  19. Comparing motivational features between feedback givers and receivers in English speaking class
  20. Examining resilience in EFL contexts: a survey study of university students in China
  21. High school EFL teachers’ oral corrective feedback beliefs and practices, and the effects of lesson focus
  22. L3 acquisition of aspect: the influence of structural similarity, analytic L2 and general L3 proficiency
https://doi.org/10.1515/iral-2021-0190
Keywords for this article
cognitive processes; multiple-choice implicature test; validation; validity

Articles in the same Issue

  1. Frontmatter
  2. Research Articles
  3. Investigating the impact of task complexity on uptake and noticing of corrective feedback recasts
  4. Consequences of the comparative fallacy for the acquisition of grammatical aspect in Spanish
  5. Incorporating peer feedback in writing instruction: examining its effects on Chinese English-as-a-foreign-language (EFL) learners’ writing performance
  6. Listener engagement: the missing link in research on accented speech
  7. Enhancing English spatial prepositions acquisition among Spanish learners of English as L2 through an embodied approach
  8. Lexical and grammatical collocations in beginning and intermediate L2 argumentative essays: a bigram study
  9. When concept-based language instruction meets cognitive linguistics: teaching English phrasal verbs with up and out
  10. Validation of a multiple-choice implicature test: insights from Chinese EFL learners’ cognitive processes
  11. A longitudinal study of topic continuity in Chinese EFL learners’ written narratives
  12. Miscommunicated referent tracking in L2 English: a case-by-case analysis
  13. Rule-based or efficiency-driven processing of expletive there in English as a foreign language
  14. When are performance-approach goals more adaptive for Chinese EFL learners? It depends on their underlying reasons
  15. Teaching L2 Spanish idioms with semantic motivation: should this be done proactively or retroactively?
  16. Role of individual differences in incidental L2 vocabulary acquisition through listening to stories: metacognitive awareness and motivation
  17. Measuring and profiling Chinese secondary school English teachers’ language mindsets: an exploratory study of non-native teachers’ perceived L2 proficiency loss
  18. The role of working memory in the effects of models as a written corrective strategy
  19. Comparing motivational features between feedback givers and receivers in English speaking class
  20. Examining resilience in EFL contexts: a survey study of university students in China
  21. High school EFL teachers’ oral corrective feedback beliefs and practices, and the effects of lesson focus
  22. L3 acquisition of aspect: the influence of structural similarity, analytic L2 and general L3 proficiency
Sign up now to receive a 20% welcome discount
Institutional Access
How does access work?
Have an idea on how to improve our website?
Please write us.
© 2025 De Gruyter Brill
Downloaded on 20.11.2025 from https://www.degruyterbrill.com/document/doi/10.1515/iral-2021-0190/html?lang=en
Scroll to top button