Investigating test-takers’ strategy use in task-based L2 pragmatic speaking assessment
-
Soo Jung Youn
Soo Jung Youn is an Assistant Professor of Applied Linguistics at Northern Arizona University, USA. Her research interests include language testing and assessment, L2 pragmatics, interactional competence, task-based language teaching, and mixed methods research. Her research is recently published in
Language Testing, System, Applied Linguistics Review , andJournal of English for Academic Purposes .Nick Zhiwei Bi is an Associate Professor in Applied Linguistics and TESOL at the University of Shanghai for Science and Technology, China. His research interests focus on self-regulated learning and L2 learners’ strategic processing in language use and test-taking situations.
Abstract
This study explicates cognitive validity of task-based L2 pragmatic speaking assessment by examining reported strategy use of test takers at varying performance levels across different task types. Thirty university-level ESL learners completed four pragmatic speaking tasks that differ in the formality of pragmatic actions. Two trained raters scored the task-based pragmatic performances using analytical rating criteria and displayed a satisfactory level of consistency and accuracy in scoring the performances. The test-takers’ retrospective reports were transcribed and analyzed to develop a valid coding scheme that consists of cognitive, metacognitive, and pragmatic strategies. An association between the test-takers’ pragmatic performances scored by the trained raters and their reported strategy use was examined. The higher-ability test takers utilized diverse strategies more frequently, ranging from varied pragmatic strategies to strategies specifically related to managing task demands, compared to the lower-ability test takers. Further, the test takers utilized distinct types of strategies appropriate to handling unique pragmatic task situations and complexities involved in each pragmatic assessment task. These findings explain how the test takers cognitively interacted with the assessment tasks and what strategies potentially led to successful pragmatic performances. The implications of examining pragmatic strategy use were discussed in terms of advancing practices of teaching and assessing L2 pragmatics.
About the authors
Soo Jung Youn is an Assistant Professor of Applied Linguistics at Northern Arizona University, USA. Her research interests include language testing and assessment, L2 pragmatics, interactional competence, task-based language teaching, and mixed methods research. Her research is recently published in Language Testing, System, Applied Linguistics Review, and Journal of English for Academic Purposes.
Nick Zhiwei Bi is an Associate Professor in Applied Linguistics and TESOL at the University of Shanghai for Science and Technology, China. His research interests focus on self-regulated learning and L2 learners’ strategic processing in language use and test-taking situations.
References
Bachman, Lyle & Adrian Palmer. 1996. Language testing in practice. Oxford: Oxford University Press.Suche in Google Scholar
Bachman, Lyle & Adrian Palmer. 2010. Language assessment in practice. Oxford: Oxford University Press.Suche in Google Scholar
Barkaoui, Khaled, Lindsay Brooks, Merrill Swain & Sharon Lapkin. 2013. Test-takers’ strategic behaviors in independent and integrated speaking tasks. Applied Linguistics 34(3). 304–324.10.1093/applin/ams046Suche in Google Scholar
Bax, Stephen & Cyril Weir. 2012. Investigating learners’ cognitive processes during a computer-based CAE reading test. Cambridge Research Notes, Cambridge ESOL 47. 3–14.Suche in Google Scholar
Bi, Nick. Z. 2015. A multi-dimensional examination of strategic competence in the lexico-grammar test performance of Chinese EFL university students. Australia: University of Sydney Unpublished Ph.D. dissertation.Suche in Google Scholar
Bi, Nick. Z. 2016. The impact of strategic processing on lexico-grammatical test performance. In MaryAnn Christison & Nick Saville (eds.), Advancing the field of language assessment. Studies in language testing (SiLT), vol. 42. 229–248. Cambridge: Cambridge University Press.Suche in Google Scholar
Bond, Trevor G. & Christine M. Fox. 2007. Applying the Rasch model. New York: Routledge.Suche in Google Scholar
Bowles, Melissa A. 2010. The think-aloud controversy in second language research. New York: Routledge.10.4324/9780203856338Suche in Google Scholar
Brown, Penelope & Stephen C. Levinson. 1987. Politeness: Some universals in language usage. Cambridge: Cambridge University Press.10.1017/CBO9780511813085Suche in Google Scholar
Cohen, Andrew D. 1998. Strategies in learning and using a second language. Harlow, England: Longman.Suche in Google Scholar
Cohen, Andrew D. 2005. Strategies for learning and performing L2 speech acts. Intercultural Pragmatics 2. 275–301.10.1515/iprg.2005.2.3.275Suche in Google Scholar
Cohen, Andrew D. 2009. The coming of age of research on test-taking strategies. Language Assessment Quarterly 3. 307–331.10.2307/j.ctt1ckpccf.11Suche in Google Scholar
Cohen, Andrew D. & Thomas A. Upton. 2007. ‘I want to go back to the text’: Response strategies on the reading subtest of the new TOEFL®. Language Testing 24(2). 209–250.10.1177/0265532207076364Suche in Google Scholar
Cohen, Andrew D. & Isobel K. Wang. 2018. Fluctuations in the functions of language learner strategies. System 74. 169–182.10.1016/j.system.2018.03.011Suche in Google Scholar
Eckes, Thomas. 2011. Introduction to many-Facet Rasch measurement. Frankfurt: Peter Lang.10.3726/978-3-653-04844-5Suche in Google Scholar
Grabowski, Kirby. 2009. Investigating the construct validity of a test designed to measure grammatical and pragmatic knowledge in the context of speaking. Columbia University Unpublished Ph.D. dissertation.Suche in Google Scholar
Green, John M. & Rebecca Oxford. 1995. A closer look at learning strategies, L2 proficiency, and gender. TESOL Quarterly 29(2). 261–297.10.2307/3587625Suche in Google Scholar
Hudson, Thom D, Emily Detmer & James D. Brown. 1992. A framework for testing cross-cultural pragmatics (Technical Report #2). Honolulu, HI: University of Hawai’i, Second Language Teaching and Curriculum Center.Suche in Google Scholar
Hudson, Thom D, Emily Detmer & James D. Brown. 1995. Developing prototype measures of cross-cultural pragmatics (Technical Report #7). Honolulu, HI: University of Hawai’i, Second Language Teaching and Curriculum Center.Suche in Google Scholar
Ishihara, Noriko & Andrew D. Cohen. 2010. Teaching and learning pragmatics: Where language and culture meet. Harlow, UK: Pearson.Suche in Google Scholar
Kasper, Gabriele. 2006. Speech acts in interaction: Towards discursive pragmatics. In J. Kathleen Bardovi-Harlig, César Félix-Brasdefer & Alwiya S. Omar (eds.), Pragmatics and language learning, vol. 11, 281–314. HI: University of Hawai’i at Manoa, National Foreign Language Resource Center.Suche in Google Scholar
Kasper, Gabriele & Kenneth R. Rose. 2002. Pragmatic development in a second language. Oxford: Blackwell.Suche in Google Scholar
Khalifa, Hanan & Cyril Weir. 2009. Examining reading: Research and practice in assessing second language reading, Studies in Language Testing, vol. 29. Cambridge: UCLES/Cambridge University Press.Suche in Google Scholar
Leech, Geoffrey. 1983. Principles of pragmatics. London: Longman.Suche in Google Scholar
Linacre, John M. 1994. Sample size and item calibration stability. Rasch Measurement Transactions 7. 328.Suche in Google Scholar
Linacre, John M. 2012a. Facets Rasch measurement computer program, version 3.70.1. Chicago: Winsteps.com.Suche in Google Scholar
Linacre, John M. 2012b. Winsteps Rasch Tutorial 2. Retrieved from Winsteps & Facets Rasch Software website: https://www.winsteps.com/a/winsteps-tutorial-2.pdfSuche in Google Scholar
Liu, Jianda. 2007. Developing a pragmatics test for Chinese EFL learners. Language Testing 24(3). 391–415.10.1177/0265532207077206Suche in Google Scholar
Long, Mike. 2015. Second language acquisition and task-based language teaching. Oxford: Wiley Blackwell.Suche in Google Scholar
McNamara, Tim. 1996. Measuring second language performance. New York: Addison Wesley Longman.Suche in Google Scholar
McNamara, Tim & Ute Knoch. 2012. The Rasch wars: The emergence of Rasch measurement in language testing. Language Testing 29(4). 555–576.10.1177/0265532211430367Suche in Google Scholar
Norris, John M. 2009. Task-based teaching and testing. In Mike H. Long & Catherine Doughty (eds.), Handbook of second and foreign language teaching. Cambridge: Blackwell.Suche in Google Scholar
Pan, Yi-Ching & Yo In’nami. 2015. Relationship between strategy use, listening proficiency level, task type, and scores in an L2 listening test. The Canadian Journal of Applied Linguistics 18(2). 45–77.Suche in Google Scholar
Phakiti, Aek. 2003. A closer look at the relationship of cognitive and metacognitive strategy use to EFL reading achievement test performance. Language Testing 20(1). 26–56.10.1191/0265532203lt243oaSuche in Google Scholar
Phakiti, Aek. 2008. Strategic competence as a fourth-order factor model: A structural equation modeling approach. Language Assessment Quarterly 5(1). 20–42.10.1080/15434300701533596Suche in Google Scholar
Plakans, Lia. 2009. Discourse synthesis in integrated second language writing assessment. Language Testing 26(4). 561–587.10.1177/0265532209340192Suche in Google Scholar
Purpura, James E. 1998. Investigating the effects of strategy use and second language test performance with high- and low-ability groups: A structural equation modeling approach. Language Testing 15(3). 333–379.10.1177/026553229801500303Suche in Google Scholar
Purpura, James E. 2013. Cognition and language assessment. In Anthony J. Kunnan (ed.), The companion to language assessment, 1452–1476. Oxford, UK: Wiley-Blackwell.10.1002/9781118411360.wbcla150Suche in Google Scholar
Roever, Carsten. 2005. Testing ESL pragmatics. Frankfurt, Germany: Peter Lang.10.3726/978-3-653-04780-6Suche in Google Scholar
Roever, Carsten. 2011. Testing of second language pragmatics: Past and future. Language Testing 28(4). 463–481.10.1177/0265532210394633Suche in Google Scholar
Roever, Carsten, Catriona Fraser & Catherine Elder. 2014. Testing ESL sociopragmatics: Development and validation of a web-based test battery. Frankfurt am Main: Peter Lang.10.3726/978-3-653-04598-7Suche in Google Scholar
Ross, Steven & Gabriele Kasper (eds). 2013. Assessing second language pragmatics. Basingstoke, UK: Palgrave Macmillan.10.1057/9781137003522Suche in Google Scholar
Rupp, Andre A., Tracy Ferne & Hyeran Choi. 2006. How assessing reading comprehension with multiple-choice questions shapes the construct: A cognitive processing perspective. Language Testing 23(4). 441–474.10.1191/0265532206lt337oaSuche in Google Scholar
Searle, John. 1976. A classification of illocutionary acts. Language in Society 5(1). 1–23.10.1017/S0047404500006837Suche in Google Scholar
Shively, Rachel L. 2010. From the virtual world to the real world: A model of pragmatic instruction for study abroad. Foreign Language Annals 43(1). 105–137.10.1111/j.1944-9720.2010.01063.xSuche in Google Scholar
Song, Xiaomei. 2005. Language learner strategy use and English proficiency on the Michigan English language assessment battery. Spaan Fellow Working Papers in Second or Foreign Language Assessment 3. 1–25.10.1207/s15434311laq0303_2Suche in Google Scholar
Taguchi, Naoko. 2011. Teaching pragmatics: Trends and issues. Annual Review of Applied Linguistics 31. 289–310.10.1017/S0267190511000018Suche in Google Scholar
Taguchi, Naoko. 2015. Instructed pragmatics at a glance: Where instructional studies were, are, and should be going. Language Teaching 48(1). 1–50.10.1017/S0261444814000263Suche in Google Scholar
Taguchi, Naoko & Carsten Roever. 2017. Second language pragmatics. Oxford: Oxford University Press.Suche in Google Scholar
Taguchi, Naoko & YouJin Kim (eds). 2018. Task-based approaches to teaching and assessing pragmatics. Amsterdam: John Benjamins.10.1075/tblt.10Suche in Google Scholar
Vandergrift, Larry. 2003. Orchestrating strategy use: Toward a model of the skilled second language listener. Language Learning 53(3). 463–496.10.1111/1467-9922.00232Suche in Google Scholar
Weir, Cyril. 2005. Language testing and validation: An evidence based approach. Basingstoke: Palgrave Macmillan.10.1057/9780230514577Suche in Google Scholar
Xi, Xiaoming & Pam Mollaun. 2011. Using raters from India to score a large-scale speaking test. Language Learning 61(4). 1222–1255.10.1111/j.1467-9922.2011.00667.xSuche in Google Scholar
Yang, Hui-Chun & Lia Plakans. 2012. Second language writers’ strategy use and performance on an integrated reading-listening-writing task. TESOL Quarterly 46(1). 80–103.10.1002/tesq.6Suche in Google Scholar
Youn, Soo Jung. 2015. Validity argument for assessing L2 pragmatics in interaction using mixed methods. Language Testing 32(2). 199–225.10.1177/0265532214557113Suche in Google Scholar
Youn, Soo Jung. 2018a. Task design and validity evidence for assessment of L2 pragmatics in interaction. In Naoko Taguchi & You-Jin Kim (eds.), Task-based approaches to teaching and assessing pragmatics, 217–246. Amsterdam: John Benjamins.10.1075/tblt.10.09youSuche in Google Scholar
Youn, Soo Jung. 2018b. Task-based needs analysis of L2 pragmatics in an EAP context. Journal of English for Academic Purposes 36. 86–98.10.1016/j.jeap.2018.10.005Suche in Google Scholar
Appendix A: Example role-play assessment task and rating criteria (Youn 2015)
Situation: You have an appointment with a professor Morgan Brown today to ask for a recommendation letter for a scholarship for international students from your department and to ask a few questions about a course project. Your professor is meeting with you outside of the office hour since you have a class during the office hour. Now you’re about to visit your professor. You just enter to a professor’s room.
Task: You will receive role-play cards that describe what you’re going to tell your professor. Please have a conversation with your professor naturally.
For undergraduate participants: This professor teaches Economy 101 that you’re taking this semester.
For graduate participants: This professor is one of the faculty members in your department. Although he/she is not your advisor, you’ve known this professor for about 1 year and you’re currently taking a course from this professor.
A role-play card for requesting a recommendation letter
| Jessie | Professor |
|---|---|
| 1. After greeting, ask for a recommendation letter for the department scholarship that you will apply. The letter is due in one week. | |
| Respond to the request. Inform the student that you will write a letter and ask when the due date is, if the student doesn’t tell you. | |
| Inform students that you have a conference next week and you’re leaving tonight. Tell him/her that you will do your best to submit the letter by the deadline, but ask the student if the letter can be submitted a bit late. | |
| 2. Respond to what the professor says and tell the professor that you will check with your department office. | |
| 3. Inform the professor of two options of providing a letter, hard copy or electronic submissions through a website. Ask for the professor’s preference. | Prefer an electronic submission as you will be traveling. |
Rating Criteria for Role-play with a Professor (Youn 2015)
| Score | Contents Delivery | Language Use | Sensitivity to Situation | Engaging with Interaction | Turn Organization |
|---|---|---|---|---|---|
| 3 | – Clear, concise, fluent (esp. speech act delivery) – Smooth topic initiations with appropriate transitional markers and clear intonations (i.e. smooth turn initiation) #1: a letter request & letter submission option #2: need for a meeting & decide a time #3: respond to professor’s request & explain a situation | – Pragmatically appropriate linguistic expressions (bi-clausal, conditional: I was wondering if, I don’t think I can; modal verbs, would, could, might) – Good control of grammar and vocabulary that doesn’t obscure meaning | – Consistent evidence of awareness and sensitivity to situations exists in contents or tone #1: request along with explanations about the scholarship; acknowledge a short letter due #2: explanations for a meeting request #3: handle a face-threatening refusal with acceptable reasons or accept a request | – A next turn shows understandings of a previous turn throughout the interaction (i.e. shared understanding) – Evidence of engaging with conversation exists (e.g. clarification questions, backchannel, acknowledgement tokens) Note: Non-verbal cues also serve as acknowledgement, so no need to heavily rely on the amount of discourse markers. | – Complete adjacency pairs (e.g. question & answer, granting a request & thank) – Interactionally fluid without awkward pauses or abrupt overlap Note: Interactionally meaningful pauses include those before refusal and between disagreements |
| 2 | – Generally smooth, but occasionally unclear, or unnecessarily wordy – Abrupt topic initiation (in terms of contents) – Unclear transitional cues (e.g. unclear intonation and stress) | – Able to use modal verbs in mono-clausal (e.g. could, can, might), but doesn’t or inconsistently use complex structures for pragmatic meaning – Linguistic expressions are occasionally inaccurate and a bit limited that sometimes obscure meaning | – Inconsistent evidence of awareness and sensitivity to situations (e.g. explain the letter request, but not acknowledge a short letter due) | – Some evidence of engaging with the conversation, but not consistent – A next turn doesn’t sometimes show understandings of previous turns | – Some turns are delayed and a next turn is absent in adjacency pairs (e.g. absence of answers & thank) – Sometimes abruptly cut off a previous turn |
| 1 | – Delivery is choppy, fragmented, and minimal (due to lack of language competence) | – Expressions sound abrupt, direct, or not polite enough (e.g. I need, I want, I can’t) – Linguistic expressions are inaccurate and quite limited that obscure meaning | – Little evidence of situational sensitivity (e.g. not acknowledge a short letter due, insist turning in the letter on time, lack of explanations for refusal) | – Noticeable absence of discourse markers – Evidence of not achieving a shared understanding | – Noticeably abrupt overlap or no pauses between disagreements and refusal – Noticeably long pauses or noticeable cutoff between turns |
Appendix B: Example reported strategies
| Strategies | Examples |
|---|---|
| Comprehending task | Like I know how to answer this question and I know how to make a long response of this question and give good reason, because I understood everything in it. |
| Translation | I think, I translate. I use translate |
| Linking prior knowledge or experience | And that way I can associate this with what’s going on in my previous semester. Like we also have this kind of meeting, we need to talk about the time, work on our project. |
| Recalling appropriate L2 linguistic knowledge | I guess vocabulary if we have I don’t know vocabulary like vocabulary knowledge more than it will be much easier considering that the other students because I’m international student so sometimes maybe if we pick the wrong type of vocabulary then it will have the different meaning. |
| Putting yourself in the task situation | I tried Tom to put himself in the shoes of the professor. Imagine that I’m a really busy person, which really needs precise subject lines to subdivide when open the email |
| Setting goals | I’m trying to give the professor the most logical reason to avoid having the presentation on the next week instead of scheduled two weeks from now. |
| Planning | Okay so for this one, I’m thinking first I need to show my appreciation first, thank you for meeting me, even this is not your office hour and then state the purpose. |
| Evaluating their own performances | I thought of strategies, but I could not come up with one that would make it less direct as I did. I think that I wasn’t really understanding what I was responding to the professor. |
| Evaluating the execution of plans | I felt like I should have given an option, which I thought I would based on further conversation. Like who knew it’s gonna be so short. I wanted to ask like you might ask some other classmates, you know. |
| Evaluating emotional status | The second one, because you’re declining the professor’s request, maybe the difficulty is you’re feeling guilty or you feel like “Oh I’m so sorry that I cannot do this”. So it’s actually more so this feeling. |
| Assessing task-related situations | I saw Sunday was empty but at the same time I thought okay, if I choose Sunday for meeting with my friend, maybe it’s not what you thought I should do, and that’s why I tried to negotiate more to make the task longer and then I tried to imagine that I have a lot of homework and then the one thing I understood that I can’t like wait so long because the assignment is on Friday and that’s why I decide to do it on Monday and Tuesday. |
| Pragmatic awareness: Target-language culture | This is the professor giving a request. Yeah, you don’t turn down a request from a professor. Otherwise your study will be, I don’t know what happen with them. |
| Pragmatic awareness: One’s own culture | In my culture, well maybe in Asian culture if you speak softly to your elders or somebody that you respect more that means that you are honoring him or her more. So yeah, that’s the cultural part of this. |
| Situation-related sociopragmatic strategies | So in hope that the professor recognizes that I’m not taking the exam lightly nor I’m thinking the presentation lightly. But I’m just dividing my time. |
| Situation-related pragmalinguistic strategies | And also when the professor said she will be busy this week, I used the hedging like “that would be totally okay too” if you cannot so. So it’s like both more formal language and also more hedging. |
| Situation-related interactional strategies | But if I can’t do that, I usually provide with another option. And I thought that you would say, “Oh I see”. And I would say, “Oh you know you can probably ask some other students”. |
© 2019 Walter de Gruyter GmbH, Berlin/Boston
Artikel in diesem Heft
- Frontmatter
- The pragmatics of amicable interstate communication
- A relational account of communication on the basis of slips of the tongue
- Investigating test-takers’ strategy use in task-based L2 pragmatic speaking assessment
- How to put yourself in someone else’s shoes: The role of point-of-view in the interpretation of mixed messages
- Yan Huang: The Oxford Handbook of Pragmatics
- Cliff Goddard Zhengdao Ye: “Happiness” and “Pain” across Languages and Cultures
Artikel in diesem Heft
- Frontmatter
- The pragmatics of amicable interstate communication
- A relational account of communication on the basis of slips of the tongue
- Investigating test-takers’ strategy use in task-based L2 pragmatic speaking assessment
- How to put yourself in someone else’s shoes: The role of point-of-view in the interpretation of mixed messages
- Yan Huang: The Oxford Handbook of Pragmatics
- Cliff Goddard Zhengdao Ye: “Happiness” and “Pain” across Languages and Cultures
