Evaluating a collaborative and responsive project to develop language assessment literacy

Claudia Harsch; Sibylle Seyferth; Salomé Villa Larenas

doi:10.1515/cercles-2021-2020

Article

Evaluating a collaborative and responsive project to develop language assessment literacy

Claudia Harsch , Sibylle Seyferth and Salomé Villa Larenas

Published/Copyright: November 11, 2021

Published by

Become an author with De Gruyter Brill

Submit Manuscript Author Information Explore this Subject

From the journal Language Learning in Higher Education Volume 11 Issue 2

Abstract

We report on research conducted during a language assessment literacy (LAL) project set up for employees at a university language centre. Teachers, students and an external accreditation body requested a revision of the end-of-course exams to increase comparability, transparency and alignment to course aims. We used a collaborative approach where stakeholders involved in test development, quality control, use and interpretation brought their experiences, skills, knowledge and needs to the table, following Taylor’s (2013) view of differentiated LAL needs for different stakeholders. The project involves teachers (varying in numbers), five course/assessment coordinators and three researchers with a background in assessment (principal researcher, two research assistants, i.e. the authors). LAL training is needs-based, participant-driven, and evaluated in regular intervals. In response to these participant evaluations, delivery, content and focus of the training are adjusted. We present insights from five evaluation points gained during the first 3 years. We found that the collaborative and responsive LAL project was perceived as useful and effective by the participants, leading to learning gains and positive changes in assessment, teaching and coordination practices. Our insights contribute to a better understanding of the conditions that are conducive to designing and delivering effective LAL development projects.

Keywords: collaborative professional development; higher education; language assessment literacy; project evaluation; responsive design

Corresponding author: Claudia Harsch, University of Bremen, Bremen, Germany, E-mail: harsch@uni-bremen.de

Acknowledgements

We would like to thank all our participants for sharing their experiences, insights and concerns with us. Our thanks also go to student assistants Tim-Simon Holtgrefe, Vanessa Pohlmann and Michaela Puschmann for assisting with the transcriptions and with supporting the analyses of some of the questionnaire data. We also thank the anonymous reviewers for their invaluable feedback that helped improve this manuscript; all remaining errors are ours.

Appendices

Appendix A: Overview of the training project (adjustments have been highlighted in grey).

Time	Events and topics	Training and content	Collaborative Activities organised by Teachers
Term 1, 10/2015–02/2016	Teacher conference (01/2016): Input on constructive alignment of learning outcomes, teaching objectives, assessment goals, taking the CEFR as framework.
Start of LAL project
Term 2, 04–07/2016	Teacher conference (06/2017): Input on validity of end-of-course exams/achievement tests.	Seminar introduction to assessment; covering basic terms and concepts, assessing the four skills, constructs, task design. Workshops (05/2016) benchmarking writing – matching university grading to CEFR levels, deciding on pass/fail marks with reference to CEFR levels. (07/2016) writing: Construct, approaches, test specifications.	Pedagogical lunches on writing task design, comparing existing tasks and scripts.
Questionnaire 1 (10/2016), n = 18
Term 1, 10/2016–02/2017	Teacher conference (02/2017): Collaborative finalisation of test specifications writing. Collaborative decision on founding working groups.	Seminar developing communicative teaching and assessment tasks; covering aspects of task design, task complexity, engagement with communicative partners.	Working group writing: Developing test specifications and designing tasks. Pedagogical lunches on assessment criteria, rating scales, checklists.
Out of term	International symposium (02/2017): “How can we bridge the gaps in the CEFR in HE language education?” (testing strand, workshops)	Workshop (03/2017) Diagnosing reading (Huhta).
Interview 1 (03/2017), n = 14
Term 2, 04–07/2017	Teacher conference (06/2017): Report on interview results, collaborative decision on next steps and further training topics; update results from WG. Collaborative decision: Introduction of training days (out of term time).	Seminar assessment tasks that target relevant language use situations; focus on assessment tasks and differences to learning tasks, covering aspects of task design, difficulty-determining characteristics, operationalizing features of relevant target language use situations in tasks. Guest lecture (07/2017) characteristics of academic English (Siepmann).	Language-specific working groups Developing writing assessment checklists. Benchmarking writing scripts and tasks. Pedagogical lunches on comparing existing writing tasks.
Out of term		Training day (09/2017) Speaking: Construct, task design and test specifications.
Term 1, 10/17–02/18	Teacher conference (02/2018): Report on questionnaire results, collaborative decision on next steps and further training/focus in WG and symposion (03/2019). Input on summative/formative assessment, focus on how to treat errors. EALTA SIG meeting (10/2017): “Integrated assessment” (guest WS by Atta Gebril).	Training session (01/2018) Calibration writing assessment. Guest workshop (01/2018) Assessing reading - know what, how and why (Sebastiani)	Working groups speaking: Developing test specifications. Pedagogical lunches on feasibility of administering speaking tests, possible formats.
Questionnaire 2 (11/2017), n = 11
Out of term		2 Training days (02/2018) Rater training writing. Finalisation test specifications speaking.
Term 2, 04–07/2018	Teacher conference (06/2018): Presenting PPT for LTRC (“benchmarking writing”). Collaborative decision on next steps and future focus of training and WG. Preparing launch of webpage for assessment group.	[Skip seminar based on feedback in questionnaire 2]	Working groups developing speaking tasks, checklists for assessing speaking. Pedagogical lunches on assessment criteria for speaking, comparing existing approaches.
Interview 2 (05/2018), n = 9
Out of term		2 Training days (09/2018) Speaking tasks and administration. Reading construct, task design, test specifications.
Term 1, 10/2018–02/2019	Teacher conference (01/2019): Update on progress WG. Presenting PPT for symposium (“Transparency of exams across languages”). Collaborative decision on further steps based on feedback from interviews.		Working groups continuing developing speaking tasks, checklists for assessing speaking. Developing test specifications reading. Pedagogical lunches on practices assessing reading, task design, feasible approaches to assessing speaking.
[no questionnaire due to reported fatigue]
Out of term	International symposium (03/2019) “Language teaching in HE – rethinking the language learner” (testing strand, workshops collaboratively prepared and delivered by staff). Launch of homepage for assessment group.	(No training days due to teacher participation in symposium)
Term 2, 04–07/2019	Teacher conference (06/2019): Update on progress WG, collaborative decision on further steps: Collaboratively preparing assessment workshop (11/2019); introducing teacher projects.	Training session (04/2019) Listening construct, task design, test specifications.	Working groups fine-tuning checklists for assessing speaking. Finalising test specifications for reading. Developing test specifications for listening. Pedagogical lunches Task design reading and listening.
Out of term		2 Training days (09/2019) Finalising checklists on speaking. Listening tasks: Formats and development. Assessing language in use: constructs and tasks.
Questionnaire 3 (09/2019), n = 17
Term 1, 10/19–02/20	EALTA SIG meeting (11/2019) “assessment in academic contexts”. Assessment workshop (11/2019) on classroom and proficiency assessment in HE. Teacher conference (01/2020) Present results from questionnaire 3, overview over all EV rounds and general insights, discussion of impact of training and collaborative work. Collaborative decision: Abolish teacher conference, integrate with training days, prepare and deliver collaboratively.		Working groups developing listening tasks. Developing test specifications language in use. Pedagogical lunches benchmarking speaking. Comparing existent approaches to testing language in use. Comparing tasks for reading and listening across languages. Teacher projects: Comparing different levels of autonomy when delivering listening tasks. Recording administration of speaking tasks.
Out of term		2 Training days (02/2020), collaboratively developed and delivered by staff, working in language groups Rater training on speaking, language-specific task development on listening

Appendix B: Language Assessment Literacy – Interview two Guide.

[The aim is to find out how the project is perceived (usefulness, effects), whether assessment practices have changed and what influence the seminars, workshops, working groups have had (impact).]

I02_01: Did you participate in the following events (last year) at the Languages Centre?

	I02_01.1 training (seminars, workshops) on testing and assessment	I02_01.2 the working groups (test specifications/rating checklists)
Yes - which ones, why?
No - why not?

I02_02: Which activities contributed to learning more about the following areas of testing and assessment? [hand out printed table, multiple ticks possible:]

Areas	Training	Working groups
Constructive alignment (learning, teaching and assessment)
Developing test specification
Developing test items and tasks
Evaluating the quality of language tests
Developing rating scales/checklists
Using rating scales/checklists
Assessing productive performances
Assessment in the language classroom
Understanding cut-scores
Preparing learners for tests
Reliability
Validity
Authenticity
Practicability
Washback
Test development cycle

I02_03: What was useful in the training events?
I02_04: What was the least useful?
I02_05: What do you take away (from the training activities) for your classroom?
I02_06: Did the training activities lead to any changes in your assessment practice?
- I02_06.1: If so, what have you changed and why?
- I02_06.2: If not, why not?
I02_07: In your eyes, do these changes lead to better teaching, learning or assessment?
- I02_07.1: For coordinators: Do these changes lead to improvement or advantages for your coordination work? Why (not)?
- I02_07.2: Did you get any feedback from the teachers on the effects of the project?
I01_08: Given that we can’t change the institutional constraints, within these constraints, what would you (still) need to be able to do?
- I02_08.1: participate in the training events?
- I02_08.2: optimally perform assessment during the lesson and at the end of the course (what we call formative and summative assessment)?
I02_09: For which areas of assessment and evaluation would you like more information or special support?
I02_10: Would you like to say something else about testing, classroom assessment and/or the training events?

References

Baker, Beverly & Caroline Riches. 2018. The development of EFL examinations in Haiti: Collaboration and language assessment literacy development. Language Testing 35(4). 557–581. https://doi.org/10.1177/0265532217716732.Search in Google Scholar

Brunfaut, Tineke & Luke Harding. 2018. Teachers setting the assessment (literacy) agenda: A case study of a teacher-led national test development project in Luxembourg. In Daniel Xerri & Patricia Vella Briffa (eds.), Teacher involvement in high stakes language testing, 155–172. Cham: Springer.10.1007/978-3-319-77177-9_9Search in Google Scholar

Cohen, Louis, Laurence Manion & Keith Morrison. 2011. Research methods in education, 7th edn. London: Routledge.Search in Google Scholar

Cordingley, Philippa, Miranda Bell, Sarah Thomason & Antonia Firth. 2005. The impact of collaborative CPD on classroom teaching and learning - Review: What do teacher impact data tell us about collaborative CPD? London: EPPI-Centre, Social Science research Unit, Institute of Education, University of London. http://eppi.ioe.ac.uk/cms/Default.aspx?tabid=139.Search in Google Scholar

Council of Europe. 2001. A Common European Framework of Reference for language learning and teaching. Cambridge: Cambridge University Press.Search in Google Scholar

Daniel Xerri & Patricia Vella Briffa (eds.). 2018. Teacher involvement in high stakes language testing. Cham: Springer.10.1007/978-3-319-77177-9Search in Google Scholar

Davies, John McE. & Todd H. McKay (eds.). 2018. A guide to useful evaluation of language programs. Washington DC: Georgetown University Press.10.2307/j.ctvvngs5Search in Google Scholar

Fulcher, Glenn. 2012. Assessment literacy for the language classroom. Language Assessment Quarterly 9(2). 113–132. https://doi.org/10.1080/15434303.2011.642041.Search in Google Scholar

Harsch, Claudia & Sibylle Seyferth. 2020. Marrying achievement with proficiency - developing and validating a local CEFR-based writing checklist. Assessing Writing 43. https://doi.org/10.1016/j.asw.2019.100433.Search in Google Scholar

Inbar-Lourie, Ofra. 2008. Constructing a language assessment knowledge base: A focus on language assessment courses. Language Testing 25(3). 385–402. https://doi.org/10.1177/0265532208090158.Search in Google Scholar

Johnston, Bill. 2009. Collaborative teacher development. In Anne Burns & Jack C. Richards (eds.), Cambridge guide to second language teacher education, 241–249. Cambridge: Cambridge University Press.10.4324/9780203878033Search in Google Scholar

Kennedy, Aileen. 2011. Collaborative continuing professional development (CPD) for teachers in Scotland: Aspirations, opportunities and barriers. European Journal of Teacher Education 34(1). 25–41. https://doi.org/10.1080/02619768.2010.534980.Search in Google Scholar

Kremmel, Benjamin, Kathrin Eberharter, Franz Holzknecht & Eva Konrad. 2018. Fostering language assessment literacy through teacher involvement in high-stakes test development. In Daniel Xerri & Patricia Vella Briffa (eds.), Teacher involvement in high stakes language testing, 173–194. Cham: Springer.10.1007/978-3-319-77177-9_10Search in Google Scholar

Levi, Tziona & Ofra Inbar-Lourie. 2020. Assessment literacy or language assessment literacy: Learning from the teachers. Language Assessment Quarterly 2(17). 168–182. https://doi.org/10.1080/15434303.2019.1692347.Search in Google Scholar

Lynch, Brian K. 2003. Language assessment and programme evaluation. Edinburgh: Edinburgh University Press.Search in Google Scholar

Norris, John. 2016. Language program evaluation. The Modern Language Journal 100. 169–189. https://doi.org/10.1111/modl.12307.Search in Google Scholar

Seyferth, Sibylle, Antonella Lavagno, Paola Kucera & Claudia Harsch. 2018. Der GER als Grundlage zur Entwicklung sprachbergreifender Testspezifikationen [The CEFR as basis for developing test specifications acorss several languages]. In Anikó Brandt, Astrid Buschmann-Göbels & Claudia Harsch (eds.), Der Gemeinsame Europäische Referenzrahmen für Sprachen und seine Adaption im Hochschulkontext. Fremdsprachen in Lehre und Forschung Bd. 51, 118–144. Bochum: AKS Verlag.Search in Google Scholar

Stake, Robert E. 1975. Evaluating the Arts in Education. Columbus: OH: Merrill.Search in Google Scholar

Stiggins, Richard J. 1999. Evaluating classroom assessment training in teacher education programs. Educational Measurement: Issues and Practice 18. 23–27. https://doi.org/10.1111/j.1745-3992.1999.tb00004.x.Search in Google Scholar

Taylor, Lynda. 2013. Communicating the theory, practice and principles of language testing to test stakeholders: Some reflections. Language Testing 30(3). 403–412. https://doi.org/10.1177/0265532213480338.Search in Google Scholar

Vogt, Karin & Dina Tsagari. 2014. Assessment literacy of foreign language teachers: Findings of a European study. Language Assessment Quarterly 11(4). 374–402. https://doi.org/10.1080/15434303.2014.960046.Search in Google Scholar

Xu, Yueting & Gavin T.L. Brown. 2016. Teacher assessment literacy in practice: A reconceptualization. Teaching and Teacher Education 58. 149–162. https://doi.org/10.1016/j.tate.2016.05.010.Search in Google Scholar

Received: 2020-07-26

Accepted: 2021-03-13

Published Online: 2021-11-11

Published in Print: 2021-10-26

You are currently not able to access this content.

Articles in the same Issue

https://doi.org/10.1515/cercles-2021-2020

Keywords for this article

collaborative professional development; higher education; language assessment literacy; project evaluation; responsive design