A Difficulty-Informed Approach to Developing Language Assessment Literacy for Classroom Purposes

1 Introduction

Over the past two decades, language assessment literacy (LAL) has developed into a key focus within mainstream research in the field of language testing and assessment. Several models and heuristics have been proposed since the beginning of the millennium identifying some of the core components of LAL (Brindley, 2001; Davies, 2008; Fulcher, 2012; Inbar-Lourie, 2008). After the initial period with its predominantly componential perspective, the emphasis moved towards stakeholder groups and their differential needs. The central question was not so much what constitutes LAL in general but what type of assessment-related knowledge and skills specific stakeholder groups need in their contexts (Baker, 2016; Deygers & Malone, 2019; O’Loughlin, 2013; Pill & Harding, 2013; Taylor, 2013). Special attention has been paid to the needs of language teachers as one of the primary stakeholder groups (Berry et al., 2017; Gan & Lam, 2020; Vogt & Tsagari, 2014; Xu & Brown, 2017).

While the debate continues about what exactly constitutes LAL for language teachers, the literature has recently begun to focus more strongly on LAL development, levels of LAL, and learning trajectories. The new key questions are how LAL develops both in individual teachers and within specific groups of teachers, how they progress from lower to higher levels of proficiency, and how LAL programs are implemented. For institutions offering pre-service teacher education, in particular, it would be interesting to know which aspects of language assessment are relatively easy for students and may thus be acquired or developed earlier, and which aspects are relatively difficult and thus acquired or developed later. Such information about the difficulty of certain aspects of the construct could help teacher educators to customize their programs.

Extending the developmental perspective provided by previous research (Berger et al., 2023), this study sheds further light on the question as to how easy or difficult certain aspects of language assessment are for pre-service teachers of English. The focus is on differences in difficulty in the context of secondary-level teacher education in Austria. Using multi-facet Rasch analysis as an analytic tool, the study generated empirical difficulty estimates of assessment-related abilities, knowledge, and understanding (AKUs), based on post-hoc content referencing of a LAL theory test used in the teacher education program at the Department of English and American Studies, University of Vienna, Austria. Difficulty is thus construed as the probability of pre-service teachers providing a correct answer to an item representing an assessment-related AKU.

I begin this article by reviewing some of the literature on LAL development, with particular emphasis on language teachers. I then go on to describe the dataset used in this study, as well as the procedures, the main instrument, and the analysis. After presenting the findings, especially in relation to classroom-based assessment, I discuss some implications for LAL development in teacher education programs.

2 Developing Language Assessment Literacy in Teacher Education

With teachers spending a substantial amount of time diagnosing their learners’ strengths and weaknesses, tracking their achievement, monitoring their progress, and certifying their proficiency, assessment is an essential field of teacher activity. It is well documented that assessment and feedback are crucial factors in student learning (Black & Wiliam, 2010). Professional competence in language testing and assessment is therefore an essential requisite for all language teachers. Over the past two decades, there have been several attempts to determine what knowledge and skills language teachers require for assessing their students’ language competence. In an early approach, Davies (2008) emphasized the importance of skills (i.e., expertise in item writing, statistics, and test analysis), knowledge (i.e., language description and measurement), and principles (i.e., questions of validity, ethics, and professionalism). Inbar-Lourie (2008) also distinguished three dimensions for the content of LAL courses, the “what” (i.e., the description of the trait), the “how” (i.e., the assessment method and process), and the “why” (i.e., the reason and rationale for the assessment). In addition to the components of test design and development, large-scale standardized testing, classroom testing and washback, and validity and reliability, Fulcher (2012) emphasized more strongly than previous approaches the importance of understanding the context and social dimensions of language assessment.

What these approaches have in common is that they acknowledge the importance for language teachers to understand the relationships between testing, assessment, teaching, and learning, but they differ in the relative importance they accord to these areas. Over time, there has been a shift from a focus on testing to assessment and teaching, and, more recently, to one on learning. Fulcher (2021), for example, extended his earlier definition of LAL from the perspective of learning-oriented assessment, emphasizing that the primary aim is to harness assessment for the purpose of enhancing learning. Linking learning-oriented assessment to validity theory, he contended that the validity of learning-oriented assessment hinges on its potential to change individual learners. Accordingly, the main concern of LAL in the framework of learning-oriented assessment is to equip teachers with what they “need to know to harness assessment in the service of change” (Fulcher, 2021, p. 35). Drawing on Hamp-Lyons (2017), he suggested seven elements to a theory of LAL for learning-oriented assessment, or practical skills, including task design for effective learning, self- and peer assessment, timely feedback, effective teacher questioning, scaffolding of performance, lesson planning and classroom management for reflection, and management of affective impact on learners.

While the precise requirements for teachers are still under debate, recent discussions have focused more closely on LAL development. Although language testing and assessment is increasingly being anchored in second language teacher education around the globe, teachers often have an insufficient degree of LAL (Vogt & Tsagari, 2014; Xu & Brown, 2017), inadequate training for their future needs (Lam, 2015; Lee & Coniam, 2013), or misconceptions about language assessment (Berry et al., 2017). It is in this context, between the poles of increasing significance and insufficient levels of LAL, that meta-reflection about LAL development has gathered momentum. The literature providing a developmental perspective encompasses at least four interrelated aspects: (1) theoretical LAL levels, (2) empirical LAL progressions, (3) longitudinal LAL development, and (4) effective LAL pedagogies.

As regards the first aspect, the focus on stakeholder groups has brought with it a growing emphasis not just on the different types of knowledge/skills needed but also on different degrees of knowledge/skills depending on the stakeholders’ depth of involvement in certain aspects of language assessment. Several theoretical profiles and hierarchies have been suggested. While Fulcher’s (2012) earlier model essentially implied a hierarchy, with practical knowledge being the bedrock and principles and contexts featuring at higher levels, Taylor (2013) explicitly addressed the question of “differential assessment literacy” and LAL levels (p. 409). Drawing on the four levels of LAL suggested by Pill and Harding (2013), she illustrated the differential needs for test writers, university administrators, professional language testers, and classroom teachers, with the latter being hypothesized to require “multidimensional literacy” (i.e., the highest level of literacy) in relation to language pedagogy; “procedural and conceptual literacy” in relation to technical skills, local practices, sociocultural values, and personal beliefs/attitudes; but only “functional literacy” in terms of knowledge of theory, scores and decision making, and principles and concepts (Taylor, 2013, p. 410). Continuing in Taylor’s footsteps, Kremmel and Harding (2020) set out to empirically validate the hypothesized profiles by asking different stakeholder groups how knowledgeable or skilled they think they need to be within their professional roles in relation to a number of aspects of language assessment. While Kremmel and Harding also used a vertical scale with different levels, ranging from “not knowledgeable/skilled at all” to “extremely knowledgeable/skilled,” their main interest was not so much the question of levels but empirical dimensions of LAL.

Secondly, besides broad theoretical levels, the developmental view comprises the question of empirical LAL progressions which are granular enough to be useful for teacher education purposes, such as designing teaching-learning sequences for LAL courses, diagnosing students’ strengths and weaknesses, or providing focused feedback. While the work outlined above helpfully draws attention to the issue of the breadth and depth of assessment knowledge for different stakeholders, the level distinctions are mainly theoretical, and they are not really detailed enough to be operationally useful for professional development purposes. This concept of a detailed vertical progression was taken up in Berger et al.’s (2023) investigation of the perceived level of difficulty of a range of assessment-related AKUs. Using our adaptation of Kremmel and Harding’s (2020) Language Assessment Literacy Survey, for which we reformulated the original questionnaire items into statements of ability (I can ...), knowledge (I know ...), and understanding (I understand ...), we asked pre-service teachers of English in Austria to indicate on a five-point scale how well they thought they could do, know, or understand what is expressed in the statements after attending a one-semester assessment course. Scaling these data by means of multi-facet Rasch measurement, we obtained a continuum of difficulty and ability in relation to these statements. A qualitative content check of this scale helped us to extract salient features of LAL at five levels and condense the calibrations into five level descriptions, which we termed, in ascending order, emergent LAL, operational LAL, generative LAL, technical LAL, and instructional LAL. The same methodology was replicated in a follow-up study involving in-service teachers of English at all types of secondary-level schools throughout Austria (Berger & Heaney, 2022).

A third aspect of the developmental perspective is longitudinal LAL development. As Taylor (2013) admitted, not much is known yet about how LAL grows over time both in individual stakeholders and across stakeholder groups. There is hardly any research to date investigating how novices in language assessment might progress from a low level of proficiency to a higher level of proficiency, not just in terms of how they expand their knowledge base horizontally but also in terms of how they mature along a vertical dimension. This line of inquiry calls for longitudinal approaches (e.g., Xu, 2017), which are notoriously more difficult than other types of studies but necessary if we want to learn more about this under-researched area.

Finally, the developmental perspective covers LAL pedagogies, including both epistemological models underpinning program design and specific courses exemplifying implementation at the local level. The literature offers a number of descriptions of and reflections on existing language assessment courses for pre- and in-service teachers (e.g., Giraldo, 2021; Kremmel et al., 2018; O’Loughlin, 2006; Walters, 2010). Most of these courses are of a rather short duration, usually only one semester, with priority given to topics such as selecting, evaluating, and designing assessment instruments, often at the expense of attention to assessment principles such as ethics, transparency, and fairness (Giraldo, 2021). What the literature clearly lacks is work dedicated to the long-term effects of such courses. In fact, there is evidence to suggest that the impact is as short-lived as the courses themselves, and that their sustainability is questionable at best (Giraldo, 2021). The literature also lacks more general models and heuristics for LAL development in teacher education (e.g., Berger, 2012; Yan & Fan, 2021). To conclude with Fulcher (2021, p. 45), “the next step in LAL research will be the investigation of successful LAL pedagogies for language teachers and other stakeholders.”

The present study integrates the quest for empirical LAL progressions with pedagogical considerations. It is an extension of previous work done to investigate the difficulty of assessment-related abilities (Berger et al., 2023), with the express purpose of providing an empirical basis for course design. Whilst the previous study offers a tentative progression of LAL in the Austrian context, it is limited in that the findings are based purely on pre-service teachers’ perceptions of the difficulty of assessment-related AKUs, as opposed to what they can actually do, know, or understand. This study, in contrast, is based on test performance data reflecting students’ knowledge and understanding. The following research questions are addressed:

1. ) What is the empirical difficulty of the assessment-related AKUs for pre-service teachers of English at the University of Vienna?

2. ) How can the continuum from easy to difficult AKUs be characterized?

3 Institutional Context of the Study

Students in the Bachelor of Education (BEd) program in the English Department at the University of Vienna take a compulsory one-semester course on language assessment. This introductory course is designed to give pre-service teachers of English a basic grounding in some of the key areas of language assessment (Department of English and American Studies, University of Vienna, 2022). Conceptually, within a three-dimensional framework characterizing the course content (focus on formal large-scale language testing and/or classroom-based language assessment), qualifications (focus on skills, knowledge, and/or principles), and degree of collaboration (focus on self-directed and/or collaborative learning), the course can be broadly described as a knowledge-skills-and-principles-integrated, collaborative classroom assessment course (Berger, 2012). Table 1 provides a brief overview of the course (for details, see Berger & Heaney, 2019).

Besides input and theory, the course offers a strong practical component involving two project-based assignments. In groups of four to six, students develop, pilot, and revise a writing prompt suitable for a classroom achievement test, based on some chapters of a textbook. The second assignment is to select an existing grammar or vocabulary task for the same achievement test, evaluate its usefulness for the given purpose, and modify some of the items if necessary. This iterative task development and selection process is accompanied by several loops of feedback and revision, involving different types of feedback, such as the use of exemplars of construct definitions, reflective dialogues on the draft prompts, or peerfeedforward on the selected tasks. These project-based assignments provide students the opportunity to go through parts of a test development cycle in a guided and reflective way, thereby applying and extending the knowledge that they have acquired during the semester.

As can be seen in Table 1, the final grade for this course is partly based on a theory component. While in the wake of the pandemic this theory component was changed to an online open-book test, in pre-coronavirus times, it took the form of a knowledge-based paper-and-pencil test, either taken in one sitting or divided into three smaller parts written at different points during the semester. The present study uses performance data from the paper-and-pencil version of the theory test.

4 Methods

5 Results

5.1 The Difficulty of the Assessment-Related AKUs

The results showed no overfitting elements, but 49 students had infit values higher than 1.5. Removing the misfitting candidates from the data set resulted in a final sample of n = 371. The facets map in Figure 1 provides an overview of the relationships between the elements of the facets, displaying, from left to right, the measurement scale in logit units, the relative ability of the candidates, and the relative difficulty of the AKUs. More able candidates are located at the top, while less able ones are located at the bottom. Similarly, difficult items appear at the top, whereas easier ones appear at the bottom. This visual representation allows a direct comparison of elements. For example, a student plotted at the same point on the logit scale as a dichotomous item has a 50% probability of succeeding with that item.

Figure 1

Facets Map Displaying the Calibrated Elements

Research question one (What is the empirical difficulty of the assessment-related AKUs for Austrian pre-service teachers of English?) can be answered by inspecting the measures of the calibrated AKU statements, illustrative examples of which are given in the Appendix. As can be seen, the item difficulty estimates range from 5.30 logits for the most difficult item (can analyze a given item in terms of Purpura’s [2004] model of grammatical and pragmatic knowledge [logical connection]) to -4.15 logits for the easiest item (understands that the Austrian school-leaving exam is not a form of assessment as learning), covering a total range of 9.45 logits, with a mean of -0.14 (SD = 1.67) and a mean SE of 0.42. The high reliability (0.92) and strata indices (4.71), as well as the significant chi-square statistic, X² (205, n = 371) = 4273.6, p < . 00, show that the items differed significantly in terms of difficulty and that the analysis reliably separated the statements into five levels of difficulty.

6 Discussion

Due to the highly contextualized nature of LAL in general (Tsagari & Vogt, 2017; Xu & Brown, 2017) and the test instrument on which this study is based, in particular, the progressions presented here may not be generalizable to other contexts. It should be emphasized that the calibrations are not intended to represent a developmentally invariable learning trajectory. Nor do they reflect the natural order in which these AKUs are developed over time, let alone prescribe the order in which they should be developed. Building LAL is not like starting at the bottom of a ladder and then moving up to the top rung by rung. Such a view, albeit attractive, would be a gross simplification of the learning process, disregarding the many complexities and factors influencing LAL at both individual and contextual levels (e.g., Crusan et al., 2016; Yan et al., 2018).

Having said that, the findings can provide an empirical basis for the teaching of language testing and assessment, and as such they might be useful in similar settings. In particular, information about the difficulty of assessment-related AKUs can help teacher educators to make some basic pedagogical decisions, such as how to sequence the course content in a way that is not just intuitively plausible but empirically informed and consistent with the notion that developing LAL is a process of dealing with increasing difficulty and complexity, as opposed to merely covering a growing body of content. The following design principles could be useful for difficulty-informed LAL programs in pre-service teacher education:

1. On the whole, LAL programs could progress from easier to more difficult AKUs.

2. AKUs should ideally be developed in connection with practical work that has real-world relevance.

3. Easier AKUs closely associated with the day-to-day work of language teachers could be a good starting point.

4. Difficult AKUs could be integrated with easier ones in a cyclical and iterative process.

5. Difficult AKUs could be developed through deep-level processing.

6. Difficult AKUs may require more instruction time than easier ones.

7. Possible bottlenecks to learning (i.e., difficult AKUs which might impede learning or prevent students from fully understanding a certain aspect of language assessment) should be identified and addressed at appropriate points.

These design principles can be applied to both individual teaching units and the syllabus at large. By way of illustration, a revised syllabus for the assessment course at the University of Vienna which is in line with these design principles could progress as follows: Initially, the focus could be on formative, learning-oriented, and alternative assessment. These topics are closely related to the day-to-day work of language teachers in Austria, and basic AKUs relating to these topics appear at the bottom end of the difficulty spectrum. The focus could then shift to achievement testing as the most common test purpose in the Austrian language classroom. Once the students have a reasonable level of understanding of the essential links between testing, assessment, teaching, and learning, the program could move on to test development, with practical and project-based assignments being at the heart of the learning process. Although in a classic test development cycle questions of test purpose, test construct, and test method would typically be addressed in that order, for pedagogical reasons the sequence could be reversed so that it is in accordance with the difficulty continuum and perhaps more realistically reflects teachers’ actual practice. Before covering task production, the starting point could be a task selection assignment in which students choose an existing task suitable for a classroom-based achievement test in the given context. This idea that task selection takes precedence over task production is also supported in Berger et al. (2023). Initially, the focus could be on the test method, response formats, and task design before moving on to recreating the construct and specifications in a process of backwards engineering. The students could receive different types of guidance and feedback along the way, for example in the form of a checklist for identifying the task characteristics and relevant test method facets, an exemplar of a construct definition for comparison with the construct they recreated, or a reflective dialogue about possible changes to the chosen task. This process could be adapted for the next, more difficult language skill(s), this time possibly involving task production, not just selection. For example, students could be asked to develop a writing prompt suitable for a classroom achievement test. Special attention should be paid to the question as to how to leverage the results of an achievement test for formative purposes, particularly with regard to the key role that feedback plays when it comes to enhancing learning.

At a later point, the focus could shift from achievement testing to proficiency testing, another important test purpose in the Austrian educational system. A contrastive analysis of achievement test tasks and proficiency test tasks could help students to understand that the Austrian school-leaving exam is not an achievement test, an AKU which was identified as a bottleneck to fully understanding test purposes in the Austrian educational context (because understanding that the Austrian school-leaving exam is not an achievement test turned out to be much more difficult than any other negatively worded AKU relating to the SRDP). Instead of covering the principles of assessment, test usefulness, and validation linearly in a separate thematic unit, they could be integrated with the practical assignments and treated recursively at various points along the way while discussing issues of successively increasing complexity. Ideally, all assignments are linked back to the bigger picture in terms of washback, consequential validity, and the learning orientation of classroom assessment, so as to consolidate students’ understanding of the links between testing, assessment, teaching, and learning.

This approach outlined here is difficulty-sensitive in that it takes account of differences in difficulty between assessment activities. It is one possible response to the challenge of placing theory alongside practical elements; it complies with the wish of many teachers to concentrate on practical assessment matters (Vogt & Tsagari, 2014); and yet it provides knowledge-skills-and-principles-integrated teacher education, as opposed to a-theoretical and merely practice-driven teacher training.

7 Conclusion

This article contributes to current discussions of LAL development by shedding some light on the difficulty of language assessment for pre-service teachers of English. While earlier conceptualizations of LAL at different levels often remain theoretical or rely heavily on questionnaire data and stakeholders’ perceptions of difficulty, this study provides empirical difficulty estimates based on a LAL theory test used in an assessment course in the English teacher education program at the University of Vienna. The items of that test were transformed into statements of AKU, which were then scaled on the basis of multi-facet Rasch measurement. The resulting calibrations were examined qualitatively for content clusters at certain points along the scale so as to characterize the progression in general terms. Broadly speaking, the difficulty continuum ranges from basic concepts and terminology to AKUs closely associated with practical classroom activities, pedagogical aspects of assessment, formative assessment, and alternative assessment; on to AKUs relating to test methods, simple test design, scoring, and grading; then from basic aspects of validity and reliability in relation to classroom practice on to test constructs, construct definition, and issues of validation; and finally analytical skills and the application of theoretical models. Seven design principles for difficulty-informed LAL courses in teacher education programs have been suggested and illustrated.

While the approach taken here complements previous work investigating the vertical dimension of LAL, the study has several limitations. A major conceptual limitation lies in the fact that the progression is based on an existing local test used in a particular setting. This test is designed to cover the theoretical part of a one-semester assessment course, with many items requiring memorization, recall, understanding, and analysis, usually in relation to factual, conceptual, or procedural knowledge. As the practical part of the course is assessed elsewhere, comparatively few items test the practical application of the knowledge, and there are no items which would require students to use their knowledge to create something new. Thus, there is an imbalance between knowledge, skills, and principles, the core components that are often considered to be constitutive of LAL (Davies, 2008; Inbar-Lourie, 2008), and therefore the difficulty continuum is necessarily, by its very nature, partial and incomplete. Further research could redress this imbalance by focusing more strongly on the difficulty of assessment-related practical skills. Another limitation is the narrow scope of the study. Being restricted to a specific teacher education context in Austria, the difficulty continuum presented here may in a large part reflect local teaching and learning traditions, and so the results are not directly generalizable beyond this sample. Further work needs to establish to what extent the difficulty continuum is similar in other contexts. Further work could also use different conceptualizations of difficulty and employ qualitative methods to counterbalance the measurement-driven approach taken here.

Notwithstanding these limitations, the findings of the study have practical implications. Progressions such as the one presented here can be useful for program design, provided that they are considered to be an open and dynamic rather than a normative framework. While the difficulty continuum should not be mistaken for a developmentally determined learning trajectory that students follow in a lockstep fashion, information about the difficulty of assessment-related AKUs can provide an empirical basis for important pedagogical decisions, such as the order in which the content should be addressed, how much instruction time should be devoted to any particular topic, the level of cognitive involvement conducive to learning, the degree of cyclicality in the teaching process, what bottlenecks could impede learning, and how learning could be instructionally scaffolded. As the question of difficulty is not context-independent, it is hoped that more such continua will be available in the future for different settings. Together with longitudinal studies on how LAL develops over time, such empirical investigations of difficulty can help us better understand not just how pre-service teachers expand their knowledge base horizontally by covering an ever-increasing number of assessment topics, but also how they mature and develop along a vertical dimension.