Volume 4, Issue 1 - Special Issue: Language Testing and Assessment in Higher Education: Some Current Issues

This article presents and analyses a classroom-based assessment method to test students' speaking skills in a variety of professional settings in tourism. The assessment system has been implemented in the Communication in English for Tourism course, as part of the Tourism Management degree programme, at Florida Universitaria (affiliated to the University of Valencia). Based on our ESP teaching experience, we have noticed a need to design an assessment procedure that would enable us gather several samples of students' speaking competencies throughout the course. An extended process that involved research and meetings with colleagues and students led to the development of the assessment method in question, which can be described as a multimodal communicative approach to testing, organized as continuous assessment, and fully related to the course syllabus. To enhance validity and reliability, we have opted to create an assessment procedure based on a combination of testing formats, rating criteria and rating scales. The procedure involves two testers and the overall grade is based on the results obtained in seven tests. The article reviews contributions to the assessment of speaking abilities, presents our testing procedure, describes its implementation, discusses its advantages and disadvantages, and concludes by analysing its washback effect and pedagogical implications.

In recent years Finnish university language centres have increasingly developed procedures for assessing and recognising the skills in English for Specific Purposes (ESP) that students acquire in various non-formal and informal learning environments. This article describes the procedures developed by the University of Eastern Finland Language Centre to assess business students' non-formal and informal learning of ESP language and communication skills. The examinations are held every semester for students seeking exemption from the ESP courses that form part of their undergraduate degree in business and economics. In the examinations the students' skills and abilities are tested using the same criteria and methods as in the courses from which exemption is sought. The article also describes how business students perceive the exemption examination process as a means to recognising and assessing their non-formally and informally acquired ESP language and communication skills. Applying a mixed methods research design, the data presented was elicited from business students who participated in the ESP exemption examinations at the University of Eastern Finland Language Centre in the autumn semester 2012. The results of this pilot study indicate that the students regard the exemption examination process as their preferred means of demonstrating prior learning of ESP, and consider the pre-examination guidance and information provided by the teachers and the self-assessment instructions functional and sufficient. These findings encourage the further development of the ESP exemption examination system at the University of Eastern Finland Language Centre.

Multiple-choice (MC) tests are frequently used to measure language competence because they are quick, economical and straightforward to score. While degrees of correctness have been investigated for partially correct responses in combined-response MC tests, degrees of incorrectness in distractors and the role they play in determining the test-taker's final score remain comparatively unexplored. This pilot study examines degrees of incorrectness in MC test items and their potential impact on the overall reliability of scoring procedures, test micro impact, and scoring performance of individual test-takers. Investigating the above properties as quality dimensions of language testing is crucial since decisions made on the basis of such tests influence the careers and lives of test-takers. Three scoring procedures were applied to an authentic test: dichotomous, which is straightforward – correct/incorrect; polychotomous, which offers partial credit to very plausible and plausible options; and negative, which includes negative scoring when a wholly incorrect answer is selected. While these three procedures proved to be equally reliable, the polychotomous scoring method offered more sensitive and precise results, while negative scoring was particularly disadvantageous for test-takers. The polychotomous method of scoring moves away from the absolute right or wrong view and enhances sensitivity in the field of individualised language testing.

This study compares two approaches to grammatical knowledge in language assessment: the structural view that regards grammatical knowledge as vocabulary and syntax (Bachman 1990), and the communicative view that perceives it as the binary combination of grammatical form and meaning (Purpura 2004). 1,491 second-year nursing students from eight medical colleges in China took a fifteen-item English grammar test (GT) that used retired items from the Language and Use section of the Public English Test System, Level Two. Data analysis comprised a series of dimensionality assessments (DAs) based on bifactor- multidimensional item response theory (bifactor-MIRT). This involved assessing the model fit achieved by structuring the GT tasks using a single grammatical factor, the structural approach and the communicative approach, and then comparing the relative performance of the three approaches. The results indicated that 1) despite its operational attraction, a unidimensional structure was insufficient to structure the GT tasks; 2) both the structural and the communicative approaches could sufficiently explain the underlying structure of the GT tasks; but 3) the communicative approach seemed to outperform the structural approach in uncovering the factual structure of the GT tasks. The study shows how bifactor-MIRT can be used to compare grammatical knowledge theories.

This article describes a procedure for constructing C-tests for UNIcert® courses, and in doing so gives an insight into work in progress. Each year since 2005 twelve colleagues working in language centres in third-level institutions in Germany have collaboratively developed a C-test for assigning students to French courses. Our central concern is to link the C-test format to the UNIcert® certification system. As the linguistic demands of the different UNIcert® levels cannot be reliably scaled on the basis of the usual descriptors, we use teaching times as stipulated in the UNIcert® framework and correlate this with corresponding language levels that cannot be described precisely. From this we have derived a UNIcert® reference scale, which shows competence ranges for each level in terms of C-test scores. In this way it is possible to define entry levels for the different courses according to a single system of C-test scores. The scale can only be used, however, if the C-tests in question satisfy certain requirements. Accordingly, a UNIcert® reference “testlet” to determine admission to appropriate levels of French courses is being developed according to various parameters. Empirical procedures are being used to explore potentially suitable testlets. Course-related pilot testlets are tested by students taking UNIcert® courses. The results from the piloting of the testlets are compared with the reference testlet, and this determines whether or not they are used in the main test that assigns students to the appropriate levels of the UNIcert® courses for which they have registered. Grading is on the basis of total test scores and the UNIcert® reference scale. Kurzfassung Im folgenden Beitrag wird ein Verfahren zur Konstruktion von C-Tests für UNIcert®-Kurse vorgeschlagen. Er gibt einen Einblick in die Arbeit von gegenwärtig 12 Kolleginnen und Kollegen aus Sprachenzentren an Hochschulen und Universitäten der Bundesrepublik, die seit 2005 jährlich gemeinsam einen C-Test für die Einstufung in Französisch-Kurse konzipieren. Im Zentrum der Überlegungen steht die Anbindung des C-Test-Formats an das UNIcert®-Zertifikatssystem. Da sich die sprachlichen Anforderungen der einzelnen Niveaustufen mit den bekannten Deskriptoren nicht zuverlässig skalieren lassen, werden in Übereinstimmung mit der UNIcert®-Rahmenordnung Unterrichtszeit und ein inhaltlich nicht näher definierbarer Sprachstand zueinander in Beziehung gesetzt. Daraus wird die UNIcert®-Referenzskala abgeleitet, auf der für jeden Ausbildungsabschnitt ein C-Test-Punktekorridor abgebildet wird. Damit sind die Zugangsniveaus zu den Ausbildungsabschnitten einheitlich nach C-Test-Punkten definierbar. Diese Skala kann jedoch nur verwendet werden, wenn die eingesetzten C-Tests bestimmten inhaltlichen Voraussetzungen genügen. Deshalb wird ein UNIcert®-Referenztestlet für die Einstufung in Französisch-Kurse mit Hilfe verschiedener Parameter konstruiert. Potentiell geeignete Testlets werden mit empirischen Verfahren gesucht. Studierende aus UNIcert®-Kursen testen kursbezogen entsprechende Aufgaben. Die Ergebnisse aus der Pilotierung der Testlets werden mit dem Referenztestlet verglichen und entscheiden über die Verwendung im Haupttest, der zur Einstufung in die UNIcert®-Kurse verwendet wird. Die Einstufung erfolgt gemäß der im Haupttest erreichten Summenwerte und der UNIcert®-Referenzskala.

One of the central challenges faced by UNIcert®, a system of certification for university language courses, is the establishment of comparability across languages and institutions at each proficiency level. But how is this to be achieved in practice? One possible approach is the one adopted by the project “Qualitative calibration of UNIcert® Level I – Writing Skills ”, in which representatives of nine universities developed and implemented non-language-specific writing tasks and rating grids for UNIcert® Level I (B1). The article provides a detailed account of the background, aims and practical procedures of this quality assurance project. On the basis of sample student texts elicited within the project, it also discusses the methodological possibilities and limitations of a non-language-specific analysis and comparison of performance levels. Kurzfassung Die Herstellung von Vergleichbarkeit der UNIcert®-Niveaustufen über Sprach- und Institutionsgrenzen hinweg gehört zu den zentralen Herausforderungen des hochschulspezifischen Ausbildungs- und Zertifizierungssystems. Doch wie lässt sich diese Vergleichbarkeit in der Praxis herstellen? Einen möglichen Ansatz verfolgte das Projekt Qualitative Kalibrierung der Stufe UNIcert® I – Schriftlicher Ausdruck , in dem Vertreter aus neun Hochschulen sprachenübergreifend Aufgaben zum schriftlichen Ausdruck für die Stufe UNIcert® I (B1) sowie einheitlich einsetzbare Bewertungsraster entwickelten und einsetzten. Der Beitrag stellt die Hintergründe und Zielsetzung sowie das praktische Vorgehen innerhalb des Projektes zur Qualitätssicherung vor. Zudem werden die methodischen Möglichkeiten und Grenzen einer sprachenübergreifenden Analyse und Vergleichbarkeit von Niveaustufen anhand der im Rahmen des Projektes erhobenen studentischen Textbeispiele diskutiert.

The Applied Language Centre at University College Dublin offers foreign language modules to students in ten languages at CEFR levels ranging from A1 to B2. Efforts have been underway in the Centre to standardise the assessment components across languages to ensure parity between module credits and to meet student expectations as they migrate from one language module to another and one language level to the next. This process has involved close scrutiny of the reliability and validity of the existing testing techniques and attempts at improving rater reliability across languages and levels as well as the endeavour to tailor exam components to student needs. The university language modules focus on communicative ability and functional language use (students write short e-mails, respond to informational texts, etc.), so it is important that the assessment weighting and the examination material also relate to the immediate, practical needs of students (often beginners in a new language in this context). The proposed module outcomes need to be carefully tailored to the demands of university students and assessed in a fair and consistent manner across languages. The alignment between intended learning outcomes, classroom activities and assessment is crucial in this environment and requires constant evaluation and revision.

There are increasing numbers of non-native English speaking applicants to Canadian universities (AUCC 2008a, 2010), which are committed to promoting linguistic and cultural diversity (AUCC 2008b). One result of this trend is that university admissions officers, as gatekeepers, are faced with a growing and potentially confusing array of language test scores when making their decisions. These admissions decision makers need a certain amount of language assessment literacy (LAL) to enable them to make use of these language test scores effectively and ethically (O’Loughlin 2011, 2013). This article reports on the first phase of a project designed to address this challenge. The project involves the collaboration of assessment professionals and admissions officers across Canada in determining the LAL base needed for users of language test scores in university admissions decision-making. This first phase of research consisted of a survey with university admissions officers across Canada, inquiring about their knowledge, beliefs, and levels of confidence in making use of language test scores in decision-making. Results have begun to reveal the nature of the LAL needed for these users, and have suggested the most appropriate content for later informational workshops with admissions officers (Phase 2 of the project). While some evidence of misunderstanding was identified, respondents demonstrate awareness of concepts related to validity in language assessment, albeit without making use of the conventional language of the field.

This article considers the assessment challenges and prospects faced by foreign language teachers in Ukrainian universities in the context of the country's joining the Bologna Process. A major problem relates to adapting higher education to a common framework designed to facilitate comprehension and comparison of the content and outcomes of degree programmes and devising methods of quality assurance in foreign language teaching. The implementation of reforms in the area includes the improvement of assessment standards, the creation of new assessment instruments, and ensuring that teachers' professional repertoire includes skills in accurately measuring students' performance. Given the increased role of assessment in Ukrainian language education and the challenges this new demand poses to teachers, we investigated university foreign language teachers' readiness to carry out complex and multifaceted functions related to formative and summative assessment. Specifically, we replicated the European Survey of Language Testing and Assessment Needs and carried out a survey of our own design to identify specific strengths and weaknesses in the assessments performed by university foreign language teachers. The surveys yielded results that are broadly comparable with the European data, but with minor variations that mostly have to do with newly introduced forms of assessment. Our study shows that Ukrainian university foreign language teachers are quite test-wise and prepared to do a good job in language testing and assessment. The insights drawn from the data can contribute to developing syllabi for pre- and in-service teacher training in the area.

This article discusses the problem of quality in the production of language tests in the context of Italian language examinations. The concept of quality is closely related to the application of stated standards and related procedures. These standards, developed over the last thirty years, are mainly related to the concepts of the accountability and responsibility of organisations and testing bodies. In this context and for these purposes, a quality management system (QMS) could be a useful tool for the setting out of procedures and the consistent application of standards to the different phases in the production of language examinations, thereby supporting and providing evidence of the valid interpretation and use of test results (Messick 1989). Taking into account the development of standards in educational assessment (Saville 2014), the article starts with a brief exploration of the terms standard, quality , and quality management system , then goes on to discuss the potential relationship between standards, QMS and the argument-based approach to validation (Kane 2006, 2010, 2012; Bachman and Palmer 2010). The second part of the article describes, in concrete terms, the experience of the Centro per la Valutazione e le Certificazioni Linguistiche (CVCL) at the Università per Stranieri di Perugia, in applying the QMS produced by the Association of Language Testers in Europe (ALTE), considering how this application aimed to implement a set of procedures in order to achieve the stated standards through a continuous process of improvement of the overall testing system.

The testing of intercultural competence has long been regarded as the field of psychometric test procedures, which claim to analyse an individual's personality by specifying and quantifying personality traits with the help of self-answer questionnaires and the statistical evaluation of these. The underlying assumption is that what is analysed and described as a candidate's personality can be treated as an indicator of that same person's practical performance in intercultural encounters. From the point of view of a test constructor for language competence, all intercultural tests of this type raise basic questions concerning their construct and predictive validity. Against this background, this article firstly examines the shortcomings of personality-based tests of intercultural competence. Secondly, based on relevant parts of the CEFR as well as on the work of numerous contributors to the international debate, a practicable construct of intercultural communicative competence is suggested. Special attention is paid to the concept of politeness in intercultural encounters and the role of English as a lingua franca (ELF). Thirdly, a basic outline of a criterion-based test of intercultural competence in English is provided. The test procedures on which this article draws have been extensively piloted and are part of a training package including test specifications, course materials and teacher-training material.

Language testing means taking decisions: about the test taker's results, but also about the test construct and the measures taken in order to ensure quality. This article takes the German test telc Deutsch C1 Hochschule as an example to illustrate this decision-making process in an academic context. The test is used for university entry; therefore, the main stakeholders are academic and administrative staff as well as international students about to take up their studies in Germany. The test provider had to take the stakeholders' expectations and teaching traditions into account and at the same time identify disagreements and conflicting expectations. In the end, the almost unlimited possible ways of testing in this context had to be narrowed down. While not all decisions which lead to a valid form of assessment can be presented here, we provide an illustration of the key phases: the definition of the test construct, the construction of the test format and the actual examining. There is a special focus on the definition of academic German, especially in view of a very heterogeneous target-group. In addition, concrete examples from the initial test development phase are presented, from defining the format for the writing task to choosing texts for reading and testing grammar. Ongoing work on the test is illustrated by insights into the commissioning of new tasks and the rating of open-ended questions.