Research on Improving Rating Reliability for School-based Oral English Achievement Tests: The Design and Development of a Computer-aided Rating System
-
Lu Jun
Abstract
Low rating reliability has long been the primary concern in school-based oral English achievement tests. In this study, a computer-aided rating system (CARS) was developed to improve inter- and intra-rater reliability through the instantiation of rating criteria, task division and random distribution, on-line training, reliability verification and sound wave “reading” and “writing.” A rating experiment was conducted among six raters to compare intra- and inter-rater reliability between traditional rating and rating with CARS. At the end of each round of rating, a conference was held. Both quantitative and qualitative analyses show that CARS can significantly improve inter- and intra-rater reliability, mainly through helping raters use criteria more accurately and focus more attention on rating. In addition, the research has also shed light upon further study on improving rating reliability.
学校规模的英语口语课程考试一直面临着评分一致性不高的难题。为此，本研究开发了计算机辅助评分系统（CARS)，力图通过评分标准实例化、评分任务随机分布处理、单项任务评分、在线评分培训和一致性验证、利用波形图谱实现可视化评分和反馈等手段来提高评分一致性。由6位英语教师先后进行传统评分和机助评分试验，并在评分结束后围绕提纲进行座谈。通过对两次评分结果和座谈记录进行定量和定性分析表明：该系统能够帮助评分员把握评分标准、增强责任心和集中注意力，评分员内部一致性和评分员之间的一致性都比传统评分方法有显著提高。最后为进一步开展评分信度研究提出建议和启示。
© 2011 Foreign Language Teaching and Research Press Co. Ltd., Walter de Gruyter GmbH & Co. KG, Berlin/Boston and Cultural and Education Section British Embassy
Articles in the same Issue
- Automatic Extraction of Contextual Co-occurrence Chain and Its Relationship with Textual Cohesion
- Relative Clauses in Hong Kong English: A Corpus-based Perspective
- Can EFL Interactive Listening Be Validly Assessed?
- Chinese College Students' Abilities and Attitudes for Peer Review
- Research on Improving Rating Reliability for School-based Oral English Achievement Tests: The Design and Development of a Computer-aided Rating System
- Comparison Between the Students' Perceived and Expected Behavior of College English Teachers
- Empirical Studies on L2 Communication Strategies over Four Decades: Looking Back and Ahead
- Scaffolding in the Second Language Learning of Target Forms in Peer Interaction
- Chinese Abstracts
Articles in the same Issue
- Automatic Extraction of Contextual Co-occurrence Chain and Its Relationship with Textual Cohesion
- Relative Clauses in Hong Kong English: A Corpus-based Perspective
- Can EFL Interactive Listening Be Validly Assessed?
- Chinese College Students' Abilities and Attitudes for Peer Review
- Research on Improving Rating Reliability for School-based Oral English Achievement Tests: The Design and Development of a Computer-aided Rating System
- Comparison Between the Students' Perceived and Expected Behavior of College English Teachers
- Empirical Studies on L2 Communication Strategies over Four Decades: Looking Back and Ahead
- Scaffolding in the Second Language Learning of Target Forms in Peer Interaction
- Chinese Abstracts
