John Benjamins Publishing Company
Chapter 6. Exploring the benefits and challenges of web-based approaches to standard setting
Abstract
One aspect of standard setting that has been relatively consistent throughout its history is the interactional context in which expert panelists have made their judgments: face-to-face, in-person meetings. With the growth of the internet and acceleration of web-based technology, researchers have started considering how web-based approaches may address some of the inherent practical challenges for standard setting studies (see Katz et al., 2009). The worldwide pandemic of 2020 brought about an acceleration of the pervasiveness and use of web-based meeting software, making it not only a more attractive option for standard setting panels, but for a time potentially the only option. In this chapter, existing research on approaches to web-based standard setting is reviewed, and the procedural elements of three recent web-based standard setting studies conduct at ETS are summarized and compared. The accounts of web-based standard setting described in this chapter suggest that a web-based approach may be an attractive alternative to face-to-face meetings, not merely a pandemic-era necessity.
Abstract
One aspect of standard setting that has been relatively consistent throughout its history is the interactional context in which expert panelists have made their judgments: face-to-face, in-person meetings. With the growth of the internet and acceleration of web-based technology, researchers have started considering how web-based approaches may address some of the inherent practical challenges for standard setting studies (see Katz et al., 2009). The worldwide pandemic of 2020 brought about an acceleration of the pervasiveness and use of web-based meeting software, making it not only a more attractive option for standard setting panels, but for a time potentially the only option. In this chapter, existing research on approaches to web-based standard setting is reviewed, and the procedural elements of three recent web-based standard setting studies conduct at ETS are summarized and compared. The accounts of web-based standard setting described in this chapter suggest that a web-based approach may be an attractive alternative to face-to-face meetings, not merely a pandemic-era necessity.
Chapters in this book
- Prelim pages i
- Table of contents v
- Acknowledgments vii
- Series editors preface ix
- Chapter 1. Overview of research to enhance score interpretation 1
-
Part I. Vertical linking research for suites of language tests
- Chapter 2. Considerations in developing vertical scales for language tests 14
- Chapter 3. Assessment design issues in developing vertical scales for language tests 35
- Chapter 4. Statistical methodology for developing vertical scales for language tests 61
-
Part II. Enhancing score interpretation for individual language tests
- Chapter 5. Scale anchoring methodology for developing revised performance level descriptors for the TOEFL iBT® test 80
- Chapter 6. Exploring the benefits and challenges of web-based approaches to standard setting 99
- Chapter 7. Mapping TOEFL® Essentials™ speaking and writing scores to the CEFR levels 120
- Chapter 8. Designing and executing a score concordance study to establish score mapping to proficiency levels 141
-
Part III. Concluding commentary
- Chapter 9. Literacy, transparency, and (mis)interpretations in communicating with testing stakeholders 158
- Notes on contributors 168
- Index 171
Chapters in this book
- Prelim pages i
- Table of contents v
- Acknowledgments vii
- Series editors preface ix
- Chapter 1. Overview of research to enhance score interpretation 1
-
Part I. Vertical linking research for suites of language tests
- Chapter 2. Considerations in developing vertical scales for language tests 14
- Chapter 3. Assessment design issues in developing vertical scales for language tests 35
- Chapter 4. Statistical methodology for developing vertical scales for language tests 61
-
Part II. Enhancing score interpretation for individual language tests
- Chapter 5. Scale anchoring methodology for developing revised performance level descriptors for the TOEFL iBT® test 80
- Chapter 6. Exploring the benefits and challenges of web-based approaches to standard setting 99
- Chapter 7. Mapping TOEFL® Essentials™ speaking and writing scores to the CEFR levels 120
- Chapter 8. Designing and executing a score concordance study to establish score mapping to proficiency levels 141
-
Part III. Concluding commentary
- Chapter 9. Literacy, transparency, and (mis)interpretations in communicating with testing stakeholders 158
- Notes on contributors 168
- Index 171
