Validation of two measures for assessing English vocabulary knowledge on web-based testing platforms: brief assessments

Lee Drown; Nikole Giovannone; David B. Pisoni; Rachel M. Theodore

doi:10.1515/lingvan-2022-0116

Article

Validation of two measures for assessing English vocabulary knowledge on web-based testing platforms: brief assessments

Lee Drown , Nikole Giovannone , David B. Pisoni and Rachel M. Theodore

Published/Copyright: September 13, 2023

Published by

Become an author with De Gruyter Brill

Submit Manuscript Author Information Explore this Subject

From the journal Linguistics Vanguard Volume 9 Issue 1

Abstract

Two measures for assessing English vocabulary knowledge, the Vocabulary Size Test (VST) and the Word Familiarity Test (WordFAM), were recently validated for web-based administration. An analysis of the psychometric properties of these assessments revealed high internal consistency, suggesting that stable assessment could be achieved with fewer test items. Because researchers may use these assessments in conjunction with other experimental tasks, the utility may be enhanced if they are shorter in duration. To this end, two “brief” versions of the VST and the WordFAM were developed and submitted to validation testing. Each version consisted of approximately half of the items from the full assessment, with novel items across each brief version. Participants (n = 85) completed one brief version of both the VST and the WordFAM at session one, followed by the other brief version of each assessment at session two. The results showed high test-retest reliability for both the VST (r = 0.68) and the WordFAM (r = 0.82). The assessments also showed moderate convergent validity (ranging from r = 0.38 to 0.59), indicative of assessment validity. This work provides open-source English vocabulary knowledge assessments with normative data that researchers can use to foster high quality data collection in web-based environments.

Keywords: assessment; vocabulary; web-based testing

Corresponding author: Rachel M. Theodore, Department of Speech, Language, and Hearing Sciences, University of Connecticut, 2 Alethia Drive, Unit 1085, 06269-1085 Storrs, CT, USA, E-mail: rachel.theodore@uconn.edu

Funding source: National Science Foundation

Award Identifier / Grant number: DGE-1144399, DGE-1747486

Funding source: National Institutes of Health

Award Identifier / Grant number: R01DC015257, R21DC016141, T32DC017703

Acknowledgments

This work was supported by NIH NIDCD grant R21DC016141 to RMT, NSF grants DGE-1747486 and DGE-1144399 to the University of Connecticut, NIH NIDCD grant R01DC015257 to Indiana University, and by the Jorgensen Fellowship (University of Connecticut) to NG. LD was supported by NIH NIDCD grant T32DC017703. The views expressed here reflect those of the authors and not the NIH, the NIDCD, or the NSF. Portions of these data were presented at the 2021 convention of the American Speech-Language-Hearing Association.

Research funding: This work was supported by National Science Foundation (DGE-1144399, DGE-1747486) and National Institutes of Health (R01DC015257, R21DC016141, T32DC017703).

References

Anastasi, Anne & Susana Urbina. 1997. Psychological testing. New York, NY: Prentice Hall/Pearson Education.Search in Google Scholar

Beglar, David. 2010. A Rasch-based validation of the Vocabulary Size Test. Language Testing 27(1). 101–118. https://doi.org/10.1177/0265532209340194.Search in Google Scholar

Beglar, David & Paul Nation. 2007. A Vocabulary Size Test. The Language Teacher 31. 9–13.Search in Google Scholar

Bleses, Dorthe, Guido Makransky, Phillip S. Dale, Anders Højen & Burcak A. Ari. 2016. Early productive vocabulary predicts academic achievement 10 years later. Applied Psycholinguistics 37(6). 1461–1476. https://doi.org/10.1017/S0142716416000060.Search in Google Scholar

Bloom, Paul. 2002. How children learn the meanings of words. Cambridge, MA: MIT Press.Search in Google Scholar

Brysbaert, Marc & Boris New. 2009. Moving beyond Kučera and Francis: A critical evaluation of current word frequency norms and the introduction of a new and improved word frequency measure for American English. Behavior Research Methods 41(4). 977–990. https://doi.org/10.3758/brm.41.4.977.Search in Google Scholar

Colby, Sarah, Meghan Clayards & Shari Baum. 2018. The role of lexical status and individual differences for perceptual learning in younger and older adults. Journal of Speech, Language, and Hearing Research 61(8). 1855–1874. https://doi.org/10.1044/2018_jslhr-s-17-0392.Search in Google Scholar

Coxhead, Averil. 2016. Dealing with low response rates in quantitative studies. In Jim McKinley & Heath Rose (eds.), Doing research in applied linguistics, 81–90. Abingdon, Oxfordshire: Routledge.10.4324/9781315389608-8Search in Google Scholar

Coxhead, Averil, Paul Nation & Dalice Sim. 2015. Measuring the vocabulary size of native speakers of English in New Zealand secondary schools. New Zealand Journal of Educational Studies 50(1). 121–135. https://doi.org/10.1007/s40841-015-0002-3.Search in Google Scholar

Cristia, Alejandrina, Amanda Seidl, Leher Singh & Derek Houston. 2016. Test–retest reliability in infant speech perception tasks. Infancy 21(5). 648–667. https://doi.org/10.1111/infa.12127.Search in Google Scholar

Drown, Lee, Nikole Giovannone, David B. Pisoni & Rachel M. Theodore. 2023. Validation of two measures for assessing English vocabulary knowledge on web-based testing platforms: Long-form assessments. Linguistics Vanguard 9(1). 113–124.10.31234/osf.io/w48y5Search in Google Scholar

Dunn, Lloyd M. & Leota M. Dunn. 1997. PPVT-III: Peabody Picture Vocabulary Test. Circle Pines, MN: American Guidance Service.10.1037/t15145-000Search in Google Scholar

Gathercole, Susan E. & Alan D. Baddeley. 1993. Phonological working memory: A critical building block for reading development and vocabulary acquisition? European Journal of Psychology of Education 8(3). 259–272. https://doi.org/10.1007/bf03174081.Search in Google Scholar

Giovannone, Nikole & Rachel M. Theodore. 2021. Individual differences in lexical contributions to speech perception. Journal of Speech, Language, and Hearing Research 64(3). 707–724. https://doi.org/10.1044/2020_jslhr-20-00283.Search in Google Scholar

Giovannone, Nikole & Rachel M. Theodore. 2023. Do individual differences in lexical reliance reflect states or traits? Cognition 232. 105320. https://doi.org/10.1016/j.cognition.2022.105320.Search in Google Scholar

Godinho, Alexandra, Christina Schell & John A. Cunningham. 2020. Out damn bot, out: Recruiting real people into substance use studies on the internet. Substance Abuse 41(1). 3–5. https://doi.org/10.1080/08897077.2019.1691131.Search in Google Scholar

Griffin, Marybec, Richard J. Martino, Caleb LoSchiavo, Camilla Comer-Carruthers, Kristen D. Krause, Christopher B. Stults & Perry N. Halkitis. 2022. Ensuring survey research data integrity in the era of internet bots. Quality & Quantity 56. 2841–2852. https://doi.org/10.1007/s11135-021-01252-1.Search in Google Scholar

Hedge, Craig, Georgina Powell & Petroc Sumner. 2018. The reliability paradox: Why robust cognitive tasks do not produce reliable individual differences. Behavior Research Methods 50(3). 1166–1186. https://doi.org/10.3758/s13428-017-0935-1.Search in Google Scholar

Heffner, Christopher C., Pamela Fuhrmeister, Sahil Luthra, Hannah Mechtenberg, David Saltzman & Emily B. Myers. 2022. Reliability and validity for perceptual flexibility in speech. Brain and Language 226. 105070. https://doi.org/10.1016/j.bandl.2021.105070.Search in Google Scholar

Irwin, Julia R., Alice S. Carter & Margaret J. Briggs-Gowan. 2002. The social-emotional development of “late-talking” toddlers. Journal of the American Academy of Child & Adolescent Psychiatry 41(11). 1324–1332. https://doi.org/10.1097/00004583-200211000-00014.Search in Google Scholar

Landi, Nicole. 2010. An examination of the relationship between reading comprehension, higher-level and lower-level reading sub-skills in adults. Reading and Writing 23(6). 701–717. https://doi.org/10.1007/s11145-009-9180-z.Search in Google Scholar

Lewellen, Mary J., Stephen D. Goldinger, David B. Pisoni & Beth G. Greene. 1993. Lexical familiarity and processing efficiency: Individual differences in naming, lexical decision, and semantic categorization. Journal of Experimental Psychology: General 122(3). 316–330. https://doi.org/10.1037/0096-3445.122.3.316.Search in Google Scholar

Mancilla-Martinez, Jeannette, Joanna A. Christodoulou & Michelle M. Shabaker. 2014. Preschoolers’ English vocabulary development: The influence of language proficiency and at-risk factors. Learning and Individual Differences 35. 79–86. https://doi.org/10.1016/j.lindif.2014.06.008.Search in Google Scholar

McGahee, Thayer W. & Julia Ball. 2009. How to read and really use an item analysis. Nurse Educator 34(4). 166–171. https://doi.org/10.1097/nne.0b013e3181aaba94.Search in Google Scholar

Nation, Paul. 2012. The Vocabulary Size Test. 23 October. Available at: https://www.wgtn.ac.nz/lals/resources/paul-nations-resources/vocabulary-tests/the-vocabulary-size-test/Vocabulary-Size-Test-information-and-specifications.pdf.Search in Google Scholar

Nusbaum, Howard C., David B. Pisoni & Christopher K. Davis. 1984. Sizing up the Hoosier mental lexicon. Research on Spoken Language Processing Report 10(3). 357–376.Search in Google Scholar

Palan, Stefan & Christian Schitter. 2018. Prolific.ac – a subject pool for online experiments. Journal of Behavioral and Experimental Finance 17. 22–27. https://doi.org/10.1016/j.jbef.2017.12.004.Search in Google Scholar

Pearson, P. David, Elfrieda H. Hiebert & Michael L. Kamil. 2007. Vocabulary assessment: What we know and what we need to learn. Reading Research Quarterly 42(2). 282–296. https://doi.org/10.1598/rrq.42.2.4.Search in Google Scholar

Pisoni, David B. 2007. WordFam: Rating word familiarity in English. Bloomington, IN: Indiana University.Search in Google Scholar

Rodd, Jennifer. 2019. How to maintain data quality when you can’t see your participants. APS Observer 32(3). https://www.psychologicalscience.org/observer/how-to-maintain-data-quality-when-you-cant-see-your-participants.Search in Google Scholar

Rotman, Tali, Limor Lavie & Karen Banai. 2020. Rapid perceptual learning: A potential source of individual differences in speech perception under adverse conditions? Trends in Hearing 24. 1–16. https://doi.org/10.1177/2331216520930541.Search in Google Scholar

Schmitt, Norbert. 2019. Understanding vocabulary acquisition, instruction, and assessment: A research agenda. Language Teaching 52(2). 261–274. https://doi.org/10.1017/s0261444819000053.Search in Google Scholar

Schmitt, Norbert, Paul Nation & Benjamin Kremmel. 2020. Moving the field of vocabulary assessment forward: The need for more rigorous test development and validation. Language Teaching 53(1). 109–120. https://doi.org/10.1017/s0261444819000326.Search in Google Scholar

Snow, Catherine E. & Young-Suk Kim. 2007. Large problem spaces: The challenge of vocabulary for English language learners. In Richard K. Wagner, Andrea E. Muse & Kendra R. Tannenbaum (eds.), Vocabulary acquisition: Implications for reading comprehension, 123–139. New York, NY: Guilford Press.Search in Google Scholar

Storozuk, Andie, Marilyn Ashley, Véronic Delage & Erin A. Maloney. 2020. Got bots? Practical recommendations to protect online survey data from bot attacks. Quantitative Methods for Psychology 16(5). 472–481. https://doi.org/10.20982/tqmp.16.5.p472.Search in Google Scholar

Strand, Julia F., Violet A. Brown, Madeline B. Merchant, Hunter E. Brown & Julia Smith. 2018. Measuring listening effort: Convergent validity, sensitivity, and links with cognitive and personality measures. Journal of Speech, Language, and Hearing Research 61(6). 1463–1486. https://doi.org/10.1044/2018_jslhr-h-17-0257.Search in Google Scholar

Tamati, Terrin N. & David B. Pisoni. 2014. Non-native listeners’ recognition of high-variability speech using PRESTO. Journal of the American Academy of Audiology 25(9). 869–892. https://doi.org/10.3766/jaaa.25.9.9.Search in Google Scholar

Theodore, Rachel M., Nicholas R. Monto & Stephen Graham. 2020. Individual differences in distributional learning for speech: What’s ideal for ideal observers? Journal of Speech, Language, and Hearing Research 63(1). 1–13. https://doi.org/10.1044/2019_jslhr-s-19-0152.Search in Google Scholar

Wasik, Barbara A., Annemarie H. Hindman & Emily K. Snell. 2016. Book reading and vocabulary development: A systematic review. Early Childhood Research Quarterly 37. 39–57. https://doi.org/10.1016/j.ecresq.2016.04.003.Search in Google Scholar

Wiig, Elizabeth H., Eleanor Semel & Wayne A. Secord. 2013. Clinical evaluation of language fundamentals, 5th edn. Bloomington, MN: Pearson.Search in Google Scholar

Wilbiks, Jonathan M., Violet A. Brown & Julia F. Strand. 2022. Speech and non-speech measures of audiovisual integration are not correlated. Attention, Perception, & Psychophysics 84. 1809–1819. https://doi.org/10.3758/s13414-022-02517-z.Search in Google Scholar

Williams, Kathleen T. 1997. Expressive vocabulary test second edition (EVT™ 2). Journal of the American Academy of Child Adolescent Psychiatry 42. 864–872.Search in Google Scholar

Supplementary Material

This article contains supplementary material (https://doi.org/10.1515/lingvan-2022-0116).

Received: 2022-09-16

Accepted: 2023-02-27

Published Online: 2023-09-13

You are currently not able to access this content.

Articles in the same Issue

https://doi.org/10.1515/lingvan-2022-0116

Keywords for this article

assessment; vocabulary; web-based testing