Multi-modal language input: A learned superadditive effect

References

Alloway, Tracy Packiam & Ross G. Alloway. 2010. Investigating the predictive roles of working memory and IQ in academic attainment. Journal of Experimental Child Psychology 106(1). 20–29.10.1016/j.jecp.2009.11.003Search in Google Scholar

Baddeley, Alan. 2000. The episodic buffer: a new component of working memory? Trends in Cognitive Sciences 4(11). 417–423.10.1016/S1364-6613(00)01538-2Search in Google Scholar

Baier, Bernhard., Andreas Kleinschmidt & Notger Muller. 2006. Cross-modal processing in early visual and auditory cortices depends on expected statistical relationship of multisensory information. The Journal of Neuroscience 26(47). 12260–12265.10.1523/JNEUROSCI.1457-06.2006Search in Google Scholar

Beauchamp, Michael S., Kathryn E. Lee, Brenna D. Argall & Alex Martin. 2004. Integration of auditory and visual information about objects in superior temporal sulcus. Neuron 41(5). 809–823.10.1016/S0896-6273(04)00070-4Search in Google Scholar

Beers, Kylene. 1998. Listen while you read: Struggling readers and audiobooks. School Library Journal 44(4). 30–35.Search in Google Scholar

Bitan, Tali, Douglas D. Burman, Tai‐Li Chou, Dong Lu, Nadia E. Cone, Fan Cao, Jordan D. Bigio & James R. Booth. 2007. The interaction between orthographic and phonological information in children: An fMRI study. Human Brain Mapping 28(9). 880–891.10.1002/hbm.20313Search in Google Scholar

Blok, Henk. 1999. Reading to young children in educational settings: A meta-analysis of recent research. Language Learning 49(2). 343–371.10.1111/0023-8333.00091Search in Google Scholar

Blum, Irene H., Patricia S. Koskinen, Nancy Tennant, E. Marie Parker, Mary Straub & Christine Curry. 1995. Using audiotaped books to extend classroom literacy instruction into the homes of second-language learners. Journal of Reading Behavior 27(4). 535–563.10.1080/10862969509547898Search in Google Scholar

Braze, David, W. Einar Mencl, Whitney Tabor, Kenneth R. Pugh, R. Todd Constable, Robert K. Fulbright, James S. Magnuson, Julie A. Van Dyke & Donald P. Shankweiler. 2011. Unification of sentence processing via ear and eye: An fMRI study. Cortex 47(4). 416–431.10.1016/j.cortex.2009.11.005Search in Google Scholar

Bristow, Davina, Ghislaine Dehaene-Lambertz, Jeremie Mattout, Catherine Soares, Teodora Gliga, Sylvain Baillet & Jean-François Mangin. 2009. Hearing faces: How the infant brain matches the face it sees with the speech it hears. Journal of Cognitive Neuroscience 21(5). 905–921.10.1162/jocn.2009.21076Search in Google Scholar

Brown, Ronan, Rob Waring & Sangrawee Donkaewbua. 2008. Incidental vocabulary acquisition from reading, reading-while-listening, and listening to stories. Reading in a Foreign Language 20(2). 136–163.Search in Google Scholar

Bulkin, David A. & Jennifer M. Groh. 2006. Seeing sounds: Visual and auditory interactions in the brain. Current Opinion in Neurobiology 16(4). 415–419.10.1016/j.conb.2006.06.008Search in Google Scholar

Bus, Adriana G., Marinus H. Van Ijzendoorn & Anthony D. Pellegrini. 1995. Joint book reading makes for success in learning to read: A meta-analysis on intergenerational transmission of literacy. Review of Educational Research 65(1). 1–21.10.3102/00346543065001001Search in Google Scholar

Calvert, Gemma A. A, Edward T. Bullmore, Michael J. Brammer, Ruth Campbell, Steven CR Williams, Philip K. McGuire, Peter WR Woodruff, Susan D. Iversen & David Anthony S. 1997. Activation of auditory cortex during silent lipreading. Science 276(5312). 593–596.10.1126/science.276.5312.593Search in Google Scholar

Campbell, Ruth. 2008. The processing of audio-visual speech: Empirical and neural bases. Philosophical Transactions of the Royal Society B: Biological Sciences 363(1493). 1001–1010.10.1098/rstb.2007.2155Search in Google Scholar

Carney, Russell N. & Joel R. Levin. 2002. Pictorial illustrations still improve students’ learning from text. Educational Psychology Review 14(1). 5–26.10.1023/A:1013176309260Search in Google Scholar

Chang, Anna C. S 2009. Gains to L2 listeners from reading while listening vs. listening only in comprehending short stories. System 37(4). 652–663.10.1016/j.system.2009.09.009Search in Google Scholar

Chang, Anna C. S. 2011. The effect of reading while listening to audiobooks: Listening fluency and vocabulary gain. Asian Anthropology 10. 43–64.Search in Google Scholar

Chang, Anna C. S & Sonia Millett. 2014. The effect of extensive listening on developing L2 listening fluency: Some hard evidence. ELT Journal 68(1). 31–40.10.1093/elt/cct052Search in Google Scholar

Chang, Anna C. S & Sonia Millett. 2015. Improving reading rates and comprehension through audio-assisted extensive reading for beginner learners. System 52. 91–102.10.1016/j.system.2015.05.003Search in Google Scholar

Cheung, Him. 1996. Nonword span as a unique predictor of second-language vocabulary language. Developmental Psychology 32(5). 867–873.10.1037/0012-1649.32.5.867Search in Google Scholar

Choi, Jeungok. 2011. Literature review: Using pictographs in discharge instructions for older adults with low-literacy skills. Journal of Clinical Nursing 20(21–22). 2984–2996.10.1111/j.1365-2702.2011.03814.xSearch in Google Scholar

Colonnesi, Cristina, Geert Jan JM Stams, Irene Koster & Marc J. Noom. 2010. The relation between pointing and language development: A meta-analysis. Developmental Review 30(4). 352–366.10.1016/j.dr.2010.10.001Search in Google Scholar

Cotton, Jack Chilton 1935. Normal “visual hearing”. Science 82. 592–593.10.1126/science.82.2138.592Search in Google Scholar

Danan, Martine. 2004. Captioning and subtitling: Undervalued language learning strategies. Meta: Journal des traducteurs Meta: Translators’ Journal 49(1). 67–77.10.7202/009021arSearch in Google Scholar

David., McNeill 1992. Hand and mind: What gesture reveals about thought. Chicago: University of Chicago Press.Search in Google Scholar

Davis, Chris & Jeesun Kim. 1998. Repeating and Remembering Foreign Language Words: Does Seeing Help? AVSP’98 International Conference on Auditory-Visual Speech Processing. The University of Melbourne.Search in Google Scholar

Day, Richard R. & Julian Bamford. 1998. Extensive reading in the second language classroom. Cambridge: Cambridge University Press.10.1177/003368829802900211Search in Google Scholar

Esteve-Gibert, Nuria & Pilar Prieto. 2014. Infants temporally coordinate gesture-speech combinations before they produce their first words. Speech Communication 57. 301–316.10.1016/j.specom.2013.06.006Search in Google Scholar

Fevre, Deidre M., Dennis W. Moore & Ian AG Wilkinson. 2003. Tape-assisted reciprocal teaching: Cognitive bootstrapping for poor decoders. British Journal of Educational Psychology 73(1). 37–58.10.1348/000709903762869905Search in Google Scholar

Goolkasian, Paula & Paul W. Foos. 2005. Bimodal format effects in working memory. The American Journal of Psychology 118(1). 61–78.10.2307/30039043Search in Google Scholar

Guillory, Helen. 1998. The effects of key word captions to authentic French video in foreign language instruction. CALICO Journal 15(1–3). 89–108.10.1558/cj.v15i1-3.89-108Search in Google Scholar

Gyselinck, Valérie, M. F Ehrlich, Cesare Cornoldi, Rossana De Beni & Véronique Dubois. 2000. Visuospatial working memory in learning from multimedia systems. Journal of Computer Assisted Learning 16(2). 166–176.10.1046/j.1365-2729.2000.00128.xSearch in Google Scholar

Habets, Boukje, Sotaro Kita, Zeshu Shao, Asli Özyurek & Peter Hagoort. 2011. The role of synchrony and ambiguity in speech–gesture integration during comprehension. Journal of Cognitive Neuroscience 23(8). 1845–1854.10.1162/jocn.2010.21462Search in Google Scholar

Hahn, Noemi, John J. Foxe & Sophie Molholm. 2014. Impairments of multisensory integration and cross-sensory learning as pathways to dyslexia. Neuroscience and Biobehavioral Reviews 47. 384–392.10.1016/j.neubiorev.2014.09.007Search in Google Scholar

Holle, Henning & Thomas C. Gunter. 2007. The role of iconic gestures in speech disambiguation: ERP evidence. Journal of Cognitive Neuroscience 19(7). 1175–1192.10.1162/jocn.2007.19.7.1175Search in Google Scholar

Horst, Marlise, Tom Cobb, Thomas Cobb & Paul Meara. 1998. Beyond a clockwork orange: Acquiring second language vocabulary through reading. Reading in a Foreign Language 11(2). 207–223.Search in Google Scholar

Hostetter, Autumn B. 2011. When do gestures communicate? A meta-analysis. Psychological Bulletin 137(2). 297–311.10.1037/a0022128Search in Google Scholar

Hubbard, Amy L., Stephen M. Wilson, Daniel E. Callan & Mirella Dapretto. 2009. Giving speech a hand: Gesture modulates activity in auditory cortex during speech perception. Human Brain Mapping 30(3). 1028–1037.10.1002/hbm.20565Search in Google Scholar

Jacquemot, Charlotte, Christophe Pallier, Denis LeBihan, Stanislas Dehaene & Emmanuel Dupoux. 2003. Phonological grammar shapes the auditory cortex: A functional magnetic resonance imaging study. The Journal of Neuroscience 23(29). 9541–9546.10.1523/JNEUROSCI.23-29-09541.2003Search in Google Scholar

Jäncke, Lutz & Nadim Joni Shah. 2004. Hearing syllables by seeing visual stimuli. European Journal of Neuroscience 19(9). 2603–2608.10.1111/j.0953-816X.2004.03350.xSearch in Google Scholar

Jobard, Gael, Mathieu Vigneau, Bernard Mazoyer & Nathalie Tzourio-Mazoyer. 2007. Impact of modality and linguistic complexity during reading and listening tasks. Neuroimage 34(2). 784–800.10.1016/j.neuroimage.2006.06.067Search in Google Scholar

Kawase, Tetsuaki, Shuichi Sakamoto, Yoko Hori, Atsuko Maki, Yôiti Suzuki & Toshimitsu Kobayashi. 2009. Bimodal audio–visual training enhances auditory adaptation process. Neuroreport 20(14). 1231–1234.10.1097/WNR.0b013e32832fbef8Search in Google Scholar

Kellerman, Susan. 1990. Lip service: The contribution of the visual modality to speech perception and its relevance to the teaching and testing of foreign language listening comprehension. Applied Linguistics 11(3). 272–280.10.1093/applin/11.3.272Search in Google Scholar

Kelly, Spencer D., Corinne Kravitz & Michael Hopkins. 2004. Neural correlates of bimodal speech and gesture comprehension. Brain and Language 89(1). 253–260.10.1016/S0093-934X(03)00335-3Search in Google Scholar

Kelly, Spencer D., Tara McDevitt & Megan Esch. 2009. Brief training with co-speech gesture lends a hand to word learning in a foreign language. Language and Cognitive Processes, 24(2). 313–334.10.1080/01690960802365567Search in Google Scholar

Kelly, Spencer D., Aslı Özyürek & Eric Maris. 2010. Two sides of the same coin: Speech and gesture mutually interact to enhance comprehension. Psychological Science 21(2). 260–267.10.1177/0956797609357327Search in Google Scholar

Krahmer, Emiel & Marc Swerts. 2005. How children and adults produce and perceive uncertainty in audiovisual speech. Language and Speech 48(1). 29–53.10.1177/00238309050480010201Search in Google Scholar

Latifi, Mehdi, Ali Mobalegh & Elham Mohammadi. 2011. Movie subtitles and the improvement of listening comprehension ability: Does it help? The Journal of Language Teaching and Learning 1(2). 18–29.Search in Google Scholar

Levie, W. Howard & Richard Lentz. 1982. Effects of text illustrations: A review of research. ECTJ 30(4). 195–232.10.1007/BF02765184Search in Google Scholar

Lewis, David. 2001. Reading contemporary picturebooks: Picturing text. New York: Routledge.Search in Google Scholar

Lewis, Gwyneth & David Poeppel. 2014. The role of visual representations during the lexical access of spoken words. Brain and Language 134. 1–10.10.1016/j.bandl.2014.03.008Search in Google Scholar

Littleton, Karen, Clare Wood & Pav Chera. 2006. Interactions with talking books: Phonological awareness affects boys’ use of talking books. Journal of Computer Assisted Learning 22(5). 382–390.10.1111/j.1365-2729.2006.00183.xSearch in Google Scholar

MacSweeney, Mairéad, Edson Amaro, Gemma Calvert, Ruth Campbell, Anthony David, Philip McGuire, Steve Williams, Bencie Woll & Michael Brammer. 2000. Silent speechreading in the absence of scanner noise: An event-related fMRI study. NeuroReport 11(8). 1729–1733.10.1097/00001756-200006050-00026Search in Google Scholar

Man, Kingson, Jonas T. Kaplan, Antonio Damasio & Kaspar Meyer. 2012. Sight and sound converge to form modality-invariant representations in temporoparietal cortex. The Journal of Neuroscience 32(47). 16629–16636.10.1523/JNEUROSCI.2342-12.2012Search in Google Scholar

Markham, Peter L. 1993. Captioned television videotapes: Effects of visual support on second language comprehension. Journal of Educational Technology Systems 21(3). 183–191.10.2190/59JJ-0RBN-KKVL-C2YGSearch in Google Scholar

Marsh, Emily E. & Marilyn Domas White. 2003. A taxonomy of relationships between images and text. Journal of Documentation 59(6). 647–672.10.1108/00220410310506303Search in Google Scholar

Masoura, Elvira V. & Susan E. Gathercole. 1999. Phonological short-term memory and foreign language learning. International Journal of Psychology 34(5–6). 383–388.10.1080/002075999399738Search in Google Scholar

Mastroberardino, Serena, Valerio Santangelo, Fabiano Botta, Francesco S. Marucci & Marta Olivetti Belardinelli. 2008. How the bimodal format of presentation affects working memory: An overview. Cognitive Processing 9(1). 69–76.10.1007/s10339-007-0195-6Search in Google Scholar

Mayer, Richard E. & Roxana Moreno. 2003. Nine ways to reduce cognitive load in multimedia learning. Educational Psychologist 38(1). 43–52.10.1207/S15326985EP3801_6Search in Google Scholar

McGurk, Harry & John MacDonald. 1976. Hearing lips and seeing voices. Nature 264. 746–748.10.1038/264746a0Search in Google Scholar

Menne, Joy M. & John W. Menne. 1972. The relative efficiency of bimodal presentation as an aid to learning. Educational Technology Research and Development 20(2). 170–180.10.1007/BF02768415Search in Google Scholar

Miller, George A. 1956. The magical number seven, plus or minus two: Some limits on our capacity for processing information. Psychological Review 63(2). 81–97.10.1037/h0043158Search in Google Scholar

Mol, Suzanne E. & Adriana G. Bus. 2011. To read or not to read: A meta-analysis of print exposure from infancy to early adulthood. Psychological Bulletin 137(2). 267–296.10.1037/a0021890Search in Google Scholar

Mol, Suzanne E., Adriana G. Bus & Maria T. De Jong. 2009. Interactive book reading in early education: A tool to stimulate print knowledge as well as oral language. Review of Educational Research 79(2). 979–1007.10.3102/0034654309332561Search in Google Scholar

Mol, Suzanne E., Adriana G. Bus, Maria T. De Jong & Daisy J. Smeets. 2008. Added value of dialogic parent–child book readings: A meta-analysis. Early Education and Development 19(1). 7–26.10.1080/10409280701838603Search in Google Scholar

Montali, Julie & Lawrence Lewandowski. 1996. Bimodal reading: Benefits of a talking computer for average and less skilled readers. Journal of Learning Disabilities 29(3). 271–279.10.1177/002221949602900305Search in Google Scholar

Nikolajeva, Maria & Carole Scott. 2001. How picturebooks work. New York: Routledge.Search in Google Scholar

Papagno, Costanza, Tim Valentine & Alan Baddeley. 1991. Phonological short-term memory and foreign-language vocabulary learning. Journal of Memory and Language 30(3). 331–347.10.1016/0749-596X(91)90040-QSearch in Google Scholar

Peterson, Matthew, Kevin Wise, Yilin Ren, Zongyuan Wang & Jiachen Yao. 2017. Memorable metaphor: How different elements of visual rhetoric affect resource allocation and memory for advertisements. Journal of Current Issues and Research in Advertising 38(1). 65–74.10.1080/10641734.2016.1233155Search in Google Scholar

Pujolà, Joan-Tomàs. 2002. CALLing for help: Researching language learning strategies using help facilities in a web-based multimedia program. ReCALL 14(2). 235–262.10.1017/S0958344002000423Search in Google Scholar

Rasinski, Timothy V. 1990. Effects of repeated reading and listening-while-reading on reading fluency. The Journal of Educational Research 83(3). 147–151.10.1080/00220671.1990.10885946Search in Google Scholar

Reisberg, Daniel, John Mclean & Anne Goldfield. 1987. Easy to hear but hard to understand: A lip-reading advantage with intact auditory stimuli. In Barbara Dodd & Ruth Campbell (eds.) Hearing by eye: The psychology of lip-reading, 97–114. Hillsdale, NJ: Earlbaum.Search in Google Scholar

Ross, Lars A., Dave Saint-Amour, Victoria M. Leavitt, Daniel C. Javitt & John J. Foxe. 2007. Do you see what I am saying? Exploring visual enhancement of speech comprehension in noisy environments. Cerebral Cortex 17(5). 1147–1153.10.1093/cercor/bhl024Search in Google Scholar

Scarborough, Hollis S. & Wanda Dobrich. 1994. On the efficacy of reading to pre-schoolers. Developmental Review 14(3). 245–302.10.1006/drev.1994.1010Search in Google Scholar

Sénéchal, Monique & Laura Young. 2008. The effect of family literacy interventions on children’s acquisition of reading from kindergarten to grade 3: A meta-analytic review. Review of Educational Research 78(4). 880–907.10.3102/0034654308320319Search in Google Scholar

Service, Elisabet. 1992. Phonology, working memory, and foreign-language learning. The Quarterly Journal of Experimental Psychology 45(1). 21–50.10.1080/14640749208401314Search in Google Scholar

Shankweiler, Donald W., Einar Mencl, David Braze, Whitney Tabor, Kenneth R. Pugh & Robert K. Fulbright. 2008. Reading differences and brain: Cortical integration of speech and print in sentence processing varies with reader skill. Developmental Neuropsychology 33(6). 745–775.10.1080/87565640802418688Search in Google Scholar

Sipe, Lawrence R. 1998. How picture books work: A semiotically framed theory of text-picture relationships. Children’s Literature in Education 29(2). 97–108.10.1023/A:1022459009182Search in Google Scholar

Skipper, Jeremy I., Virginie Van Wassenhove, Howard C. Nusbaum & Steven L. Small. 2007. Hearing lips and seeing voices: How cortical areas supporting speech production mediate audiovisual speech perception. Cerebral Cortex 17(10). 2387–2399.10.1093/cercor/bhl147Search in Google Scholar

Stewart, Melissa & Inmaculada Petusa. 2004. Gains to language learners from viewing target language closed-captioned films. Foreign Language Annals 37(3). 438–442.10.1111/j.1944-9720.2004.tb02701.xSearch in Google Scholar

Sueyoshi, Ayano & Debra M. Hardison. 2005. The role of gestures and facial cues in second language listening comprehension. Language Learning 55(4). 661–699.10.1111/j.0023-8333.2005.00320.xSearch in Google Scholar

Sumby, William H. & Irwin Pollack. 1954. Visual contribution to speech intelligibility in noise. The Journal of the Acoustical Society of America 26(2). 212–215.10.1121/1.1907309Search in Google Scholar

Taylor, Gregory. 2005. Perceived processing strategies of students watching captioned video. Foreign Language Annals 38(3). 422–427.10.1111/j.1944-9720.2005.tb02228.xSearch in Google Scholar

Van Atteveldt, Nienke, Alard Roebroeck & Rainer Goebel. 2009. Interaction of speech and script in human auditory cortex: Insights from neuro-imaging and effective connectivity. Hearing Research 258(1). 152–164.10.1016/j.heares.2009.05.007Search in Google Scholar

Van Atteveldt, Nienke M., Elia Formisano, Leo Blomert & Rainer Goebel. 2007. The effect of temporal asynchrony on the multisensory integration of letters and speech sounds. Cerebral Cortex 17(4). 962–974.10.1093/cercor/bhl007Search in Google Scholar

Vandergrift, Larry & Jeremy Cross. 2014. Captioned video: How much listening is really going on? Contact 40(3). 31–33.Search in Google Scholar

Vanderplank, Robert. 1988. The value of teletext sub-titles in language learning. ELT Journal 42(4). 272–281.10.1093/elt/42.4.272Search in Google Scholar

Verhagen, Josje, Paul Leseman & Marielle Messer. 2015. Phonological memory and the acquisition of grammar in child L2 learners. Language Learning 65(2). 417–448.10.1111/lang.12101Search in Google Scholar

Vogel, Edward K., Geoffrey F. Woodman & Steven J. Luck. 2001. Storage of features, conjunctions, and objects in visual working memory. Journal of Experimental Psychology: Human Perception and Performance 27(1). 92–114.10.1037/0096-1523.27.1.92Search in Google Scholar

Wagner, Petra, Zofia Malisz & Stefan Kopp. 2014. Gesture and speech in interaction: An overview. Speech Communication 57. 209–232.10.1016/j.specom.2013.09.008Search in Google Scholar

Werner, Sebastian & Uta Noppeney. 2010. Superadditive responses in superior temporal sulcus predict audiovisual benefits in object categorization. Cerebral Cortex, 20(8). 1829–1842.10.1093/cercor/bhp248Search in Google Scholar

Willems, R. M., A. Özyürek & P. Hagoort, 2007. When language meets action: The neural integration of gesture and speech. Cerebral Cortex 17(10). 2322–2333.10.1093/cercor/bhl141Search in Google Scholar

Winke, Paula, Susan Gass & Tetyana Sydorenko. 2010. The effects of captioning videos used for foreign language listening activities. Language Learning and Technology 14(1). 65–86.Search in Google Scholar

Yanguas, Iñigo. 2009. Multimedia glosses and their effect on L2 text comprehension and vocabulary learning. Language Learning and Technology 13(2). 48–67.Search in Google Scholar