What role does the body play with language? My research investigates this question from different perspectives using different methods, but it is built on a single theoretical framework. Working from the position that language is an embodied ability that evolved from bodily communication systems (gestures, facial expressions, eye gaze, etc.) in our evolutionary past, my work investigates how language interacts with the body in present-day communication.
I focus primarily on hand gestures that spontaneously accompany speech. These gestures are interesting because they occur simultaneously with speech but reflect meaning in a distinct way from words.
For example, the form and movement of different drinking gestures—a gentle movement with a small precision grasp depicts sipping from a sake cup whereas a more abrupt movement with a closed fist depicts drinking from a large beer mug—capture the meaning of these two actions in a direct
way. In contrast, spoken words reflect meaning only indirectly
through the particular conventions of a language. For example, the words, “nomu” in Japanese, and, “drink” in English, are utterly unrelated to the actual act of imbibing. This difference is exactly why co-speech gesture is so interesting—it offers a direct visual complement to the conventional symbols of a language, and when combined with those symbols, provides a more complete “picture” of what a speaker means.
My research on the relationship between speech and gesture spans social, psychological, and neural levels. On the social and psychological levels, I use behavioral methods to demonstrate that gestures influence how children and adults comprehend language in different social contexts, and how they think during the explanation of difficult concepts. On the neural level, I use event-related potentials (ERPs) to show that gestures influence speech at multiple stages of language comprehension. See below for PDFs on these different topics.
My research and a paper
I co-authored with a Colgate student were cited in a BBC article
that focused on body movements and how they can help you learn. For a general background on gesture, see this article in Scientific American Mind
. For more on second-language learning and instruction, visit Colgate's Center for Language and Brain
If you are interested in more research on gesture, brain, and language, click here for a link to a special issue (June, 2007) on the topic in the journal, Brain and Language
. Or visit the webpage for the International Society for Gesture Studies (ISGS
) to learn more about the field of gesture research more broadly. For more on my work and the work of my collaborators, please visit my page on Research Gate
BA, Washington University, St. Louis, 1991; MA (1997), PhD (1999), University of Chicago
for complete CV.
Language Learning and the Liberal Arts
Dr. Kelly is the co-director of Colgate's Center for Language and Brain
(CLB) which focuses on, among other things, issues involved in foreign language instruction and learning. Check out the video below to learn how the CLB promotes the psychological and social benefits of learning a foreign language at Colgate.
(Hyperlinks lead to PDF files or free download sites)
- Hirata, Y., Kelly, S. D., *Huang, J., & *Manansala, M. (2014). Effects of hand gestures on auditory learning of second-language vowel length contrasts. Journal of Speech, Language, and Hearing Research, 57(6), 2090-2101.
- Holler, J., Schubotz, L., Kelly, S., Hagoort, P., Schuetze, M. & Özyürek, A. (2014). Social eye gaze modulates processing of speech and co-speech gesture. Cognition, 132, 696-697.
- Kelly, S., *Healey, M., Özyürek, A., & Holler, J. (2014). The processing of speech, gesture, and action during language comprehension. Psychonomic Bulletin & Review, 1-7.
- Kelly, S., Hirata, Y., *Manansala, M., & *Huang, J. (2014). Exploring the role of hand gestures in learning novel phoneme contrasts and vocabulary in a second language. Frontiers in Psychology, 5, 673.
- Holler, J., Kokal, I., Toni, I., Hagoort, P., Kelly, S., & Özyürek, A. (2014). Eye'm talking to you: speakers' gaze direction modulates co-speech gesture processing in the right MTG. Social Cognitive & Affective Neuroscience.
- Church, R. B., Kelly, S., & Holcombe, D. (2013).Temporal synchrony between speech, action and gesture during language production. Language and Cognitive Processes, 1-10.
- *Hinzman, L., & Kelly, S. D. (2013). Effects of emotional body language on rapid out-group judgments. Journal of Experimental Social Psychology, 49, 152–155.
- Kelly S. D., Hansen B. C., & *Clark D. T. (2012). “Slight” of hand: The processing of visually degraded gestures with speech. PLoS ONE 7(8): e42620.
- Kelly, S. D., & *Lee, A. (2012). When actions speak too much louder than words: Gesture disrupts word learning when phonetic demands are high. Language and Cognitive Processes, 27, 793-807.
- Kelly, S. D., *Byrne, K., & Holler, J. (2011). Raising the ante of communication: Evidence for enhanced gesture use in high stakes situations. Information, 2, 579-593.
- Kelly, S. D., Ozyurek, A., & Maris, E. (2010). Two sides of the same coin: Speech and gesture mutually interact to enhance comprehension. Psychological Science, 21, 260-267.
- Kelly, S. D., *Creigh, P., & *Bartolotti, J. (2010). Integrating speech and iconic gestures in a Stroop-like task: Evidence for automatic processing. Journal of Cognitive Neuroscience, 22, 683-694.
- Hirata, Y., & Kelly, S. (2010). The effects of lips and hands on auditory learning of second language speech sounds. Journal of Speech, Language and Hearing Research, 53, 298-310.
- Kelly, S. D., *McDevitt, T., & *Esch, M. (2009). Brief training with co-speech gesture lends a hand to word learning in a foreign language. Language and Cognitive Processes, 24, 313-334.
- *Pratt, N., & Kelly, S. D. (2008). Emotional states influence the neural processing of affective language, Social Neuroscience, 3, 434-442.
- Kelly, S. D., Manning, S., & *Rodak, S. (2008). Gesture gives a hand to language and learning: Perspectives from cognitive neuroscience, developmental psychology and education. Language and Linguistics Compass, 2, 569-588.
- Ozyurek, A. & Kelly, S. D. (2007). Gesture, brain and language. Brain and Language, 101, 181-184.
- Kelly, S. D., *Ward, S., *Creigh, P., & *Bartolotti, J. (2007). An intentional stance modulates the integration of gesture and speech during comprehension. Brain and Language, 101, 222-233.
- Molfese, D. L., Fonaryova Key, A. Kelly, S., Cunningham, N., Terrell, S., Fergusson, M., Molfese, V. & Bonebright, T. (2006). Below-average, average, and above-average readers engage different and similar brain regions while reading. Journal of Learning Disabilities, 39, 352-363.
- Kelly, S. D., & *Goldsmith, L. (2004). Gesture and right hemisphere involvement in evaluating lecture material. Gesture, 4, 25-42.
- Kelly, S. D., *Kravitz, C., & *Hopkins, M. (2004). Neural correlates of bimodal speech and gesture comprehension. Brain and Language, 89, 253-260.
- Molfese, D. L., Molfese, V. J., Key, S., Mogdlin, A. & Kelly, S., & Terrell, S. (2002). Reading and cognitive abilities: Longitudinal studies of brain and behavior changes in young children. Annals of Dyslexia, 52, 121-140.
- Kelly, S. D., Iverson, J., *Terranova, J., *Niego, J., *Hopkins, M., & *Goldsmith, L. (2002). Putting language back in the body: Speech and gesture on three timeframes. Developmental Neuropsychology, 22, 323-349.
- Kelly, S. D., Singer, M., Hicks, J., & Goldin-Meadow, S. (2002). A helping hand in assessing children's knowledge: Instructing adults to attend to gesture. Cognition and Instruction, 20, 1-26.
- Goldin-Meadow, S., Nusbaum, H., Kelly, S. D., & Wagner, S. (2001). Explaining math: Gesture lightens the load. Psychological Science, 12, 516-522.
- Molfese, D., Molfese, V., & Kelly, S. D. (2001). The use of brain electrophysiology techniques to study language and reading: A basic guide for the beginning consumer of electrophysiology information. Learning Disabilities Quarterly, 24 (3), 177-188.
- Kelly, S. D. (2001). Broadening the units of analysis in communication: Speech and nonverbal behaviours in pragmatic comprehension. Journal of Child Language, 28, 325-349.
- Church, R. B., Kelly, S. D., & Lynch, K. (2000). Multi-modal processing over development: The case of speech and gesture detection. Journal of Nonverbal Behavior, 24 (2), 151-174.
- Kelly, S. D., Barr, D., Church, R. B., & Lynch, K. (1999). Offering a hand to pragmatic understanding: The role of speech and gesture in comprehension and memory. Journal of Memory and Language, 40, 577-592.
- Kelly, S. D., & Church, R. B. (1998). A comparison between children's and adults' ability to detect children's representational gestures. Child Development, 69, 85-93.
- Kelly, S. D., & Church, R. B. (1997). Children's ability to detect nonverbal behaviors from other children. Cognition and Instruction, 15(1), 107-134.
* denotes Colgate undergraduate