PhD School in Psychology

Department of Cognitive Science

Supervisor: Anna Babarczy

The Sound-Language Interface: The Processing of Concrete and Abstract Sound-Related Language

Introducing the research area

Auditory imagery is an auditory experience of a sound event constructed from long-term memory in the absence of direct instigation of that experience. The nature of auditory imagery associated with concrete (e.g., The boy rang the bell) and abstract (e.g., The press rang alarm bells) sound-related language is targeted. The representation of abstract sound-related language is scarcely investigated and timely area in cognitive linguistics. The major objective of this study is to compare the usage of concrete and abstract sound-related languages in terms of activation of auditory imagery. Activating auditory imagery means building a representation of the sound described in a sentence. The study also investigates the circumstances under which language users construct auditory images to ground their understanding of sound events as conveyed by language.

Brief introduction of the research place

The research facility is located at the Department of Cognitive Science. Its primary tasks include researches in psycholinguistics, child language acquisition, memory and learning. The department also provides elective courses, BA courses and MA courses in cognitive science.

History and context of the research

Modality-specific theories of cognition propose that the processing of sound-related language can activate auditory imagery under certain task conditions (e.g., Barsalou, 1999). This prediction has been confirmed by several experiments (e.g., Kemmerer et al., 2008; Kiefer et al., 2008). Radical embodiment theories also suggest that concrete and abstract concepts are grounded in sensorimotor systems of the brain (Lakoff and Johnson, 1999). Two questions, however, have remained unanswered: (1) under what task conditions construct language users auditory imagery, and (2) is there a difference between concrete and abstract sound-related languages in terms of activation of auditory imagery?

Aim of the research

The study has two major objectives: (1) to determine under what task conditions can auditory imagery be activated when processing concrete and abstract sound-related languages and (2) to examine the accessibility of concrete representations in abstract sound-related situation models at the deep processing level (Barsalou, 1999; Louwerse and Jeaniaux, 2008). Deep processing refers to the activation of non-linguistic mental simulations as a result of task demand (deep conceptual processing).

Methods

Computer reaction time study of concrete and abstract languages use E-Prime software. All the experiments presented in the next section (Results) used different types of sound stimuli: congruent (matching), incongruent (mismatching) and unrelated (category-external) sounds. The three experimental groups (congruent, incongruent, unrelated sound condition) were manipulated between subjects. The participants performed different tasks in the three experiments reported in Results: in Experiment 1, the participants were making sensibility judgements on visually presented sentences, in Experiment 2, different groups of participants verified sounds in a sentence-sound matching task and in Experiment 3, participants read sentences word-by-word in a self-paced manner.

Sound Stimuli: consisted of everyday objects, animal sounds or sounds of musical instruments. The sound files were used in a different study by Marcell, Borella, Greene, Kerr & Rogers (2000).

Sentence Stimuli: examples of the critical sentences of Experiment 1 can be seen below. Experiments 2 and 3 also used the sentences of Experiment 1 with minor modifications. Congruent, incongruent and unrelated types of sound examples are presented here together with the sentences.

Procedure: 

In the following, three experiments are presented that investigate the sound-language interface. Experiment 1 and 3 investigated whether or not sound stimuli affect linguistic processing. Experiment 2 used a more explicit task: participants had to consciously compare language stimuli to sound stimuli.

Experiment 1: Participants were asked to read the sentences appearing on the computer screen and press the ENTER key if they thought the sentence made sense or the SPACE key if they did not think so. They were also instructed not to pay attention to the sounds they would hear during the sentences. The sentence appeared in the centre of the computer screen in synchrony with the sound. The sentences appeared one after the other in random order. The sounds were presented binaurally and continuously until the subject responded by pressing either of the two keys. There was no limit on response time, i.e., subjects could spend as much time reading the sentences and making their sensibility judgements as they wished, however, they were also asked to respond to the sentences as quickly as possible.

Experiment 2: Participants were asked to read sentences word by word at a fixed rate (250 ms/word), however, they could not return to previously read words. Upon reaching the end of the sentences (in synchrony with the last word) they heard a sound; their task was to decide as quickly as possible whether or not this sound could have been produced by the event described by the sentence. Crucially, participants had to give a negative response to sound stimuli after abstract sentences, since the events described by these sentences could not have produced any concrete sound whatsoever (e.g., the event described by the sentence "The press rang alarm bells" does not imply a sound from a ringing bell, therefore participants were required to respond with "no" to such sentences). A practice phase preceded the actual tasks.

Experiment 3: Participants were asked to read sentences one word at a time and they could move to the next word by pressing a key when they were ready, however, they could not return to previously read words (self-paced reading paradigm). At the end of the sentences they heard a sound (in synchrony with the last word); their task was to read the sentences and answer a control question after each sentence. Critical sound stimuli were presented and played until the participant finished reading the sentence.

Participants: all participants were Hungarian university students of BME (Budapest University of Technology and Economics). Participants received course credit for completing the experiments.

Results

cf. Fekete and Babarczy (submitted, 2010): A psycholinguistic analysis of 'fictive' sound events,  Lingua.

In Experiment 1, mean sensibility judgement times were measured. Results show that sentences were processed faster in the congruent condition compared to the incongruent condition in both sub-samples (concrete or abstract). Mean reading times of sentences in the unrelated condition did not differ significantly from the no sound condition. The following figure illustrates the mean reading times  of the critical sentences in milliseconds. Linking bars represent levels of significance.

In Experiment 2, mean sound verification times were measured. Participants reacted to unrelated sounds significantly faster than to congruent or incongruent sounds (reverse facilitation effect). Incongruent sounds were verified significantly slower than congruent sounds in the concrete sub-sample of critical sentences. However, there was no significant difference in RTs between the congruent and incongruent conditions for the abstract sentences. Incongruent sounds were verified significantly slower when presented together with concrete sentences than with abstract ones. The following figure illustrates the mean verification times of the sounds in milliseconds.

In Experiment 3, reading times were measured in the region of the verbs (last words). Sentences in the congruent condition were not processed differently from those in the incongruent condition. Unrelated (category-external) sounds significantly inhibited processing relative to related (category-internal) sounds. Sound-related language therefore does not automatically evoke specific auditory imagery during the reading task judging by the lack of congruency-effect (congruent items processed faster than incongruent ones). Rather, category-internal sounds (congruent or incongruent) automatically prime (or, both are compatible with) linguistic processing in a spreading activation manner (see, Anderson, 1983; Neely, 1991). The following figure illustrates the mean reading times of the critical verbs in milliseconds in the sound conditions.

This finding is therefore inconsistent with the claim that modality-specific representations are inherent part of conceptual representations in the auditory domain (Kiefer et al., 2008).  Data provided evidence against Radical Embodiment theories (Lakoff and Johnson, 1999). It is shown that, even in the concrete domain, (shallow) language processing depends on linguistic forms (the language system), rather than on non-linguistic mental models.

Future research will follow up on Experiment 2 with the following four modifications:

1.) using within-subject experimental design to rule out the potential use of strategies

2.) reducing task demand in the abstract-congruent condition by asking participants to think about the literal meaning of idioms and respond positively to such trials (for a similar explicit task instruction, cf. Mashal, 2008)

3.) using the same verbs in the abstract condition as in the concrete condition to enable verb-item comparisons

4.) presentation of linguistic material and sound stimulus in a temporal asynchrony instead of the simultaneous presentation design.

Expected impact and further research

This research will provide a better understanding of the status of concrete representations in abstract situation models. Importantly, it has been shown that when "deep-processing" abstract sentences the irrelevant feature (i.e. the concrete sound representation) is diminished rather than (i) activated to the same extent as to the concrete sentences, (ii) inhibited or (iii) not activated at all. The study has implications for abstract situation models in other domains, such as motor representations (metaphors describing motor actions, e.g., grasp the idea) or spatial representations (e.g., the prices went up).

Publications, references, links

Anderson, J.R., 1983: A spreading activation theory of memory. Journal of Verbal Learning and Verbal Behavior 22, pp. 261–295

Barsalou, L.W., 1999: Perceptual symbol systems. Behavioral and Brain Sciences 22. pp. 577–660

Fekete, I., Babarczy, A. (submitted, 2010): A psycholinguistic analysis of "fictive" sound events, Lingua

Kemmerer, D., Castillo, J.G., Talavage, T., Patterson, S., Wiley, C., 2008: Neuroanatomical distribution of five semantic components of verbs: evidence from fMRI. Brain and Language 107 (1). pp. 16–43

Kiefer, M., Sim, E.J., Herrnberger, B., Grothe, J., Hoenig, K., 2008: The sound of concepts: four markers for a link between auditory and conceptual brain systems. The Journal of Neuroscience 28 (47). 12224–12230

Lakoff, G., Johnson, M., 1999: Philosophy in the flesh: The embodied mind and its challenge to western thought. New York, NY: Basic Books

Louwerse, M.M., Jeaniaux, P., 2008: How fundamental is embodiment to language comprehension? Constraints on embodied cognition. In: B.C. Love, K. McRae, & V.M. Sloutsky (Eds.), Proceedings of the 30th Annual Conference of the Cognitive Science Society (pp. 1313–1318). Austin, TX: Cognitive Science Society

Marcell, M.M., Borella, D., Greene, M., Kerr, E., Rogers, S., 2000: Confrontation naming of environmental sounds. Journal of Clinical and Experimental Neuropsychology 22, pp. 830–864

Mashal, N., Faust, M., Hendler, T., Jung-Beeman, M., 2008: Hemispheric differences in processing the literal interpretation of idioms: Converging evidence from behavioral and fMRI studies. Cortex 44 (7), pp. 848–860

Neely, J.H., 1991: Semantic priming effects in visual word recognition: A selective review of current findings and theories. In: D. Besner & G. Humphreys (Eds.), Basic processes in reading: Visual word recognition (pp. 236–264), Hillsdale, NJ: Erlbaum

Other Personal Publications

Babarczy, A., Bencze, I., Fekete, I., Simon, E. (in press): A metaforikus nyelvhasználat egy korpuszalapú elemzése [A corpus-based analysis of metaphoric language use], Általános Nyelvészeti Tanulmányok [Studies in General Linguistics]

Fekete, I., Pléh, Cs. (in press): Bidirectional and Unidirectional Comitative Constructions in Hungarian: a Psycholinguistic Investigation at the Interface of Argument Structure and Semantics, Acta Linguistica Hungarica

Fekete, I. (2010): A nyelvi szemantika a kognitív tudomány perspektívájából  [Linguistic semantics from a cognitive science perspective], Magyar Pszichológiai Szemle [Hungarian Journal of Psychology], Vol. 65. (2), pp. 383–416

Fekete, I., Babarczy, A. (in press): Főnévi fogalmak konkrétsági, elképzelhetőségi és definiálhatósági értékeinek összefüggései [Concreteness, imageability and definability values of noun concepts], Magyar Pszichológiai Szemle [Hungarian Journal of Psychology]

Fekete, I., Pléh, Cs. (in press): "Ne viccelődj a rendőrökkel": egy- és kétirányú társas viszonyok a nyelvben [Don’t joke around with policemen: uni- and bidirectional social relations in language], Magyar Pszichológiai Szemle [Hungarian Journal of Psychology].

Fekete, I., Pléh, Cs. (2008): A nyelvi relativizmus és az idegrendszeri plaszticitás [Linguistic relativism and neuroplasticity], Magyar Pszichológiai Szemle [Hungarian Journal of Psychology], Vol. 63. (1), pp. 181–211

Fekete, I., Pléh, Cs. (2008): A nyelvi relativizmus és az idegrendszeri plaszticitás [Linguistic relativism and neuroplasticity], In Pléh, Cs. (szerk.): A fejlődési plaszticitás és az idegrendszer, Akadémiai Kiadó, Budapest, pp.181–211

Fekete, I. (2007): Könyvismertetés Melvyn A. Goodale és A. David Milner: Sight Unseen: An Exploration of Conscious and Unconscious Vision c. könyvéről. (Oxford University Press, 2004) Magyar Pszichológiai Szemle [Hungarian Journal of Psychology], Vol 62. (4), pp. 571–573 (2007)

Fekete, I. (2007): Könyvismertetés Diane Pecher és Rolf A. Zwaan (szerk.): Grounding Cognition: The Role of Perception and Action in Memory, Language, and Thinking  c. könyvéről. (Cambridge University Press, 2005) Magyar Pszichológiai Szemle [Hungarian Journal of Psychology], Vol 62. (2), pp. 253–256 (2007)