 |
BMe Research Grant |
|
Doctoral School of Psychology (Cognitive Science)
Department of Cognitive Science
Supervisor: Professor Dr. Csaba Pléh
Metaphor Processing and the Right Hemisphere
Abstract
Understanding a defining human-specific capacity, the language is not the only reason why studying the neural processing of metaphorical expressions is essential. Metaphorical, non-literal and ambiguous layers of meaning pose a serious challenge on machine based language processing, for example, in the case of content – and not keyword – based search. The present research concentrates on linguistic dimensions of processing metaphorical expressions that turn the right hemisphere on, despite the fact that the primary speech centers are located in the left hemisphere. The deeper understanding of how the brain analyzes language could bring more efficient language and content analyzing technologies.
The research institute
At BME’s Cognitive Science Department, researchers of the brain and cognition can work in a unique interdisciplinary environment: natural scientific experimental research methodology is extended by engineering aspects within the framework of cognitive science to pursue a comprehensive understanding of neural processes. One of the language lab’s most important research areas is the study of metaphorical and abstract language.
The history and broader context of the research
There are a number of tools available for machine based language analysis in Hungarian (such as word frequency measures or syntactic analyzers), and in a variety of other languages too, for example co-occurrence dictionaries available for more than 150 languages, association norms, the analyzers of semantic relationships and other psycholinguistic indices and synonym based lexical databases. However, ambiguous and non-literal expressions still remain invisible for machines.
The brain processing of figurative language is likely to require special resources, since besides the left hemisphere’s primary speech centers (Broca's and Wernicke's area) some right hemisphere areas are also involved in the interpretation. The right hemisphere, considered "mute" for decades [1] seems to have a decisive role in processing pragmatics [2], context [3], indirect requests [4], lexical ambiguity [5], jokes [6], irony [7] and metaphors as well [8].
Understanding the neural (sub)systems dealing with the extraction of deeper layers of meaning can bring us closer to the engineering aspect of the question: what are the language factors and processing steps that results understanding in the human brain?
The aim of the study
The most important theoretical frameworks link the right hemispherical processing of metaphorical expressions to the processing of unusual semantic relations [9, 10]. Apparently, only novel metaphors seem to activate the right hemisphere, since conventional ones (such as "stormy relationship" or "booming business") have already been lexicalized owing to their frequent occurrence – in spite of their non-literal interpretation – and hence they are processed by the left hemisphere.
The main research question is whether novel literal expressions, just like novel metaphors, evoke right hemispherical processing advantage. According to the graded salience hypothesis [9], the answer is not, as the literal meaning is always salient, hence processed by the left hemisphere. The right hemisphere is activated only by the non-salient, figurative meaning of novel metaphorical expressions. However, the coarse semantic coding theory [10] proposes that any new expressions (be it literal or not) activate the right hemisphere, as only the right hemisphere’s coarse coding allows for the processing of large semantic distances. The left hemisphere’s fine coding is involved in the processing of narrow semantic relationships between closely associated words.
Figure 1: The coarse semantic coding theory: The left hemisphere’s fine semantic coding involves narrow semantic fields, while the right hemisphere’s coarse semantic coding requires broad semantic fields [10]
The present research is not simply comparing the two models of right hemispherical language processing, but can shed light on the engineering aspects of non-literal language processing by the thorough understanding of the brain’s functioning.
Methods
To examine the right hemisphere’s language capacities, an fMRI and a divided visual field experiment were carried out. The fMRI experiment was realized in cooperation with the research team of the Frei Universität Berlin. This method of indirect measurement of cerebral blood oxygen levels allows the precise localization of changes in brain activity. However, it is not sensitive to the exact temporal aspects of processing, therefore this research was complemented by the divided visual field experiment at BME’s Department of Cognitive Science. In this setting, words are projected either to the left or right visual field, which – due to the crossing of the nasal optic nerve – first reach the contralateral hemisphere. Although information is available relatively quickly to the other hemisphere too, the small, but definite time difference is sufficient to draw conclusions regarding the two hemispheres’ cognitive abilities. Lateralized stimulus presentation was controlled by an eye-tracker.
Figure 3: Neural background of the divided visual field paradigm
Figure 2: An eye-tracker system
Subjects
Forty adult volunteers (20 females) participated in the German experiment, while forty-five adult volunteers (23 women) in the Hungarian one. Since speech centers are less likely to be located in the left hemisphere of left-handed (73%) as compared to right-handed persons (96%), only right-handed, native speakers (of German and Hungarian, respectively) were included in the study, who had no history of neurological or psychiatric disorders. Subjects received course credit for their participation. The studies were run with the permissions of the universities’ ethical committees.
Stimulus material
Because of the right hemisphere’s sensitivity to language, the stimulus material had to meet several conditions. To control for contextual effects, metaphorical (and literal) expressions appeared in isolation. A unique linguistic construct of German language is the flexibly combinable noun-noun compound words, which created an opportunity to have the simplest possible, single-word presentation of metaphorical expressions. Four categories of compound words were involved: conventional metaphors ("chair-leg"), conventional literal expressions ("alarm-signal"), novel metaphors ("rubber-oath") and novel literal phrases ("wooden-cup"). The words’ emotional valence, arousal value, and image-ability – factors possibly influencing the right hemisphere – were all controlled for. The Hungarian stimulus material consisted of four categories of adjective noun word pairs ("brilliant idea" / "ruffled skirt" / "worn idea" / "drawing swing" respectively), controlled for the same psycholinguistic variables.
Experimental designs
For the fMRI experiment in the scanner of the D.I.N.E. lab at Freie Universität Berlin words were displayed on a goggle monitor for two seconds with 6-10 seconds difference, adjusted to the dynamics of cerebral blood flow.
In the eye-tracking experiment – in line with the divided visual field paradigm – adjectives appeared first centrally (400 ms), and after a short break (400 ms) nouns appeared for 180 ms randomly, either to the left or to the right of the central fixation cross at a distance of 1.6° visual angle, in the periphery of vision.
Results
Both experiments have led to surprising results. Apparently, if the numerous psycholinguistic variables are controlled for there is no right hemispherical processing advantage for novel metaphors. Earlier studies might have not measured the effect of figurativeness, but some other, uncontrolled dimension instead. Theories concerning the right hemisphere’s language should most likely not be interpreted on a primarily conceptual but on a contextual level. Further on, processing semantic distance is probably of secondary significance compared to resolving the ambiguity of metaphorical expressions.
Figure 4: fMRI contrast of novelty: All conventional (warm colors) vs. all novel expressions (cold colors)
The fMRI experiment
Interestingly, all conventional noun-noun compound words activated right temporoparietal regions. However, if we take into consideration that the coarse semantic coding theory [10] suggests that the left hemisphere’s fine coding is activated by closely associated semantic relationship, the results are not surprising. In compound words the second constituents are not close associates of the first constituents (e.g. "leg" is not a close associate of "chair"), which requires combinatorial semantic processing of the two words – a kind of function the activated area is responsible for [11]. Novel expressions activated mainly left inferior frontal areas (LIFG), presumably because integrating two words in an unusual way requires a fine coded meaning selection [12] and morpho-syntactic unification [13]. Intriguingly, conventional metaphors – compared to conventional literal expression – activated the LIFG, suggesting that even if they become conventionalized, their interpretation require deeper semantic processing and greater effort than literal ones. The most interesting result, however, is that compared to novel literal expressions, novel metaphorical compound words also activated left-hemispheric areas, including the temporal pole and the posterior STS. Therefore, these areas are solely responsible for processing metaphoricity – in the left hemisphere. During the conceptual analysis of one of the constituents of a metaphorical expression, one abstract semantic property has to be extracted, while many other concrete ones need to be filtered out. Most probably this high level linguistic analysis requires the left hemisphere’s language resources.
Figure 5: fMRI contrasts of figurativeness. Images A, B, C: All metaphorical expressions (warm colors) vs. all literal expressions. Image D: conventional metaphors (warm colors) vs. conventional literal expressions (cold colors). Images E, F: novel metaphors (warm colors) vs. novel literal expressions (cold colors)
The divided visual field experiment
According to the results, all adjective-noun word pairs were processed faster and more accurately in the left hemisphere. That is, if the psycholinguistic variables posing a potential computational demand on the right hemisphere are carefully controlled for, the right hemisphere advantage in processing semantically distant concepts and specifically novel metaphors cannot be detected. Most experiments reporting such results previously did not necessitate the two words to be combined into a new conceptual unit. Integrating two words into a meaningful whole perhaps represents a much greater computational challenge than coarse coding requiring left hemispheric resources.
Expected impact, further research
If later confirmed, the results might turn a several decade long research trend by challenging the right hemisphere theory of metaphor processing, further clarifying our idea regarding the right hemisphere’s language competencies as well as figurative language. The research of metaphorical expressions has most probably been deeply intertwined with contextual effects, while crucial semantic factors such as the abstract-concrete dimension or categorization processes have received little attention. In fact, the latter may be key factors in brain as well as in algorithmic machine processing: these factors are much better defined compared to the vast amount of characteristics of context.
Publications, Links, References
Related publications
Forgács, B., Bohrn, I., Baudewig, J., Hofmann, M. J., Pléh, Cs., & Jacobs, A. M. (submitted, under review), Neural Correlates of Combinatorial Semantic Processing of Literal and Figurative Noun Noun Compound Words
Forgács, B., Bohrn, I., Pléh, Cs., & Jacobs, A. M. (2011), Long Distances within Single Words: Bridging the Metaphor Gap. Poster presented at the workshop on Processing and appreciating creative figurative language in Heidelberg. Heidelberg University, Germany, May 11-12th
Forgács, B. (2010), Is it really semantic distance that turns the right hemisphere on? Paper presented at the 8th International Conference on Researching and Applying Metaphor (RaAM8), Amsterdam, The Netherlands, June 30 - July 3
Forgács, B. (2010). Metaphors in Conversations: A Conceptual or a Pragmatic Trick? Poster presented at DUCOG II. (2nd Dubrovnik Conference on Cognitive Science), Dubrovnik, Croatia. May 6-9
Links.
http://en.wikipedia.org/wiki/Language_technology
http://en.wikipedia.org/wiki/Content_analysis
http://corpora.informatik.uni-leipzig.de/
http://w3.usf.edu/FreeAssociation/Intro.html
http://hu.wikipedia.org/wiki/Broca-ter%C3%BClet
http://en.wikipedia.org/wiki/Wernicke%27s_area
http://en.wikipedia.org/wiki/Pragmatics
http://en.wikipedia.org/wiki/FMRI
http://en.wikipedia.org/wiki/Eye_tracking
http://www.languages-of-emotion.de/en/dine/home.html
http://en.wikipedia.org/wiki/Temporoparietal_junction
http://en.wikipedia.org/wiki/Inferior_frontal_gyrus
http://en.wikipedia.org/wiki/Temporal_pole
http://en.wikipedia.org/wiki/Superior_temporal_sulcus
References
[1] Sperry, R. W. (1985), Consciousness, personal identity, and the divided brain, In D. F. Benson, & E. Zaidel (Eds.), The Dual Brain: Hemispheric Specialization in Humans (pp. 11-26), New York, Guilford Press
[2] Van Lancker Sidtis, D. (2006)., Does functional neuroimaging solve the questions of neurolinguistics? Brain and Language, 98, 276-290
[3] Vigneau, M., Beaucousin, V., Hervé, P.-Y., Jobard, G., Petit, L., Crivello, F., Mellet, E., Zago, L., Mazoyer, B., & Tzourio-Mazoyer, N. (2011), What is right-hemisphere contribution to phonological, lexico-semantic, and sentence processing? Insights from a meta-analysis, NeuroImage, 54, 577–593
[4] Weylman, S. T., Brownell, H. H., Roman, M., & Gardner, H. (1989), Appreciation of indirect requests by left and right damaged patients. The effects of verbal context and conventionality of wording, Brain and Language, 36, 580–591
[5] Faust, M., & Chiarello, C. (1998), Sentence context and lexical ambiguity resolution by the two hemispheres, Neuropsychologia, 3, 827–83
[6] Marinkovic, K., Baldwin, S., Courtney, M. G., Witzel, T., Dale, A. M., & Halgren, E. (2011), Right hemisphere has the last laugh: neural dynamics of joke appreciation, Cognitive, Affective, & Behavioral Neuroscience, 11, 113–130
[7] Eviatar, Z., & Just, M. A. (2006), Brain correlates of discourse processing: An fMRI investigation of irony and conventional metaphor comprehension, Neuropsychologia, 44, 2348-2359
[8] Schmidt, G. L., & Seger, C. A. (2009), Neural correlates of metaphor processing: The roles of figurativeness, familiarity and difficulty. Brain and Cognition, 71(3), 375-386
[9] Giora, R. (2003),. On our mind: Salience, context and figurative language. New York: Oxford University Press
[10] Beeman, M. J. (1998), Coarse semantic coding and discourse comprehension. In Beeman, M. & Chiarello, C. (Eds.), Right hemisphere language comprehension: Perspectives from cognitive neuroscience (pp. 255-284), Mahwah, NJ: Erlbaum
[11] Graves, W. W., Binder, J. R., Desai R. H., Conant, L. L., & Seidenberg, M. S. (2010), Neural correlates of implicit and explicit combinatorial semantic processing, Neuroimage, 53, 638-646
[12] Jung-Beeman, M. (2005), Bilateral brain processes for comprehending natural language, Trends in Cognitive Sciences, 9(11), 512-518
[13] Hagoort, P. (2005), On Broca, brain, and binding: A new framework, Trends in Cognitive Sciences, 9(9), 416-423