Subtitling of Oprah ’ s Interview of Michael Jackson from a Multi-modal Approach

This paper is about the introduction and use of Multimodal Analysis (MDA) as a research methodology in subtitling of audiovisual products. Its aim is to show the contribution of the MDA to the analysis of multi-semiotic resources in audiovisual products and their influences on the subtitling. Although many scholars have realized that audiovisual products employ different types of signs to create meanings, the focus remains primarily on their linguistic features. The paper elaborates on the theoretical and practical aspects of the MDA methodology, and provides a case study to show how the methodology works in the subtitling process of audiovisual products, in particular the subtitling of interviews. Finally, it concludes the paper with some suggestions for further research.


Introduction
In recent years, foreign audiovisual products are more easily accessed globally.To make these products more appealing to achieve higher ratings, they more often than not take advantage of various semiotic resources like images, sounds, videos and music, and so forth to engage audience in their performances.Therefore, the translation techniques and strategies of the verbal discourse produced in these media products are significantly different from those of pure textual products.Quite a number of studies on the translation of multi-media products have been published in the last 20 years.But they are often limited in scope, dealing mainly with linguistic and cultural matters.

MDA
Multimodal text analysis (MDA) assumes that "the meaning of a film, a TV ad, a web page, a cartoon, a comic book, is the composite process/product of different selected semiotic resources" (Baldry & Thibault, 2006).Actually, an increasing number of discourses can be regarded as multimodal nowadays.There are two reasons for this: language is not considered as the sole meaning-making mode like before and the traditional boundaries between verbal and non-verbal modes are blurring.For example, written texts, which seem to contain only one mode, can use graphic means like bolding, size and spacing as their semiotic resources.A growing number of researchers have realized that to describe communication between humans, semiotic systems other than language must be accounted for, which is more true to the multi-media products.Among all the theoretical tools, SFL (systemic functional linguistics) is the one most often deployed in the MDA.Halliday is known for developing SFL, who is the first linguist to consider language as a resource for construing meaning, coining the expression "Language as social semiotic" in the early 70s.He argued that semiotic resources are "system[s] of meanings that constitute the 'reality' of the culture".(Halliday, 1978) The paramount concern of social-semiotic theory is the meaning and all its forms.Meaning is generated in social environments and social interactions.
On the basis of Michael Halliday's social semiotic approach to language study, Kress & van Leeuwen (1996) and O'Toole (1994) provided the foundations for multimodal research in the 1980s and 1990s.Since then, a group of researchers attempt to account for how different modes work together to construe meaning.Kress & van sculpture and architecture. Martinec (2000) explored the actions and gestures as semiotic resources in meaning-making.Jewitt (2006) conducted educational research from a multimodal perspective.Others (Martinec, 2001;Mercedes & Inmaculada, 2012;Hooda & Forey, 2015) argue that semiotic resources like gestures, movements and space may represent interpersonal meta-function.Some researchers attempt to combine different disciplines with MDA in discourse analysis.For instance, Forceville (2009) applied cognitive approach in multimodal analysis.
As the research goes deeper, much progress has been made in this field.MDA is not confined to any individual semiotic resource anymore and has moved on to the study of inter-semiotic relations, which accounts for how multi-modes work together and their interrelationship in texts.Royce (1998a;1998b;2007) tries to explore the interrelationship between different modes in the framework of inter-semiotic complementarity.There is also research in text-image relations (Bateman, 2008;Liu & O' Halloran, 2009;Martinec, 2005;Unsworth & Cleirigh, 2009), gesture and speech (Martinec, 2004) and language, images and mathematical symbolism (Lemke, 1998;O' Halloran, 2008).Till now, the central areas of multimodal research are as follows : 1) intersemiosis (Jewitt, 2009): the interaction of semiotic choices; 2) design, production and distribution of multimodal resources in social settings.(O'Halloran, 2011;Van Leeuween, 2008) and 3) resemiotization (Iedema, 2001;2003).In analysis of multimodal texts, one should keep in mind the following facts: 1) Various semiotic resources work simultaneously and collectively as modal ensembles, designed so that each mode has a specific task.(Kress, 2010).2) Modes are the result of a social and historical shaping of materials chosen by a society for representation (Kress, 2010).A certain mode in this culture does not necessarily cover the same field or is used for the same purpose in other cultures.3) All modes are dynamic since that they are generated or transformed in accordance to the need of society.SF theory is chosen as a qualified theoretical tool for MDA is because that the three metafunctions of SF ( interpersonal, experiential and textual meta-function) provide a semiotic framework to explore meaning-making beyond language.For those of us who are interested in how language acts both socially and semiotically, this is useful because it lets us describe and explain how social reality is encoded in language, both in terms of how language is a means of reflecting on things and how it is a means of acting (symbolically) on people (Halliday, 1978).

Audiovisuals Translation
As its name suggests, audiovisuals are made to be both heard (audio) and seen (visual) simultaneously.Films, television programs, theatrical plays, musicals, opera, web pages, and video games all fall into this category.As the range of audiovisual products expands, the scope of audiovisual translation increases as well.It is no longer confined to translating verbal discourse of films but includes commercials, sitcoms, cartoons, documentaries and interviews etc.
According to various translation targets and different ways of representing translation results, translation can fall into the following categories: subtitling, dubbing, drama translation, literary translation, literary translation and simultaneous interpreting.
The following table can distinctly demonstrate the differences and similarities between these five types of translation.In this article, subtitling is the issue of concern.

Subtitling
From the above table, it can be seen that a number of distinct features of subtitling differentiate it from other main types of translation: 1) It serves as a synchronous semiotic channel in audiovisuals.2) It is additive.Therefore, some information in the subtitling that overlaps the information contained in other semiotic resources can be omitted in subtitling process to make it concise.
3) It is a transient and polysemiotic text.The basic standard of subtitling is described by Díaz C. & Anderman (2009): "Subtitles are said to be most successful when not noticed by the viewer."To achieve this goal, subtitlers have to deal with some constraints inherent in the process of subtitling in order not to distract viewers' attention from appreciating the program.The major constraints of audiovisual programs are twofold: space and time.(Díaz C. & Anderman, 2009) 1) Space: the space allowed for a subtitle is limited so it won't cover the actions on the screen.Two lines of text are the norm.The length of a subtitle should allow audience enough time to read it and simultaneously catch what's going on the screen.2) Time: the duration of subtitles should be in accordance with the change of images.They shouldn't switch too fast so that audience can fully take in information contained in them.Meanwhile they should follow the pace of image change.
To comply with these constraints and minimize the negative effects of extra processing efforts made by viewers, a few translation strategies are proposed by researchers: 1) Subtitles should use the most possibly concise expressions to provide viewers enough time to appreciate images and actions on screen.Deletion or condensation of redundant, unnecessary oral expressions often accompanies the process of subtitling.
2) The arrangement of words should be carefully designed to enhance readability of subtitles.The simpler and more common the syntactic structure is employed, the less efforts are required to comprehend the meaning.

Subtitling and MDA
Apart from dealing with these constraints, some translation scholars started to account for modes other than language in subtitling.Gottlieb (2001) holds that visual and audio factors which made their contributions to verbal modes should be taken into account in translating process.His viewpoint is shared by some other researchers of subtitling.Delabastita (1989) and Chuang (2006) argue that "Subtitle translation is intersemiotic translation, and that all the semiotic modes involved in the film text contribute meanings according to their functional specialization rather than verbal elements alone."Although many scholars have realized that audiovisual products employ different types of signs to create meanings, the focus remains primarily on their linguistic features.Gambier (2006) points out this paradox: "We are ready to acknowledge the interrelations between the verbal and the visual, between language and non-verbal, but ... the multisemiotic blends of many different signs are not ignored but they are usually neglected or not integrated into a framework."and he believes that "the multimodal discourse analysis will help to develop awareness and analysis of the integration of semiotic resources in AV." The reason of employing multimodal text analysis in the translation of audiovisual products is that audiovisuals are a type of multimodal phenomena.As Delia (2010) puts it: "Screen products (from this point onwards SP) are polysemiotic; in other words, they are made up of numerous codes that interact to produce a single effect."Different types of semiotic modes function simultaneously when generating meanings.To interpret this discourse practice, translators need to recognize and interpret the meanings arising from multiple semiotic resources deployed in the audiovisuals.The polysemiotic nature of audiovisual products can be demonstrated in the following table:

Subtitling of Interviews
Till now, a great number of the ATV translation studies have focused on the subtitling of films, with little attention paid to that of interviews.TV interviews, like other audiovisuals, combine scores of various modes to charm audience and boost audience rate.While watching interviews, along with the ongoing conversations between hosts and guests, the audience might also hear the lyrics of songs and simultaneously be exposed to written information like letters, articles and poetry.On the other hands, interviews contain nonverbal sounds, like music with no lyrics, breathing, laughter, crying ect.The people engaged in interviews also have their facial expressions, gestures, body movements, costumes, hairstyles, and so forth to provide additional information.All these semiotic modes can create meanings together.For the above reasons, to achieve a success in subtitling of interviews, non-textual modes also require special attention.
The semiotic modes listed as following are those which most frequently make their contributions in an interview: the spoken mode, the written mode, the mode of body movement, the movement of head and the mode of sound effects.

Figure 1. Multimodal subtitle translation
As we can see in Figure 1, the shape of the source text is different from that of the target text.The reasons behind this change of shape are twofold: 1) In the process of subtitling, certain meanings may lose or gain in these aspects; 2) Those meaning potentials contained in these modes might be realized in different modes in the target texts.Translators should be aware of that the realization of a certain mode may not be realized by the same mode in the subtitling process, for example, the meanings conveyed by movements in interviews in SL might be expressed by the verbal mode in interviews in target language.

Translation of Oprah's Interview of Michael Jackson
Michael Jackson is an American singer, songwriter, record producer, dancer and actor.He is dubbed the King of Pop.His incomparable music talents made him a global figure in popular culture for over four decades.When he was alive, there were quite a few rumors going around about him endlessly, including his relationship with his father, his excessive love for children, the change of his skin color and facial features etc.However, he was a fiercely private entertainer and had refused to give an interview for 14 years.That is why Oprah's interview of Michael drew a worldwide audience of 90 million people."It was the most exciting interview I had ever done."Oprah says when she looks back on her career life, "It certainly was going to be the most watched interview I had ever done." As an excellent talk show host, she approached these sensitive issues skillfully.
Different from Michael's performances on stage which are full of passions, emotions and sexual signals; in real life Michael is quite shy and reserved around people.From this interview, we can see that Michael is very light-hearted and has the most beautiful laugh ever.Sometimes when confronted with the craftiest question, he still tried to soften it and treat it in an amusing way, which may be accompanied by his funny analogy and light tone.All these personality features and sentiments were displayed by his verbal responses, gaze, movements, non-verbal sounds, tone and speech tempo in the interview.Translators should take these multi-semiotic resources into consideration in the process of subtitling.There's so much garbage and so much trash.It's so untrue, they're complete lies.

Questions about Michael's Father
Those are some of the things I wanted to talk about.
Before this topic, Oprah and Michael talked about Michael Jackson's father who had been excessively strict with him.His disciplinarian created endless traumas in Michael's adolescence, something he couldn't get rid of even after entering his adulthood.Despite of all these unhappy memories, Michael Jackson still harbored tender and forgiving sentiments towards his father and spoke of this history with peaceful and loving mentality.
Therefore, there is no surprise that when Oprah asked about whether Michael Jackson could truly forgive his father for all the traumas inflicted on him in his childhood, Michael stopped for a while, inhaling and exhaling a deep breath, with one side of his shoulders rising and dropping accordingly.His gaze was disengaged from the eyes of Oprah by looking sideway.This disengagement showed that he gave this question a careful thought once again.Then he looked back, re-established the eye contact and said: "I do forgive."Here, a Finite "do" is put in front of "forgive" to make an emphasis.Through all these signals, we can see that Michael Jackson thought about this question seriously and confirmed wholeheartedly that he could really let the past pass and forgive all the horrible things his father did to him.So this sentence should be translated as follow: Example 1:

我真的可以原谅。
Chinese words "真的可以" are used to translate the Finite "do" in the source language."真的" (really) is utilized to express the emphasis."可以" (can) is to show his ability of letting go of the miserable experience.
The emotion of Michael Jackson had been stable all the time until another thing occurred to him.After expressing forgiveness towards his father, he abruptly changed the topic and brought up the untrue reports about his personal life.He demonstrated a spasm of irritation by slowing his speech tempo and wearing a severe facial expression.His statement "It's so untrue" was accompanied by the shaking of head.These two modes worked together as a denial in this proposition.He paused at the word "untrue" and "lies" respectively to emphasize them.Michael used head movements and pauses to emphasize lexical items.From these resources, we can see that Michael was furious about these reports and wanted to deny them officially.Thus these sentences should be translated as follow: Example 2: There's so much garbage and so much trash that's written about me.It's so untrue, they're complete lies.

媒体对我的各种报道都是一派胡言。它们都是凭空捏造，胡说八道。
The first sentence in source language is an existential sentence, which starts with the word "there"."There" in this sentence serves to lead the following clause and has no ideational function.To comply with the expression habit of Chinese and put the "untrue reports" as the center of focus, "媒体对我的各种报道" (all sorts of media reports about me) is chosen as the Subject in target language."一派胡言" (a total of nonsense) is used to translate the word "garbage", which completely deny those reports."Untrue" is translated as "凭空捏造" (sheer fabrication) to state the total falsehood of these reports.
Then the lens closed up and caught the solemn countenance of Michael when he continued to say: "Those are some of the things I wanted to talk about."He frowned slightly as he said this.This expression is a combination of active and negative Affect (Martinec, 2001) by moving the corners of the mouth and eyes downward, in order to show one's hatred.His direct gaze towards Oprah displays his willingness of involvement in this subject and intention to "put this matter straight".It can be seen from these signals that he had planned to openly vent his dissatisfaction about these media reports long time ago.Thus this sentence should be translated as follow: Example 3: Those are some of the things I wanted to talk about.

我老早就想谈这些事了。
Michael's facial expression adds emphasis to the word "wanted".When translating this sentence, "老早"(from early on) is inserted in the sentence to express the emphasis conveyed by the knitting eyebrows of Michael."我"(I) is chosen as the Subject, with "some of the things" picked as the Object to shift the focus onto Michael.

Questions about the Change of Skin Color
In the later part of the interview, Oprah brought up another sensitive topic: whether Michael Jackson had bleached his skin.Michael's countenance started to get serious again when Oprah approached this question step by step.When Oprah asked Michael Whether he had ever bleached his skin because he didn't like his skin color and related about how people's skin could be "bleached", Michael shook his head and rolled eyes to show his impatience of hearing these ridiculous words.He cut off Oprah's statement by blurting out the word "okay" and started to deny this speculation, saying that there was no such thing as "skin bleaching" and the change of skin color was a sort of disease he couldn't control.His head nodded to exaggerate his denial.

Example 4:
Okay, but number one, this is the situation.I have a skin disorder that destroys the pigmentation of the skin, it's something that I cannot help.Okay.

第一，事实是这样。我患了皮肤失调症，皮肤黑色素被破坏。我也控制不了。
The two "okay" in source language text should be omitted to make the expression concise."Situation" is translated as "事实" (the truth) to make a clear-cut distinction between his following statement, which is the true situation, and the false speculation from ordinary people.The "have" in front of "a skin disorder" is translated as "患了" (suffer from) to let the audience know that the change of skin color is caused by an illness he suffers from.When translating "it's something that I cannot help", a "too" is added in the sentence to intensify the emphasis.
Then he added: "when people make up stories that I don't want to be who I am, it hurts me."His voice cracked a little when he uttered the word "hurt".After finishing the sentence, he paused for a while, closed his eyes and lowered his head, which are the obvious signs of negative Affect (Martinec, R., 2001).The integration of language and movements can be seen as amplification of the Affect.It seemed that he really felt deeply sad that people held such a speculation about his changed skin color and considered him as a person who despised his identity as a black man.

Example 5:
…but when people make up stories that I don't want to be who I am, it hurts me.

但是当人们编造谎言，说我自我嫌弃时候，我很痛心。
In the source language, "it" was chosen as the Subject in the second clause, referring to the false speculation coming from people.To emphasize his hurt feeling, "我" is picked as the Subject to transfer the focus on Michael's anguished emotion."痛心" (grieved) is employed to express his grievances.
As Michael sees it, people should put their focus on the art of his music, rather than his private life.As for the behaviors of peeking into other people's personal life and taking endless delight in talking about it, Michael showed nothing but his disdain.A smile appeared on his face when he said: "…not about who he went out with last night…what's wrong with…I mean that's what important to me."To Michael, personal life means nothing compared with the appreciation of real art.When Michael listed all those insignificant stuff people cared about, he rolled his eyes all the time.But when he said that art was what he really cared about, he looked directly at the eyes of Oprah.This change of gaze focus also showed what he cared and what he cared nothing about.

Example 6:
…not about who he went out with last night…why he sits out in the sun…what's wrong with…I mean that's what important to me.
(人们)关心的不应该是他昨晚跟谁约会……为啥他要日光浴……他的哪方面出了什么问题……我是 说艺术方面的事才是我真正关心的。 "That" in the source language refers to the real art, therefore it is translated as "艺术方面的事" (things about art) to clarify its reference and make an emphasis."What important to me" is translated into "我真正关心的" (what I care most), transferring "我" (I) from an object to a Subject, in the sense of having the information focus on Michael.

Conclusion
This paper explores the concept of multimodality in the process of subtitle translation.It has discussed how different semiotic modes contribute to generating different meanings in audiovisual products, especially in interview.The translator has to consider the distribution and integration of the meanings of multimodes in the source and target texts.Therefore, MDA is deployed as a theoretical framework in the discussion of proper translation strategies of audiovisual products.Till now, much of the ATV translation studies have focused on the subtitling of films, with little attentions paid to that of interviews.To satisfy the needs of audience of having a comprehensive understanding of the characters of interviewees, some semiotic resources which have made their due contributions in creating meanings should be accounted for.Secondly, translators should be aware of that the realization of certain mode may not be realized by the same mode in the subtitling process, for example, the meaning conveyed by movements in the source language AV might be expressed by verbal mode in the AV in target language.However, more research is needed on the exploration of how semiotic resources function interpersonally as well as ideationally and textually in a systematic way.