Subtitle Preference While Viewing Native Language Movie: The Effect of Sound

Abstract

Abstract:

Movies and television programs are generally dubbed or subtitled in the local language recently. Subtitle preference or obligatory was found in studies of text and graphics integration and television viewing. From previous findings, researchers found that the fixation of subtitle was not affected while the soundtrack was turned off. Researchers concluded that subtitle and sound was overlapped and sound was redundant while they prefer to watch subtitles. The purpose of this study planed to investigate the sound effect on subtitle preference while viewing native language movie. Chinese and English subtitle movie clips were chose as materials, half with sound and the other was turned off. Students were recruited from a normal university as participants. Eye tracking technique was used as data collecting. After the eye movements data collecting, an inventory of the movie clip content was tested. Total eye movement, areas of interest of subtitle and video, and subtitle pieces were analyzed to find the effects of sound. Results showed that participants fixated more on subtitle area than video when the sound was turned off. Fixation number, fixation durations on subtitle of movie without sound were significantly higher than movie with sound. More generally, the purpose of subtitle preference was mainly for reading and understanding, the sound of video can inactive and help subtitle for movie content processing.

Key words: subtitle preference, eye movements, movie viewing, sound

摘要：

字幕作为电影和电视节目的重要组成部分，在视频观看过程中起到辅助加工的作用。先前的研究认为字幕是视频语言的直接转译，字幕与声音信息具有很高的重叠性，所以关闭视频的声音不影响被试对字幕的加工。研究以电影视频作为刺激材料，采用眼动技术探讨了声音对电影观看过程中字幕偏好性的影响作用。实验通过控制视频的声音信息，探讨声音对字幕偏好性加工的影响作用。结果发现：视频的声音信息会对字幕的加工产生影响，关闭声音后被试对字幕的加工时间和次数会增加，无声条件下对字幕的注视次数和注视时间占总注视的比例超过了50%，说明声音和字幕对视频的理解具有相互辅助的作用。

关键词: 字幕偏好性, 眼动, 电影观看, 声音

王福兴白学军闫国利周宗奎. 电影观看过程中的字幕偏好性：声音的影响作用[J]. 心理科学, 2012, 35(1): 44-49.

References

Boatner, E. B. (1980). Captioned Films for the Deaf. Retrieved 02-21, 2010, from http://www.cfv.org/caai/nadh84.pdf Carroll, P. J., Young, J. R., & Guertin, M. S. (1992). Visual analysis of cartoons: A view from the far side. In K. Rayner (Ed.), Eye movements and visual cognition: Scene perception and reading (pp. 444-461). New York: Springer-Verlag. Chapdelaine, C., Gouaillier, V., Beaulieu, M., & Gagnon, L. (2007). Improving video captioning for deaf and hearing-impaired people based on eye movement and attention overload. Paper presented at the Conference on Human Vision and Electronic Imaging XII, San Jose,CA. d'Ydewalle, G., & Gielen, I. (1992). Attention allocation with overlapping sound, image, and text. In K. Rayner (Ed.), Eye movements and visual cognition: Scene perception and reading (pp. 415-427). New York: Springer-Verlag. d'Ydewalle, G., Praet, C., Verfaillie, K., & Rensbergen, J. V. (1991). Watching subtitled television: Automatic reading behavior. Communication Research, 18(5), 650-666. d’Ydewalle, G., & De Bruycker, W. (2007). Eye movements of children and adults while reading television subtitles. European Psychologist, 12(3), 196-205. d’Ydewalle, G., Van Rensbergen, J., & Pollet, J. (1987). Reading a message when the same message is available auditorily in another language: The case of subtitling. In O. R. J. K. & A. Levy-Schoen (Eds.), Eye movements: From physiology to cognition (pp. 313-321). Amsterdam: Norht-Holland. De Bruycker, W., & d'Ydewalle, G. (2003). Reading native and foreign language television subtitles in children and adults. In J. Hy?n?, R. Radach & H. Deubel (Eds.), The mind's eye: Cognitive and applied aspects of eye movement research (pp. 671-684): North-Holland. Hannus, M., & Hyona, J. (1999). Utilization of illustrations during learning of science textbook passages among low-and high-ability children. Contemporary Educational Psychology, 24(2), 95-123. Hegarty, M., & Just, M. A. (1993). Constructing mental models of machines from text and diagrams. Journal of Memory and Language, 32, 717-742. Jensema, C. J., Danturthi, R. S., & Burch, R. (2000). Time spent viewing captions on television programs. American Annals of the Deaf, 145(5), 464-468. Linebarger, D. L. (2001). Learning to read from television: The effects of using captions and narration. Journal of Educational Psychology, 93(2), 288-298. Norwood, M. J. (1988). Captioning for deaf people: An historical overview. Paper presented at the Speech to Text: Today and Tomorrow. Proceedings of a Conference at Gallaudet University, Washington, D.C. Rayner, K. (1998). Eye movements in reading and information processing: 20 years of research. Psychological Bulletin, 124, 372-422. Rayner, K. (2009). Eye movements and attention in reading, scene perception, and visual search. The Quarterly Journal of Experimental Psychology, 62(8), 1457-1506. Rayner, K., Rotello, C. M., Stewart, A. J., Keir, J., & Duffy, S. A. (2001). Integrating text and pictorial information: Eye movements when looking at print advertisements. Journal of Experimental Psychology: Applied, 7(3), 219-226. Takeda, M. (1999). Eye movements while viewing a foreign movie with subtitles. In W. Becker, H. Deubel & T. Mergner (Eds.), Current Oculomotor Research: Physiological and Psychological Aspects (pp. 313-316). New York: Kluwer Academic. Vatikiotis-Bateson, E., Eigsti, I. M., Yano, S., & Munhall, K. G. (1998). Eye movement of perceivers during audiovisual speech perception. Perception and Psychophysics, 60(6), 926-940. Wang, F., Zhou, Z., Zhao, X., Bai, X., & Yan, G. (Under Review). Subtitle preference while viewing native language movie: The effect of subtitle familiarity. 俞自萍, 曹愈, & 曹凯. (1996). 色盲检查图(第五版). 北京: 人民卫生出版社.

[1]	. The effects of the word predictability on eye movements of skilled and less-skilled developing readers in Chinese Sentnces reading [J]. , 2020, (6): 1369-1375.
[2]	. Effects of Inter-character Spacing on the Reading of Chinese Young and Older Adults: An Eye Movement Study [J]. , 2020, (1): 68-74.
[3]	. Inserting Spaces Benefits Chinese Developmental Dyslexic Children for Reducing their Visual Crowding: Evidence from Eye Movements [J]. , 2019, (4): 834-840.
[4]	. The differences of the contextual predictability effect between highly-skilled and low-skilled developing readers during parafoveal processing: Evidences from eye movements [J]. , 2019, (4): 848-853.
[5]	. Parafoveal Processing of Character N+1 of Deaf Readers [J]. , 2019, (4): 997-1003.
[6]	Jian GUAN Wen-Rui Li. The Competition between Inhibition of Return and Emotional Attention Bias: Evidence from Eye Movements [J]. Psychological Science, 2018, (6): 1353-1358.
[7]	. The perceptual span of second graders in Chinese primary school [J]. Psychological Science, 2018, (4): 849-855.
[8]	. Hazard Perception in Young Drivers: the Role of Hazard Types [J]. Psychological Science, 2018, (2): 447-452.
[9]	. Time Course of Linguistic Information Extraction from Parafoveal Words During Eye Fixations in Chinese Reading [J]. Psychological Science, 2017, 40(6): 1289-1295.
[10]	. Text-diagram Integration in Multimedia Learning: The Effects of Synchronized Cues and Learners’ Prior Knowledge [J]. Psychological Science, 2017, 40(5): 1104-1110.
[11]	. The Review of Eye Movement Research of Deaf Readers [J]. Psychological Science, 2017, 40(3): 553-558.
[12]	. The multi-sensory channel processing of face emotion [J]. Psychological Science, 2016, 39(1): 77-82.
[13]	. Experience Advantage of Driver to percept Pedestrian’s Hazard Under Car-following Condition [J]. Psychological Science, 2015, (6): 1340-1346.
[14]	Long SUN CHANG Ruo-Song. Review of Research on Drivers’ Hazard Perception [J]. Psychological Science, 2014, 37(6): 1354-1358.
[15]	. Effects of Strokes’ Pixel Removal on Chinese Sentence Reading: Evidence from Eye Movements [J]. Psychological Science, 2014, 37(3): 521-527.