Paper short abstract:

This paper reports multimodal analysis results of 9 web lectures. Features such as non-linguistic or para-language cues were observed corresponding to the places where meta-language has a "stress" or "topic" function, leading to an understanding of a lecture.

Paper long abstract:

The number of foreign students in Japan exceeded 200,000 in 2014, and international students from Europe also increased significantly from 6,370 in 2014 to 7,268 in 2015 (JASSO 2016). However, it is often said that Japanese lectures cannot be sufficiently understood, and training students to understand lectures before studying abroad in Japan is an urgent issue.

To understand lectures, it is not only necessary to have language ability, but also the ability to understand linguistic cues, non-linguistic cues and para-lingual cues (Mohri 2014, Mohri et al. 2016a). Especially, linguistic cue meta-language expressions are involved in a lecture discourse and presentation of important information, and are known to provide important "clues" for understanding the lecture (Nakai and Toramaru 2010, Mohri et al. 2016b). In addition, with regard to non-verbal information given by lecturers, its relevance to lecture understanding was pointed out (Szatrowski 2010). However, no research has yet been reported on comprehensive analysis of the lecture's "language information", "non-verbal information" and "paralinguistic information".

Therefore, in this research, we conducted a survey using a multimodal analysis method covering all elements related to transmission of information. The procedures were as follows:

1) From about 4000 web lectures of universities all over Japan, we extracted 9 typical web lectures (about 120 minutes in total) as case studies, and

2) We annotated where meta-language is used in a lecture discourse, and carried out labeling of non-verbal behavior and paralinguistic behavior. As a result of this analysis, at places where meta-language has a "stress" function, features such as the lecturer's "pose", "line of sight to a recording video (imaginary viewer)", "change of voice quality (loudness/pitch) and speed", and "moving of hands in front and swinging them up and down", were observed just before those places. In addition, at places where it had a "topic" function, features such as non-linguistic cues and para-lingual cues like gaze and hand movements were observed, leading to an understanding of a lecture.