Language activities through the screen of an electronic terminal are indispensable to our everyday life. We specifically investigate a human act of “reading” on a screen and performing research on its measurement, modeling, and support in this project. Specifically, we regard an act of “reading” a text on a screen as the interaction of the following three factors: (1) semantic structure of an object text; (2) image features, such as a layout and character decoration; and (3) a reader’s visual sense and language cognitive process. Our goal is to develop a method for presenting a text in a readable form, as well as studying the measurement and modeling method.
Text-gaze alignment using language model
Gaze-text alignment using image registration (by Pascual Martínez-Gómez)
The analysis of “reading” requires extracting a “word” to which a reader is attentive each time. It is nevertheless difficult in actual conditions to realize sufficient accuracy for analysis in an ordinary gaze measuring environment. Accordingly, we study a method with enhanced accuracy in associating of a gaze and a text by introducing a language model and a cognitive model in this project. Progress in IT technology has diversified objects for reading and has also complicated the skill of reading. It is expected that improved measurement accuracy of reading will facilitate log collection in a natural environment, which can then contribute to research of a reading model. (Akito Yamaya, Goran Topić, and Pascual Martinez-Gomez)
Modeling of gaze prediction based on linguistic analysis and text structure
Word reading sequence in Dundee corpus (by Tadayoshi Hara)
The appearance and contents of a text have inseparable relation in the act of “reading.” However, text segmentation and reader attributes have been investigated independently of the language processing of a text in many cases. We accordingly investigate methods for analysis and prediction of movement of a gaze recorded using a gaze measuring device, with comparison to (1) the structure and semantics of a text; (2) image features; and (3) reader attributes. Conventional natural language processing addresses “what is written,” whereas “what is read” has been treated as a subject in cognitive psychology or brain science. The language processing of “the way of being read,” particularly addressing the image feature of a document and the reader’s situation is a novel approach. It is presumed to induce broad application including support for reading and writing of on-line information. (Pascual Martinez-Gomez, and Tadayoshi Hara)
Document layout optimization based on semantic analysis of contents
Proposed workflow of paraphrase-based document layout optimization (by Yusuke Kido).
Positioning of document constituents is devised considering the continuity of regions and beautiful printing in document layout optimization by a computer. Usually, neither the text contents nor the text itself are examined or rewritten. We have developed a method of optimizing a document layout more flexibly by closely positioning a chart and a text that mentions it or adjusting the text segment size by paraphrasing. (Yusuke Kido and Kyohei Tomita)