Find out what a user is doing before the first utterance: Discrimination of user's internal state using non-verbal information

In this research, we propose a method for estimating user's internal state (thinking or embarrassed) before the utterance toward a spoken dialogue system. Modeling user's internal state such as belief, skill or familiarity and introducing these model to the dialogue system should be useful to make flexible responses. However, because conventional estimation of internal state is based on the linguistic information of the previous utterance, it cannot estimate a user's internal state before the user's first utterance. We focus on a user's multimodal features such as filler word, silence, or face direction before the user's input utterance in order to model the user's internal state. The dialogue data were collected on the Wizard of Oz basis as training and test materials. Finally, we conducted an experiment for discrimination with two classification schemes and the hierarchical method obtained higher discrimination accuracy than that of pair-wise method.