Multi-sensor based state prediction for personal mobility vehicles

Jamilah Abdur-Rahim, Yoichi Morales, Pankaj Gupta, Ichiro Umata, Atsushi Watanabe, Jani Even, Takayuki Suyama, Shin Ishii

Research output: Contribution to journalArticlepeer-review

13 Citations (Scopus)


This paper presents a study on multi-modal human emotional state detection while riding a powered wheelchair (PMV; Personal Mobility Vehicle) in an indoor labyrinth-like environment. The study reports findings on the habituation of human stress response during self-driving. In addition, the effects of "loss of controllability", change in the role of the driver to a passenger, are investigated via an autonomous driving modality. The multi-modal emotional state detector sensing framework consists of four sensing devices: electroencephalograph (EEG), heart inter-beat interval (IBI), galvanic skin response (GSR) and stressor level lever (in the case of autonomous riding). Physiological emotional state measurement characteristics are organized by time-scale, in terms of capturing slower changes (long-term) and quicker changes from moment-to-moment. Experimental results with fifteen participants regarding subjective emotional state reports and commercial software measurements validated the proposed emotional state detector. Short-term GSR and heart signal characterizations captured moment-to-moment emotional state during autonomous riding (Spearman correlation; ρ = 0.6, p < 0.001). Short-term GSR and EEG characterizations reliably captured moment-to-moment emotional state during self-driving (Classification accuracy; 69.7). Finally, long-term GSR and heart characterizations were confirmed to reliably capture slow changes during autonomous riding and also of emotional state during participant resting state. The purpose of this study and the exploration of various algorithms and sensors in a structured framework is to provide a comprehensive background for multi-modal emotional state prediction experiments and/or applications. Additional discussion regarding the feasibility and utility of the possibilities of these concepts are given.

Original languageEnglish
Article numbere0162593
JournalPloS one
Issue number10
Publication statusPublished - 2016 Oct

ASJC Scopus subject areas

  • Biochemistry, Genetics and Molecular Biology(all)
  • Agricultural and Biological Sciences(all)
  • General


Dive into the research topics of 'Multi-sensor based state prediction for personal mobility vehicles'. Together they form a unique fingerprint.

Cite this