Vision-based finger tapping detection without fingertip observation

A machine learning approach is investigated in this study to detect a finger tapping on a handheld sur-face, where the movement of the surface is observed visually; however, the tapping finger is not directly visible. A feature vector extracted from consecutive frames captured by a high-speed camera that observes a surface patch is input to a convolutional neural network to provide a prediction label indicating whether the surface is tapped within the sequence of consecutive frames (“tap”), the surface is still (“still”), or the surface is moved by hand (“move”). Receiver operat-ing characteristics analysis on a binary discrimination of “tap” from the other two labels shows that true positive rates exceeding 97% are achieved when the false positive rate is fixed at 3%, although the generaliza-tion performance against different tapped objects or different ways of tapping is not satisfactory. An in-formal test where a heuristic post-processing filter is introduced suggests that the use of temporal history information should be considered for further improve-ments. biho.