We have been working on detecting an object with an accelerometer from many moving objects in a camera view. Our previous method relied on knowledges of specific appearance of the moving objects such as colors, which limited the practical use. In this paper, in order to reduce this dependency on the knowledges of object appearance as much as possible, we apply our method to natural feature points detected in the camera view. To deal with numerous feature points in real time, we propose a method to speed up the computation. We also investigate a method to cope with natural features that are not always tracked continuously. Experimental results show that the proposed method is successfully applied to natural feature points detected and tracked in real time.