Real-time character recognition in video frames has been attracting great attention from developers since scene text recognition was recognized as a new field of Optical Character Recognition (OCR) applications. Some oriental languages such as Japanese and Chinese have thousands of characters, and the character recognition takes much longer time in general compared with European languages. Speed-up of character recognition is crucial to develop software for mobile devices such as Smart Phones. This paper proposes a binary tree-based clustering technique that can keep the precision as quite high as possible. The experimental results show that the character recognition using the proposed clustering technique is 8.3 times faster than the full linear matching at mere 0.22% precision drop. When the proposed method is combined with the Sequential Similarity Detection Algorithm (SSDA) and a PCA-based dimensionality reduction, we can achieve 36.2 times faster character matching at 0.29% precision drop.