Real-time character recognition in video frames has been attracting great attention from developers since scene text recognition was recognized as a new field of Optical Character Recognition (OCR) applications. There are thousands of characters in some oriental languages such as Japanese and Chinese, and the character recognition takes much longer time in general compared with European languages. Speed-up of character recognition is crucial to develop software for mobile devices such as Smart Phones. This paper proposes a binary tree-based clustering technique with CDA (Canonical Discriminant Analysis) that can keep the accuracy as quite high as possible. The experimental results show that the character recognition using the proposed clustering technique is 10.2 times faster than the full linear matching at mere 0.04% accuracy drop. When the proposed method is combined with the Sequential Similarity Detection Algorithm (SSDA), we can achieve 12.3 times faster character matching at exactly the same accuracy drop.