Processing text files as is: Pattern matching over compressed texts, multi-byte character texts, and semi-structured texts

Masayuki Takeda, Satoru Miyamoto, Takuya Kida, Ayumi Shinohara, Shuichi Fukamachi, Takeshi Shinohara, Setsuo Arikawa

Research output: Chapter in Book/Report/Conference proceedingConference contribution

9 Citations (Scopus)

Abstract

Techniques in processing text files “as is” are presented, in which given text files are processed without modification. The compressed pattern matching problem, first defined by Amir and Benson (1992), is a good example of the “as-is” principle. Another example is string matching over multi-byte character texts, which is a significant problem common to oriental languages such as Japanese, Korean, Chinese, and Taiwanese. A text file from such languages is a mixture of single-byte characters and multi-byte characters. Naive solution would be (1) to convert a given text into a fixed length encoded one and then apply any string matching routine to it; or (2) to directly search the text file byte after byte for (the encoding of) a pattern in which an extra work is needed for synchronization to avoid false detection. Both the solutions, however, sacrifice the searching speed. Our algorithm runs on such a multi-byte character text file at the same speed as on an ordinary ASCII text file, without false detection. The technique is applicable to any prefix code such as the Huffman code and variants of Unicode. We also generalize the technique so as to handle structured texts such as XML documents. Using this technique, we can avoid false detection of keyword even if it is a substring of a tag name or of an attribute description, without any sacrifice of searching speed.

Original languageEnglish
Title of host publicationString Processing and Information Retrieval - 9th International Symposium, SPIRE 2002, Proceedings
EditorsAlberto H. F. Laender, Arlindo L. Oliveira
PublisherSpringer-Verlag
Pages170-186
Number of pages17
ISBN (Print)3540441581, 9783540441588
DOIs
Publication statusPublished - 2002 Jan 1
Externally publishedYes
Event9th International Symposium on String Processing and Information Retrieval, SPIRE 2002 - Lisbon, Portugal
Duration: 2002 Sep 112002 Sep 13

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume2476
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Other

Other9th International Symposium on String Processing and Information Retrieval, SPIRE 2002
CountryPortugal
CityLisbon
Period02/9/1102/9/13

ASJC Scopus subject areas

  • Theoretical Computer Science
  • Computer Science(all)

Fingerprint Dive into the research topics of 'Processing text files as is: Pattern matching over compressed texts, multi-byte character texts, and semi-structured texts'. Together they form a unique fingerprint.

  • Cite this

    Takeda, M., Miyamoto, S., Kida, T., Shinohara, A., Fukamachi, S., Shinohara, T., & Arikawa, S. (2002). Processing text files as is: Pattern matching over compressed texts, multi-byte character texts, and semi-structured texts. In A. H. F. Laender, & A. L. Oliveira (Eds.), String Processing and Information Retrieval - 9th International Symposium, SPIRE 2002, Proceedings (pp. 170-186). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 2476). Springer-Verlag. https://doi.org/10.1007/3-540-45735-6_16