Efficient data transfer scheme using word-pair-encoding-based compression for large-scale text-data processing

Hasitha Muthumala Waidyasooriya; Daisuke Ono; Masanori Hariyama; Michitaka Kameyama

doi:10.1109/APCCAS.2014.7032862

Efficient data transfer scheme using word-pair-encoding-based compression for large-scale text-data processing

Hasitha Muthumala Waidyasooriya, Daisuke Ono, Masanori Hariyama, Michitaka Kameyama

INF - Computer and Mathematical Sciences

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

3 Citations (Scopus)

Abstract

Large-scale data processing is very common in many fields such as data-mining, genome mapping, etc. To accelerate such processing, Graphic Accelerator Units (GPU) and FPGAs (Feild-Programmable Gate-Array) are used. However, the large data transfer time between the accelerator and the host computer is a huge performance bottleneck. In this paper, we use a word-pair-encoding method to compress the data down to 25% of its original size. The encoded data can be decoded from any position without decoding the whole data file. For some algorithms, the encoded data can be processed without decoding. Using Burrows-Wheeler algorithm based text search, we show that the data amount and transfer time can be reduced by over 70%.

Original language	English
Title of host publication	2014 IEEE Asia Pacific Conference on Circuits and Systems, APCCAS 2014
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	639-642
Number of pages	4
Edition	February
ISBN (Electronic)	9781479952304
DOIs	https://doi.org/10.1109/APCCAS.2014.7032862
Publication status	Published - 2015 Feb 5
Event	2014 IEEE Asia Pacific Conference on Circuits and Systems, APCCAS 2014 - Ishigaki Island, Okinawa, Japan Duration: 2014 Nov 17 → 2014 Nov 20

Publication series

Name	IEEE Asia-Pacific Conference on Circuits and Systems, Proceedings, APCCAS
Number	February
Volume	2015-February

Other

Other	2014 IEEE Asia Pacific Conference on Circuits and Systems, APCCAS 2014
Country	Japan
City	Ishigaki Island, Okinawa
Period	14/11/17 → 14/11/20

Keywords

Succinct data structures
big data
data compression

ASJC Scopus subject areas

Electrical and Electronic Engineering

Access to Document

10.1109/APCCAS.2014.7032862

Fingerprint Dive into the research topics of 'Efficient data transfer scheme using word-pair-encoding-based compression for large-scale text-data processing'. Together they form a unique fingerprint.

Cite this

Waidyasooriya, H. M., Ono, D., Hariyama, M., & Kameyama, M. (2015). Efficient data transfer scheme using word-pair-encoding-based compression for large-scale text-data processing. In 2014 IEEE Asia Pacific Conference on Circuits and Systems, APCCAS 2014 (February ed., pp. 639-642). [7032862] (IEEE Asia-Pacific Conference on Circuits and Systems, Proceedings, APCCAS; Vol. 2015-February, No. February). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/APCCAS.2014.7032862

Efficient data transfer scheme using word-pair-encoding-based compression for large-scale text-data processing. / Waidyasooriya, Hasitha Muthumala; Ono, Daisuke; Hariyama, Masanori; Kameyama, Michitaka.

2014 IEEE Asia Pacific Conference on Circuits and Systems, APCCAS 2014. February. ed. Institute of Electrical and Electronics Engineers Inc., 2015. p. 639-642 7032862 (IEEE Asia-Pacific Conference on Circuits and Systems, Proceedings, APCCAS; Vol. 2015-February, No. February).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Waidyasooriya, HM, Ono, D, Hariyama, M & Kameyama, M 2015, Efficient data transfer scheme using word-pair-encoding-based compression for large-scale text-data processing. in 2014 IEEE Asia Pacific Conference on Circuits and Systems, APCCAS 2014. February edn, 7032862, IEEE Asia-Pacific Conference on Circuits and Systems, Proceedings, APCCAS, no. February, vol. 2015-February, Institute of Electrical and Electronics Engineers Inc., pp. 639-642, 2014 IEEE Asia Pacific Conference on Circuits and Systems, APCCAS 2014, Ishigaki Island, Okinawa, Japan, 14/11/17. https://doi.org/10.1109/APCCAS.2014.7032862

Waidyasooriya HM, Ono D, Hariyama M, Kameyama M. Efficient data transfer scheme using word-pair-encoding-based compression for large-scale text-data processing. In 2014 IEEE Asia Pacific Conference on Circuits and Systems, APCCAS 2014. February ed. Institute of Electrical and Electronics Engineers Inc. 2015. p. 639-642. 7032862. (IEEE Asia-Pacific Conference on Circuits and Systems, Proceedings, APCCAS; February). https://doi.org/10.1109/APCCAS.2014.7032862

Waidyasooriya, Hasitha Muthumala ; Ono, Daisuke ; Hariyama, Masanori ; Kameyama, Michitaka. / Efficient data transfer scheme using word-pair-encoding-based compression for large-scale text-data processing. 2014 IEEE Asia Pacific Conference on Circuits and Systems, APCCAS 2014. February. ed. Institute of Electrical and Electronics Engineers Inc., 2015. pp. 639-642 (IEEE Asia-Pacific Conference on Circuits and Systems, Proceedings, APCCAS; February).

@inproceedings{1dc10e7ca2f34a9ebd159bc5ba41d28b,

title = "Efficient data transfer scheme using word-pair-encoding-based compression for large-scale text-data processing",

abstract = "Large-scale data processing is very common in many fields such as data-mining, genome mapping, etc. To accelerate such processing, Graphic Accelerator Units (GPU) and FPGAs (Feild-Programmable Gate-Array) are used. However, the large data transfer time between the accelerator and the host computer is a huge performance bottleneck. In this paper, we use a word-pair-encoding method to compress the data down to 25% of its original size. The encoded data can be decoded from any position without decoding the whole data file. For some algorithms, the encoded data can be processed without decoding. Using Burrows-Wheeler algorithm based text search, we show that the data amount and transfer time can be reduced by over 70%.",

keywords = "Succinct data structures, big data, data compression",

author = "Waidyasooriya, {Hasitha Muthumala} and Daisuke Ono and Masanori Hariyama and Michitaka Kameyama",

note = "Publisher Copyright: {\textcopyright} 2014 IEEE. Copyright: Copyright 2020 Elsevier B.V., All rights reserved.; 2014 IEEE Asia Pacific Conference on Circuits and Systems, APCCAS 2014 ; Conference date: 17-11-2014 Through 20-11-2014",

year = "2015",

month = feb,

day = "5",

doi = "10.1109/APCCAS.2014.7032862",

language = "English",

series = "IEEE Asia-Pacific Conference on Circuits and Systems, Proceedings, APCCAS",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

number = "February",

pages = "639--642",

booktitle = "2014 IEEE Asia Pacific Conference on Circuits and Systems, APCCAS 2014",

edition = "February",

}

TY - GEN

T1 - Efficient data transfer scheme using word-pair-encoding-based compression for large-scale text-data processing

AU - Waidyasooriya, Hasitha Muthumala

AU - Ono, Daisuke

AU - Hariyama, Masanori

AU - Kameyama, Michitaka

PY - 2015/2/5

Y1 - 2015/2/5

N2 - Large-scale data processing is very common in many fields such as data-mining, genome mapping, etc. To accelerate such processing, Graphic Accelerator Units (GPU) and FPGAs (Feild-Programmable Gate-Array) are used. However, the large data transfer time between the accelerator and the host computer is a huge performance bottleneck. In this paper, we use a word-pair-encoding method to compress the data down to 25% of its original size. The encoded data can be decoded from any position without decoding the whole data file. For some algorithms, the encoded data can be processed without decoding. Using Burrows-Wheeler algorithm based text search, we show that the data amount and transfer time can be reduced by over 70%.

AB - Large-scale data processing is very common in many fields such as data-mining, genome mapping, etc. To accelerate such processing, Graphic Accelerator Units (GPU) and FPGAs (Feild-Programmable Gate-Array) are used. However, the large data transfer time between the accelerator and the host computer is a huge performance bottleneck. In this paper, we use a word-pair-encoding method to compress the data down to 25% of its original size. The encoded data can be decoded from any position without decoding the whole data file. For some algorithms, the encoded data can be processed without decoding. Using Burrows-Wheeler algorithm based text search, we show that the data amount and transfer time can be reduced by over 70%.

KW - Succinct data structures

KW - big data

KW - data compression

UR - http://www.scopus.com/inward/record.url?scp=84937944277&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84937944277&partnerID=8YFLogxK

U2 - 10.1109/APCCAS.2014.7032862

DO - 10.1109/APCCAS.2014.7032862

M3 - Conference contribution

AN - SCOPUS:84937944277

T3 - IEEE Asia-Pacific Conference on Circuits and Systems, Proceedings, APCCAS

SP - 639

EP - 642

BT - 2014 IEEE Asia Pacific Conference on Circuits and Systems, APCCAS 2014

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 2014 IEEE Asia Pacific Conference on Circuits and Systems, APCCAS 2014

Y2 - 17 November 2014 through 20 November 2014

ER -