N-GRAM LANGUAGE MODEL ADAPTATION USING SMALL CORPUS FOR SPOKEN DIALOG RECOGNITION

Akinori Ito; Hideyuki Saitoh; Masaharu Katoh; Masaki Kohda

N-GRAM LANGUAGE MODEL ADAPTATION USING SMALL CORPUS FOR SPOKEN DIALOG RECOGNITION

Akinori Ito, Hideyuki Saitoh, Masaharu Katoh, Masaki Kohda

Research output: Contribution to conference › Paper › peer-review

Abstract

This paper describes an N-gram language model adaptation technique. As an N-gram model requires a large size sample corpus for probability estimation, it is difficult to utilize N-gram model for a specific small task. In this paper, N-gram task adaptation is proposed using large corpus of the general task (TI text) and small corpus of the specific task (AD text). A simple weighting is employed to mix TI and AD text. In addition to mix two texts, the effect of vocabulary is also investigated. The experimental results show that adapted N-gram model with proper vocabulary size has significantly lower perplexity than the task independent models.

Original language	English
Pages	2735-2738
Number of pages	4
Publication status	Published - 1997
Externally published	Yes
Event	5th European Conference on Speech Communication and Technology, EUROSPEECH 1997 - Rhodes, Greece Duration: 1997 Sept 22 → 1997 Sept 25

Conference

Conference	5th European Conference on Speech Communication and Technology, EUROSPEECH 1997
Country/Territory	Greece
City	Rhodes
Period	97/9/22 → 97/9/25

ASJC Scopus subject areas

Computer Science Applications
Software
Linguistics and Language
Communication

Cite this

@conference{d0a38e14ba0544ac88115e65400f409c,

title = "N-GRAM LANGUAGE MODEL ADAPTATION USING SMALL CORPUS FOR SPOKEN DIALOG RECOGNITION",

abstract = "This paper describes an N-gram language model adaptation technique. As an N-gram model requires a large size sample corpus for probability estimation, it is difficult to utilize N-gram model for a specific small task. In this paper, N-gram task adaptation is proposed using large corpus of the general task (TI text) and small corpus of the specific task (AD text). A simple weighting is employed to mix TI and AD text. In addition to mix two texts, the effect of vocabulary is also investigated. The experimental results show that adapted N-gram model with proper vocabulary size has significantly lower perplexity than the task independent models.",

author = "Akinori Ito and Hideyuki Saitoh and Masaharu Katoh and Masaki Kohda",

note = "Publisher Copyright: {\textcopyright} 1997 5th European Conference on Speech Communication and Technology, EUROSPEECH 1997. All rights reserved.; 5th European Conference on Speech Communication and Technology, EUROSPEECH 1997 ; Conference date: 22-09-1997 Through 25-09-1997",

year = "1997",

language = "English",

pages = "2735--2738",

}

TY - CONF

T1 - N-GRAM LANGUAGE MODEL ADAPTATION USING SMALL CORPUS FOR SPOKEN DIALOG RECOGNITION

AU - Ito, Akinori

AU - Saitoh, Hideyuki

AU - Katoh, Masaharu

AU - Kohda, Masaki

PY - 1997

Y1 - 1997

N2 - This paper describes an N-gram language model adaptation technique. As an N-gram model requires a large size sample corpus for probability estimation, it is difficult to utilize N-gram model for a specific small task. In this paper, N-gram task adaptation is proposed using large corpus of the general task (TI text) and small corpus of the specific task (AD text). A simple weighting is employed to mix TI and AD text. In addition to mix two texts, the effect of vocabulary is also investigated. The experimental results show that adapted N-gram model with proper vocabulary size has significantly lower perplexity than the task independent models.

AB - This paper describes an N-gram language model adaptation technique. As an N-gram model requires a large size sample corpus for probability estimation, it is difficult to utilize N-gram model for a specific small task. In this paper, N-gram task adaptation is proposed using large corpus of the general task (TI text) and small corpus of the specific task (AD text). A simple weighting is employed to mix TI and AD text. In addition to mix two texts, the effect of vocabulary is also investigated. The experimental results show that adapted N-gram model with proper vocabulary size has significantly lower perplexity than the task independent models.

UR - http://www.scopus.com/inward/record.url?scp=44949220766&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=44949220766&partnerID=8YFLogxK

M3 - Paper

AN - SCOPUS:44949220766

SP - 2735

EP - 2738

T2 - 5th European Conference on Speech Communication and Technology, EUROSPEECH 1997

Y2 - 22 September 1997 through 25 September 1997

ER -

N-GRAM LANGUAGE MODEL ADAPTATION USING SMALL CORPUS FOR SPOKEN DIALOG RECOGNITION

Abstract

Conference

ASJC Scopus subject areas

Other files and links

Fingerprint

Cite this