Lognormality of the distribution of Japanese sentence lengths

Sho Furuhashi, Yoshinori Hayakawa

    Research output: Contribution to journalArticle

    3 Citations (Scopus)

    Abstract

    The lengths of sentences in written texts have been reported to exhibit characteristic distributions that resemble lognormal distributions. However, the mechanism responsible for such lognormality is unclear. In this quantitative study, we analyze over 10,000 Japanese sentences from out-of-copyright Japanese texts stored on Aozora Bunko. We first confirm that sentence length distributions can be better represented by the lognormal function than by other functions (e.g., the gamma distribution). Next, under the assumption that each sentence is generated by a hierarchical branching process in terms of dependency trees, we test whether the composition of sentences can be explained by a simple multiplicative process by utilizing the Japanese dependency analyzer CaboCha. The results imply that the lognormality of sentence length distributions originates from the dependency tree depth and that a simple multiplicative model cannot accurately model the processes involved in generating sentences.

    Original languageEnglish
    Article number034004
    JournalJournal of the Physical Society of Japan
    Volume81
    Issue number3
    DOIs
    Publication statusPublished - 2012 Mar 1

    Keywords

    • Dependency tree
    • Japanese dependency structure
    • Lognormal distribution
    • Multiplicative process
    • Sentence length

    ASJC Scopus subject areas

    • Physics and Astronomy(all)

    Fingerprint Dive into the research topics of 'Lognormality of the distribution of Japanese sentence lengths'. Together they form a unique fingerprint.

  • Cite this