/

On the Vocabulary of Grammar-Based Codes and the Logical Consistency of Texts

Łukasz Dębowski
Arxiv ID: 0810.3125Last updated: 3/11/2020
The article presents a new interpretation for Zipf-Mandelbrot's law in natural language which rests on two areas of information theory. Firstly, we construct a new class of grammar-based codes and, secondly, we investigate properties of strongly nonergodic stationary processes. The motivation for the joint discussion is to prove a proposition with a simple informal statement: If a text of length n describes n^β independent facts in a repetitive way then the text contains at least n^β/log n different words, under suitable conditions on n. In the formal statement, two modeling postulates are adopted. Firstly, the words are understood as nonterminal symbols of the shortest grammar-based encoding of the text. Secondly, the text is assumed to be emitted by a finite-energy strongly nonergodic source whereas the facts are binary IID variables predictable in a shift-invariant way.

PaperStudio AI Chat

I'm your research assistant! Ask me anything about this paper.
About
Pricing
Commercial Disclosure
Contact
© 2023 Paper Studio™. All Rights Reserved.