Search Results for author:
Found 7 papers, 1 papers with code
Corrections of Zipf's and Heaps' Laws Derived from Hapax Rate Models
The article introduces corrections to Zipf's and Heaps' laws based on systematic models of the proportion of hapaxes, i. e., words that occur once.
A Simplistic Model of Neural Scaling Laws: Multiperiodic Santa Fe Processes
It was observed that large language models exhibit a power-law decay of cross entropy with respect to the number of parameters and training tokens.
Local Grammar-Based Coding Revisited
It has been known that such minimal codes are strongly universal for a strictly positive entropy rate, whereas the number of rules in the minimal grammar constitutes an upper bound for the mutual information of the source.
Bounds for Algorithmic Mutual Information and a Unifilar Order Estimator
Using both results, we show that all (also uncomputable) sources of a finite unifilar order exhibit sub-power-law growth of algorithmic mutual information and of the unifilar order estimator.
Information Theory Information Theory 62M05 (Primary) 60G10, 94A17, 94A29 (Secondary)
Is Natural Language a Perigraphic Process? The Theorem about Facts and Words Revisited
As we discuss, a stationary stochastic process is nonergodic when a random persistent topic can be detected in the infinite random text sampled from the process, whereas we call the process strongly nonergodic when an infinite sequence of independent random bits, called probabilistic facts, is needed to describe this topic completely.
A Preadapted Universal Switch Distribution for Testing Hilberg's Conjecture
Hilberg's conjecture about natural language states that the mutual information between two adjacent long blocks of text grows like a power of the block length.
Constant conditional entropy and related hypotheses
We show that constant entropy rate (CER) and two interpretations for uniform information density (UID), full UID and strong UID, are inconsistent with these laws.
