Lin Hsin Hsin Artificial Intelligence Center
FRONTIER MODELS
Frequency of Occurrence
🐜🗨
Frequency of Occurrence
MATTERS
Large Language Models (LLMs)
& Combinatorics
if frequency is ignored
300,000 words (or a large context) leads to wastage
The Combinatorial Explosion vs Real Frequency
The core issue is the gap between theoretical possibilities and actual occurrence.
📍 Theoretical Space
With a vocabulary of just 50,000 words, the number of possible sentences of even moderate length is astronomical (far exceeding the number of atoms in the universe) -- Yes, the combinatorial universe.
📍 Real Frequency
In reality, humans only use a tiny, highly structured fraction of these combinations. Meaningful language follows strict syntactic and semantic rules, making the frequency of occurrence of valid sentences highly non-uniform.
The Wastage
If a model (or a training strategy) treats all combinations as equally probable or attempts to enumerate them without weighting by frequency, it wastes immense
computational resources on nonsensical
never-occurring sequences.
This is the wastageof computing power spent on the zero-frequencyor near-zero-frequency
parts of the combinatorial space.