Zipf’s Law and Its Application in Large Language Models: A Comprehensive Analysis
摘要
Zipf’s law is an instance of a power law that describes the frequency of an event in a ranked dataset to be inversely proportional to its own rank. With the boom of AI in recent years, Zipf’s law can now be used as a benchmark to judge the naturalness of large language model (LLM) generated text, particularly in different text sizes. In this analysis, concepts like the Pareto principle and the principle of least effort were explored to see if LLM-generated text adhered to these patterns. The results indicated that while LLMs do show properties of human-Zipfian patterns, they did have some minor deviations as well as some unexpected trends as the size of the text increased.