Zipf’s law is an instance of a power law that describes the frequency of an event in a ranked dataset to be inversely proportional to its own rank. With the boom of AI in recent years, Zipf’s law can now be used as a benchmark to judge the naturalness of large language model (LLM) generated text, particularly in different text sizes. In this analysis, concepts like the Pareto principle and the principle of least effort were explored to see if LLM-generated text adhered to these patterns. The results indicated that while LLMs do show properties of human-Zipfian patterns, they did have some minor deviations as well as some unexpected trends as the size of the text increased.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Zipf’s Law and Its Application in Large Language Models: A Comprehensive Analysis

  • Advaith Sujith

摘要

Zipf’s law is an instance of a power law that describes the frequency of an event in a ranked dataset to be inversely proportional to its own rank. With the boom of AI in recent years, Zipf’s law can now be used as a benchmark to judge the naturalness of large language model (LLM) generated text, particularly in different text sizes. In this analysis, concepts like the Pareto principle and the principle of least effort were explored to see if LLM-generated text adhered to these patterns. The results indicated that while LLMs do show properties of human-Zipfian patterns, they did have some minor deviations as well as some unexpected trends as the size of the text increased.