The study introduces an information-theoretic approach for the quantitative assessment of unpredictability and, by extension, the originality of ideas. Two metrics are proposed, both of which are potentially useful for analyzing scientific or engineering texts. The first metric, Semantic Gain, quantifies the amount of new knowledge contained in a text relative to the knowledge embedded in a baseline model. The second metric captures informational unexpectedness (or surprisal score), which is grounded in information entropy and measures how surprising a given text is to a well-trained language model. Both metrics are applied to the abstracts of peer-reviewed engineering articles, and the results demonstrate that, despite being derived from different computational approaches and model architectures, the two metrics exhibit a strong linear correlation. This correlation may be interpreted as evidence of their mutual consistency and conceptual validity.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

AI-Based Metric for the Scientific Text Novelty

  • Anna Kruzenshtern,
  • Viktor Dodonov,
  • Leonid Chechurin

摘要

The study introduces an information-theoretic approach for the quantitative assessment of unpredictability and, by extension, the originality of ideas. Two metrics are proposed, both of which are potentially useful for analyzing scientific or engineering texts. The first metric, Semantic Gain, quantifies the amount of new knowledge contained in a text relative to the knowledge embedded in a baseline model. The second metric captures informational unexpectedness (or surprisal score), which is grounded in information entropy and measures how surprising a given text is to a well-trained language model. Both metrics are applied to the abstracts of peer-reviewed engineering articles, and the results demonstrate that, despite being derived from different computational approaches and model architectures, the two metrics exhibit a strong linear correlation. This correlation may be interpreted as evidence of their mutual consistency and conceptual validity.