Comparison of Semantic Reproducibility and Repeatability in Large and Small Language Models: The Impact of Temperature
摘要
This study compared two generative models of different sizes, one with 8 billion parameters (Model A) and the other with 45 billion parameters (Model B). The main goal was to analyze the semantic reproducibility of responses to the same questions formulated in different ways, while measuring the impact of temperature on text generation. The results show that the larger model (B) produces more consistent answers but is also more prone to hallucinations, with a rate of 25.3%, compared to 2.7% for Model A. Model A, although less consistent, generates fewer hallucinations, suggesting that greater uncertainty can sometimes lead to fewer systematic errors. In contrast, Model B generates longer and more detailed responses, but errors are often repeated consistently. This highlights the importance of developing more refined model evaluations, incorporating factual accuracy measures and error stability. While this is not the first study to examine temperature effects, our approach stands out by combining a semantic reproducibility score with hallucination analysis, an approach rarely explored together in the current literature.