Assessing the Stability of Text-to-Text Models for Keyword Generation Tasks

The paper investigates the stability of text-to-text T5 models in keyword generation tasks, highlighting the sensitivity of their results to subtle experimental variations such as the seed used to shuffle fine-tuning data. The authors advocate for incorporating error bars and standard deviations when reporting results to account for this variability, which is common practice in other domains of data science, but not common in keyphrase generation. Through experiments with T5 models, they demonstrate how small changes in experimental conditions can lead to significant variations in model performance, particularly with larger model sizes. Furthermore, they analyze the coherence within a family of models and propose novel approaches to assess the stability of the model. In general, the findings underscore the importance of considering experimental variability when evaluating and comparing text-to-text models for keyword generation tasks.