Measuring the Sensitivity of Image Captioning Metrics to Caption Perturbations

Image captioning systems based on deep neural networks have made significant progress in generating image descriptions in fluent language. Despite the achieved success, one of the main challenges remains automatic evaluation of the results. Recently, research efforts focused on the development of increasingly sophisticated automatic metrics have intensified significantly, so finding an appropriate way to compare metrics has also gained importance. A good automatic metric should match the ratings human evaluators would assign to captions, so a common way to evaluate them is to compute the correlation with human judgments. In this paper, we chose a different approach that could complement the results of correlation analysis in order to get a more complete insight into a particular metric. In this regard, we analyzed how various caption perturbations affect the scores of selected image captioning metrics. We constructed a dataset of various caption modifications and analyzed the changes in scores resulting from such perturbations.