Can Machines and Humans Use Negation When Describing Images?

Can negation be depicted? It has been claimed in various areas, including philosophy, cognitive science, and AI, that depicting negation through visual expressions such images and pictures is challenging. Recent empirical findings have shown that humans can indeed understand certain images as expressing negation, whereas this ability is not exhibited by machine learning models trained on image data. To elucidate the computational ability underlying the understanding of negation in images, this study first focuses on the image captioning task, specifically the performance of models pre-trained on large linguistic and image datasets for generating text from images. Our experiment demonstrates that a state-of-the-art model achieves some success in generating consistent captions from images, particularly in photographs rather than illustrations. However, when it comes to generating captions containing negation from images, the model is not as proficient as humans. To further investigate the performance of machine learning models in a more controlled setting, we conducted an additional analysis using a Visual Question Answering (VQA) task. This task enables us to specify where in the image the model should focus its attention when answering a question. As a result of this setting, the model’s performance was improved. These results will shed light on the disparities in the attentional focus between humans and machine learning models.