Multimodal Detection of Image Misrepresentation
摘要
In this chapter, we explore the latest advancements in detecting decontextualised and miscaptioned images, i.e., images whose captions or accompanying text represent their origin, context, or meaning in a false or misleading way. We examine key aspects of the field, including the development of annotated, weakly annotated, and synthetically generated datasets; advances in multimodal feature extraction, modality fusion, and detection models; approaches that incorporate external information as evidence; and the emerging role of large vision–language models in this task. Furthermore, we present our contributions within the context of the vera.ai research project, which include the creation of synthetic training datasets, real-world evaluation benchmarks, and methodologies aimed at improving model generalisation, mitigating unimodal bias, and strengthening the reliability of evidence-based verification.