Multimodal large language models (MLLMs) have demonstrated impressive performance across a wide range of vision-language tasks, primarily due to large-scale pretraining and image-text alignment strategies. However, whether these models genuinely possess visual semantic understanding—particularly the ability to accurately perceive and distinguish subtle semantic differences between highly similar images—remains underexplored and lacks systematic evaluation. While single-image benchmarks assess a model’s ability to interpret isolated visual content, they offer limited insight into its capacity to detect and reason about semantic deltas between nearly identical scenes—a skill crucial in real-world tasks such as surveillance and visual inspection. To fill this gap, we introduce DeltaMMEval, a structured benchmark that employs minimal yet meaningful semantic edits between image-text pairs, enabling precise evaluation of a model’s perceptual sensitivity, contrastive reasoning, and alignment consistency—capabilities not reliably assessed through single-image tasks. DeltaMMEval explicitly decomposes visual semantic differences into three hierarchical levels: scene-level, object-level, and attribute-level, facilitating structured attribution and fine-grained diagnostic analysis of model behavior. We also introduce Group Accuracy, a stricter metric that assesses model consistency across multiple contrastive decisions. Experimental results show that even top-tier closed-source models, such as GPT-4o, achieve group matching accuracies of only  76.70% on this benchmark, revealing a significant performance gap of nearly 20% points compared to human performance (95.68%). These findings highlight a substantial deficit in visual semantic understanding, especially in tasks requiring sensitivity to fine-grained semantic differences. The datasets will be released as soon as possible.