From Training-Free to Adaptive: Empirical Insights into MLLMs’ Understanding of Detection Information

Multimodal Large Language Models (MLLMs) exhibit impressive general-purpose capabilities, but struggle with fine-grained visual details. To mitigate this issue, a common remedy is to infuse textual information from specialized detectors (e.g., object coordinates, OCR results). However, this is typically done in a training-free manner. There is limited research on the effects of adaptive training, which has great potential for helping LLMs better comprehend the special input. This paper investigates a critical question: How does training shape an MLLM’s ability to leverage textual detection information? With representative models, we conduct a systematic study comparing three strategies: training-free infusion, full retraining, and lightweight fine-tuning. Our findings, evaluated across 10 benchmarks, reveal that a simple fine-tuning strategy is remarkably effective, outperforming its training-free counterpart by a significant margin (e.g., 6.71% average improvement). We uncover three key mechanisms behind this success: (1) Fine-tuning enhances robustness, enabling the MLLM to filter noise from imperfect detector outputs. (2) Unlike retraining from scratch, fine-tuning preserves the MLLM’s pre-trained multimodal feature synergy, avoiding performance degradation on general vision tasks. (3) Crucially, fine-tuning teaches the MLLM to understand the abstract semantics of detection information, allowing it to generalize to unseen detectors and novel information formats. Our work provides empirical evidence that adaptive fine-tuning is a vital step for effectively integrating external symbolic knowledge into MLLMs. We release our code to facilitate future research in this direction.