<p>Assessing the alignment between textual descriptions and corresponding images is fundamental to multi-modal research. In recent years, there has been a surge in the adoption of reference-free methods that utilize visual-language pre-trained models (VLMs). Empirical evidence supports that these innovative methods correlate more closely with human judgment, representing a notable progression in the field. However, due to the unknown underlying judgment mechanisms within VLMs, the metrics designed based on VLMs may exhibit some unidentified flaws. To uncover potential issues with the reference-free metrics, we employ a reinforcement learning approach to hack these metrics, guiding the model to generate sentences that better align with the metric criteria. If the metrics contain some flaws, these deficiencies will manifest in the generated sentences. On the hacking experiment, we observe that the generated sentences achieve higher metric scores, yet they also become unreadable. These inconsistencies reflect the inherent flaws within the metrics themselves. To address these issues, we propose a simple but effective approach by introducing sentences with flaws as negative samples in contrastive learning called Negative Text Contrastive Learning (NTCL). We utilize GPT-4V as an evaluation tool to analyze the generated sentences, and our results demonstrate that the NTCL method is more robust and achieves state-of-the-art performance. We hope our findings can raise awareness in the community about the importance of reference-free image captioning metrics hacking and pave the way for the design of more robust metrics.</p>