We propose a framework for reducing overdetection on in-the-wild data by unsupervised anomaly detection, where only anomaly-free images are provided for training. In recent years, the accuracy of anomaly detection has significantly improved through vision-language models (VLMs). However, related approaches suffer from severe overdetection for images captured in real-world scenarios such as car-mounted and drone-captured imagery because they often involve a wider angle of view and extremely complicated backgrounds. Therefore, in this paper, we tackle this challenge for such in-the-wild datasets by using token optimization (TO) and score correction (SC) using anomaly-free images. TO aims to obtain richer prompt description, which lead to better VLM performance while SC can directly reduce overdetection in the image domain. In the proposed framework, VLMs can effectively reduce overdetection through both TO and SC approaches. Experimental results demonstrate the superior performance of the proposed method, using LostandFound and ShanghaiTech-Campus datasets, which are publicly available in-the-wild datasets.