Detecting abnormal plums is essential in the agricultural industry to ensure quality control and maximize market value. Traditional methods of identifying defective fruits are often labor-intensive and subjective. This paper proposes a solution for abnormal plums detection with two phases: (1) plums localization and (2) anomaly detection. We use EfficientAD-M, an unsupervised learning method, for anomaly detection. Experimental results demonstrate that our proposed solution can effectively detect abnormal plums. The system achieves an accuracy of 82.66% and a throughput of 231 images per second, making it suitable for real-time applications in production line settings. These results contribute to demonstrating the potential of unsupervised learning for accurate, rapid, and cost-effective fruit quality control.