Coffee is one of Vietnam’s key agricultural exports, and ensuring the quality of harvested beans relies heavily on accurately sorting coffee fruit by ripeness. However, this process is still predominantly manual, making it labor-intensive and prone to inefficiency. This study seeks to address this issue by developing an automated system capable of detecting the ripeness of coffee fruit using state-of-the-art machine learning models. The research investigates various popular classification models, comparing their performance against a proposed custom Convolutional Neural Network (CNN) for classifying coffee fruit ripeness. The latest object detection models, including YOLOv7, YOLOv8, and RT-DETR, are evaluated for their effectiveness in detecting coffee fruit. Comparative analysis shows that the proposed CNN achieves the highest classification accuracy of 92.58%, while YOLOv8 stands out as the most well-rounded model for object detection, with a mAP@50 score of 92.47%.