Recent advancements in food processing have led to the automation of tasks traditionally performed by human operators, such as identifying food types and detecting spoilage based on external peel damage. Manual inspection approaches require significant human effort and are prone to subjectivity, which frequently leads to inefficiencies. To address these challenges, a deep learning approach is developed utilizing a customized Gabor-filter-based Convolutional Neural Network (CNN) model, implemented using Python, for both identification and spoilage detection. The methodology comprises two sequential stages: classifying the type of fruit or vegetable and assessing its freshness level. It has fine-tuned the Gabor filter-based CNN on a curated dataset encompassing various produce types and ripeness stages. In traditional practices, grading is performed manually, making it vulnerable to variability, whereas the developed automated pipeline demonstrates superior speed and objectivity. The model was rigorously evaluated across a diverse dataset of fruits and vegetables. The results indicate that the developed customized Gabor filter-based CNN architecture achieves high accuracy in both identifying food type and detecting spoilage. This demonstrates its potential to enhance precision, reduce food waste, and ensure the delivery of fresh and safe food to consumers.