A misshapen mango recognition algorithm which is named YOLOv7-AIFI based on improved YOLOv7 (You Only Look Once version 7) is proposed to improve the recognition accuracy of misshapen mango. Firstly, the AIFI (Attention-based Intra-scale Feature Interaction) module is introduced into this algorithm to reduce computational redundancy on the basis of the YOLOv7 model. Secondly, three CBAM (Convolutional Block Attention Modules) are introduced to enhance the model’s generalization capability. Finally, the MPD-IOU (Maximized Position-Dependent Intersection over Union) loss function is selected to optimize the calculation method and reduce the computational load. Experiments were conducted on the self-built mango data set to verify that the improvement of YOLOv7-AIFI can effectively improve the performance of the model and make it lightweight. Compared to YOLOv7, the accuracy of YOLOv7-AIFI identifying ripe mangoes and misshapen mangoes is improved by 5.9% and 0.17% respectively, the F1-Score is increased by 0.13%, the mean average precision (mAP) is increased by 3.34%, and the model weight is reduced by 2.7MB. Moreover, the mAP and F1-Score of YOLOv7-AIFI surpass other commonly used models comprehensively. Experimental results demonstrate that the proposed YOLOv7-AIFI model can effectively improve the accuracy and efficiency of misshapen mango recognition, providing a new idea for the recognition of misshapen mango.