Leather defect detection is of significant importance in industrial production, allowing for the early identification and elimination of defective leather to ensure that products meet high-quality standards. Although CNNs have been successfully applied to defect detection, existing CNN-based networks focus more on local features and lack the ability to represent global features, making it difficult to detect minor defects. To address these issues, we propose an Agent-Based Adaptive Feature Enhanced Network (AAFE-Net), which combines the advantages of CNNs and Transformers. By using the Adaptive Weighted Feature Enhancement Module (AWFEM) and the AgentMobileViT Module (AMVTM), AAFE-Net achieves efficient feature extraction and fusion. Additionally, we have collected a dataset specifically for leather defect detection. Experiments conducted on both private and public datasets for defect detection demonstrate that AAFE-Net outperforms existing methods in terms of mIoU and mPA evaluation metrics, while also consuming fewer parameters and FLOPs than most existing methods, exhibiting high detection performance and computational efficiency.