<p>Fish play a crucial role in marine ecosystems, making their detection and classification essential for efficient control and monitoring of the aquatic environment. However, the current approaches face challenges such as computational complexity, fine-grained species variation, object occlusion, underwater noise, and the lack of large annotated datasets. Furthermore, wild fish recognition in open-set conditions presents a fine-grained classification problem, compounded by poor underwater visibility and the need to distinguish visually similar species, all of which hinder both ecological research and automation in aquaculture systems. The study aims to introduce a comprehensive fish species detection and classification approach that overcomes the limitations of conventional methods, including small object recognition, misclassification, and limited data availability. The proposed model utilizes a bilateral filtering technique for noise removal and an unsharp mask filtering for image enhancement to ensure high-quality input for the detection model. Object annotation is performed to label each pixel to train the detection algorithm with high precision. Finally, the model introduces a space-to-depth convolution approach and employs separable convolutions to capture finer details of small objects, integrated with the YOLOv11 architecture to improve detection and classification accuracy. Experimental results demonstrate that the introduced model outperforms current state-of-the-art approaches among several performance metrics. Particularly, it attains 99.82% accuracy in fish species detection and classification. These findings emphasize the model’s effectiveness in fine-grained species recognition and its potential applications in intelligent aquaculture, endangered species tracking, and marine biodiversity research.</p>