Underwater object detection (UOD) faces challenges due to factors such as blurring, color distortion, and reduced visibility in underwater environments.This study focuses on enhancing underwater species detection using an improved YOLO11 model. The methodology leverages transfer learning with pre-trained weights, optimizing the model’s loss functions such as bounding box loss, confidence loss, and class loss for accurate detection, localization, and classification. Comparative performance evaluations across YOLOv10 and YOLO11 indicate that YOLO11 outperforms its predecessors in key metrics, including precision, recall, and mean Average Precision (mAP). Specifically, YOLO11 achieved an overall precision of 0.817 and recall of 0.701, with an mAP50 of 0.767 and mAP50-95 of 0.472. Larger species, such as jellyfish, stingray, and starfish, have higher precision and recall. While YOLOv10 has overall precision of 0.769 and recall of 0.714, with an mAP50 of 0.739 and mAP50-95 of 0.446, which shows that YOLO11 is better than its predecessors. Despite these advances, detection accuracy for smaller or less frequent species like puffins and penguins remains relatively lower. Overall, the YOLO11 model shows considerable improvement in detection capabilities, offering a promising solution for real-time applications in marine biology, ecological monitoring, and underwater robotics.