Fish surface damage detection is an indispensable part of aquaculture. This study proposes an improved YOLOv7 network based on the coordinate attention (CA) mechanism and the Scylla-intersection over Union (SIOU) loss for automatic fish surface damage identification. The CA is embedded in the backbone of the YOLOv7 to capture the direction-aware and position-sensitive information across channels, enabling the model to enhance the discriminative feature of fish damage and suppress background interference. Moreover, the SIOU localization loss is adopted to replace the complete cross union (CIoU) loss in the original YOLOv7. It improves the convergence speed and the detection efficiency of the model by considering the orientation information between the real box and the predicted box. A fish surface damage dataset is constructed using data collected from an actual farm vessel. Extensive experimental results demonstrate that the proposed method is superior to the original YOLOv7 network, with an enhancement in average precision (at an intersection over the union of 0.5) from 78.90% to 80.60%.