<p>Prolonged hypoxic conditions pose a significant threat to the survival of fish in aquaculture, often leading to mass mortality events. Abnormal fish behavior, particularly under hypoxic stress, can be an early warning indicator of decreasing dissolved oxygen levels in water. However, existing methods for detecting hypoxic stress behavior in fish are affected by the lighting, occlusion, and turbidity in real aquaculture environments. This results in low accuracy in detecting hypoxic stress behaviors. In this paper, we propose a real-time detection method, YOLOv8n-HSB, designed to enhance the accuracy of detecting hypoxic stress behavior in tilapia within recirculating aquaculture systems. Key improvements of our approach include (1) the introduction of the Multi-scale Fusion Pyramid Network (MFP-Net), which enhances small object detection by adding a specific layer at the bottom of the feature pyramid and improving feature fusion based on Bi-directional Feature Pyramid Network (BIfpn) architecture for the neck structure; (2) the development of the C2f-Occlusion Perception (C2f-OP) module in the backbone by integrating Mobile Inverted Residual Bottleneck Convolution (MBConv) and Effective Squeeze-and-Excitation (ESE), improving the model’s ability to capture crucial local features; and (3) the replacement of conventional Convolution (Conv) layers with Dynamic Convolution (DConv) modules integrated with ParameterNet (P-DConv), enhancing the model’s capacity to process complex information and extract fine-scale features of fish. Experimental results demonstrate that the YOLOv8n-HSB model is highly effective for detecting hypoxic stress behavior in tilapia.&#xa0;Compared to the original YOLOv8n model, the AP_@0.5:0.95 increases by 4.05%. The AP_@0.5 reaches 96.12%, outperforming existing state-of-the-art methods. This study provides a novel method for monitoring the abnormal behavior of fish in hypoxic environments, offering practical significance for smart aquaculture systems.</p>