The ocean’s abundant resources have led coastal countries to commit to developing marine ranches. Automatic underwater seafood species detection technology contributes to the development of the industry. The technology still faces challenges, such as small-sized targets, complex underwater backgrounds, and hardware resource limitations. We propose HLD-DETR, a lightweight model for efficient underwater small seafood species detection based on the improved RT-DETR architecture. We propose a Hierarchical Partial Fusion network (HPFnet), an efficient feature extraction network with the Hierarchical Partial Feature Integration (HPFI) block, achieving reduced model parameters. We introduce a multi-scale attention mechanism called Low Frequency Attention and High Frequency Attention (LoHi) for mitigating background interference. To strengthen the model’s feature fusion capability, we also integrate DySample in the task. The experimental evaluation supports the potential effectiveness of these modules. HLD-DETR consistently demonstrates superior performance on the URPC dataset over other classic models with a reduction in parameter count by 27.8% when evaluated against the baseline.