Accurate counting of shrimp larvae is essential for feed optimization, density control, and yield prediction in aquaculture. However, larvae counting is challenging due to their small size, high flexibility, and frequent occlusion in dense scenes. Manual methods are time-consuming, error-prone, and potentially harmful to the larvae, while existing automated solutions are often expensive and impractical. In this paper, we propose a Transformer-based framework called ShrimpFormer-X for end-to-end shrimp larvae counting and localization. Unlike conventional methods that rely solely on regression or detection, ShrimpFormer-X integrates both counting and localization tasks. It introduces a hybrid encoder that applies attention selectively to high-level features while leveraging bidirectional fusion to enhance multi-scale representation. A decoupled cross-attention decoder processes contextual and spatial information separately by concatenating image features and positional encodings. Furthermore, a density partition module identifies dense and sparse regions to enable adaptive decoding. Extensive experiments on a large-scale shrimp larvae dataset demonstrate that ShrimpFormer-X achieves state-of-the-art performance in both low and high-density conditions. The model outperforms existing methods in both accuracy and computational efficiency while providing reliable localization.