<p>Synthetic aperture radar (SAR) object detection in complex scenes such as harbors and near-shore areas is often affected by speckle noise and structural clutter, which leads to missed detections of small objects, false alarms, and unstable localization. To address these issues, we propose Scatter-Aware Interaction Network(SAI-Net), an end-to-end detection framework tailored for complex scattering backgrounds. SAI-Net consists of three cooperative components. First, a Shift-wise Conv (SWC) backbone performs multi-scale feature extraction by introducing shift-wise spatial interactions to enlarge the effective receptive field with low overhead, yielding more stable structural scattering representations. Second, we introduce a BiFPN-Rep Residual(BRR) Fusion neck that combines adaptive resampling alignment with re-parameterized residual fusion within a bidirectional pyramid information flow, enabling stable multi-scale fusion with consistent cross-scale alignment. Finally, we design Enhanced AIFI(E-AIFI) for scatter-aware interactive encoding, which jointly models long-range consistency aggregation via adaptive sparse self-attention, detail compensation via a spatially enhanced feed-forward module, and multi-scale spatial interactions to improve both global discriminability and localizability. Extensive experiments and visual analyses on OGSOD and HRSID demonstrate that SAI-Net achieves consistent improvements in overall accuracy as well as small-object and strict localization metrics, validating the effectiveness of the proposed scatter-aware interaction and cross-scale fusion design for challenging SAR scenarios.</p>