The detection head framework critically influences the balance between classification and localization in small object detection, yet existing designs often neglect task-specific feature interactions, leading to optimization conflicts. To address this, we propose Bi-Level Attention Head (BLAH), a novel framework that harmonizes dual-task learning through structured attention mechanisms and adaptive loss optimization. BLAH introduces two key innovations: (1) Channel Group Self-Attention (CGSA) stacks, which dynamically recalibrate channel-group dependencies to align classification and localization features, resolving spatial-channel decoupling limitations in conventional attention. (2) Dual-Task Attention (DTA), integrating global channel attention for classification robustness (translation invariance) and local spatial attention for precise localization (translation variability), enabling synergistic task interaction without computational overhead. Further, we design a Differentiable Task-Balanced Loss (DTBL) that adaptively modulates gradients between tasks via cosine similarity constraints, ensuring stable optimization without extra parameters. Extensive experiments on MS COCO and VisDrone demonstrate BLAH’s superiority. When integrated with DETR, Deformable DETR, and YOLOv10, BLAH achieves +1.2% mAP on COCO over state-of-the-art detectors (e.g., YOLO-based, DETR-based) while maintaining inference efficiency, and significantly improves small-object detection (e.g., +4.5% \(AP_S\) on YOLOv12). Ablation studies validate each component’s necessity.