In the domain of object detection, the utilization of multi-scale features has been demonstrated to substantially enhance the performance of the DETR model and its variants on the COCO benchmark. However, this improvement is often accompanied by a significant increase in token sequence length, resulting in a considerable computational overhead. We observe that, in multi-scale inter actions, not all tokens involved in the computation contribute equally to the final prediction. To address this issue, we propose a HybridScale-GuideTokens DETR method and design three core modules aimed at efficiently compressing fore ground tokens and high-confidence background tokens to reduce computational costs. These three modules are as follows: (1) Foreground-Background Token Selector (FBTS), which selectively removes low-confidence background tokens to minimize redundant computation; (2) Foreground Intra-Scale Feature Interaction (FIntraSF), which optimizes the interaction between foreground tokens and high-confidence background regions, retaining only meaningful computational interactions; (3) Guide Token Cross-Scale Background Fusion (GTCBF), which restores filtered background features to enhance multi-scale feature representation. Extensive experimental results on the COCO dataset demonstrate that the HybridScale-GuideTokens DETR achieves an AP of 50.8, outperforming the DINO + Sparse DETR baseline by 2.6 AP. Moreover, the proposed modules can be seamlessly integrated into other DETR variants. For instance, when incorporated into the DINO framework, our method reduces computational costs by 44%, accelerates inference speed by 39%, and improves accuracy by 0.4 points. Finally, compared to DINO + FPN, our method reduces memory usage by a factor of three, striking a strong balance between efficiency and accuracy.