The primary difficulty in Moment Retrieval (MR) and Highlight Detection (HD) tasks is the accurate localization of moments that align with a text query, coupled with the evaluation of the correlation between the video at various temporal positions and the query. DETR-like methodologies, due to their superior effectiveness in multimodal interactions, have emerged as the predominant paradigm in addressing MR/HD tasks. However, these methodologies are deficient in jointly augmenting both video and text modalities, thus constraining the performance. To this end, we introduce MS-DETR, a Transformer-based architecture specifically designed for MR/HD. Our model comprises three key components: 1) Local Temporal Enhancer, which incorporates the continuity of local temporal features into video coding to increase the receptive field in the temporal dimension; 2) Modality Synergy Encoder, to achieve cross-modal synergy learning with two-way cross-attentions for efficient feature fusion of text and video; and 3) Cross-Modality Decoder, to achieve cross-modal collaborative query with two-step cross-attentions, forming a more accurate moment interval prediction. We validate the collaborative learning ability of MS-DETR on multiple mainstream MR/HD datasets, and demonstrate the effectiveness of our proposed techniques through extensive ablation studies.