<p>Video Moment Retrieval (MR) and Highlight Detection (HD) have garnered considerable interest owing to the increasing need for video analysis. Although existing methods based on attention mechanism have made substantial progress, they neglect the correlation and difference among video clips, being detrimental to overall perception capacity of text queries. Motivated by such a challenge, we propose a novel framework called Temporal Refinement and Multi-grained Matching DETR (TRMM-DETR). Specifically, it leverages a video temporal refinement module to address the issue of insufficiently extracting the correlation between adjacent video clips, enhancing the ability to recognize video clips through learning the correlation and difference between adjacent clips. Additionally, we devise a cross-modal multi-grained matching module, which calculates the similarity between video clips, text words and the complete text query, facilitating the integration of these factors to obtain comprehensive matching results between video clips and text queries. It helps avoid detection error caused by the correlation between clips and specific words within the text. Eventually, in view of the inherent huge gap between different modalities, we eliminate the dreary concatenation operation and instead utilize cross-attention on video and audio tokens to narrow this gap. Extensive experiments verify the effectiveness of the proposed method, which outperforms state-of-the-art methods by 3.09% in R1@0.7 on QVHighlights dataset, and 5.01% in BK on TVSum dataset.</p>