While DETR-like detectors have performed successfully in generic object detection tasks, their accuracy significantly declines in real-world scenarios where images are often corrupted by degradation. A natural solution is to enhance degraded images before detection, but the misaligned optimization objectives between image enhancement and object detection tasks often lead to suboptimal results. To address this, we present an end-to-end degradation-oriented object detection framework, named Semantic DETR, which enhances the performance of DETR through a collaborative approach that combines semantic prompt and feature refinement. First, we introduce a realistic degradation simulation pipeline to simulate real-world degradations. Besides, we develop a LoRA-based image enhancement backbone by integrating low-rank adaptation into pre-trained backbone, enabling robust feature extraction from degraded inputs. Furthermore, we propose a semantic prompt generation pipeline that extracts target-aware semantic prompt from degraded images, which is incorporated into the transformer encoder to guide the model to efficiently identify objects. Finally, we introduce a feature refinement-guided collaborative learning strategy, which combines multi-scale feature refinement module with multiple auxiliary detection heads to improve query assignment under degraded conditions. Extensive experiments on both generic and task-specific degraded datasets demonstrate the effectiveness and generalizability of Semantic DETR.