<p>Fabric defect detection is a crucial step in the textile manufacturing pipeline. For fabrics with monotonous patterns and simple backgrounds, existing algorithms can already meet industrial requirements in terms of detection accuracy and real-time performance. However, when it comes to diverse defect types with complex backgrounds, especially those with significant scale variations, current detection methods still fall short. To enhance fabric defect detection performance, this paper proposes an improved model based on RT-DETR, named PEA-MAN-DRFD-DETR (PMD-DETR). First, we design a novel PConv-Efficient Attention Block (PEA-Block) applied to the backbone network, which balances local and global feature space information through partial convolution (PConv) and cross-channel interactive learning. This not only reduces redundant computations within the model but also enhances the feature extraction capability for fabric defects in complex backgrounds. Second, we replace the feature fusion strategy in the Cross-scale Feature Fusion (CCFF) module with a Mixed Aggregation Network (MAN) to optimize multi-scale feature interaction. During feature fusion, we employ the deep robust feature downsampling (DRFD) module instead of traditional convolutional downsampling to better preserve fine-grained defect details in shallow features, thereby improving the representation capability of low-dimensional features. Experimental results show that compared to the original RT-DETR, PMD-DETR improves AP50 by 3.1% and AP50:95 by 1.8% on the Alibaba Cloud Tianchi Fabric Dataset, while reducing parameter count and computational cost by 5%, all while maintaining a high frame rate and meeting real-time performance requirements.</p>