Fabric defect detection techniques are significantly important to improve the quality of the fabric industry. However, most existing models face challenges related to overgeneralization due to relying only on the in-distribution, which represents normal samples for training. In this paper, we introduce an improved approach to address overgeneralization issues and augment the accuracy of fabric defect detection and localization. Our approach leverages a U-Net architecture with end-to-end modeling in the context of unsupervised deep learning-based defect detection. We propose the Gamma-Weighted Discrepancy loss function ( \({L}_{\text{GWD}}\) ) to tackle feature matching and defect detection more effectively. Moreover, we optimized the quality of the synthetic anomalies generated during training to reduce the gap between simulated anomalies and real-world defects. The innovative strategy focuses on enhancing the distinctiveness and dissimilarity of synthetic anomalies compared to normal patterns. Experimental results conducted on the AITEX fabric and Carpet-MVTecAD datasets demonstrate the effectiveness of these enhancements with high average AUC scores of 98.4% at the image level and 98.1% at the pixel level. The proposed model demonstrates superior performance compared to existing state-of-the-art deep learning methods in the detection and localization of fabric defects, significantly contributing to progress in this domain.