The pattern generation method based on diffusion models has achieved success in various fields. However, the denoising process often neglects the extraction of image details and the establishment of long-range dependencies between high and low dimensions. Consequently, the generated ethnic patterns lack clear outlines and authenticity. To address these issues, this paper proposes an Adaptive Local Diffusion Network (ALD-Net) based on a U-Net, which integrates Multi-Path Convolutional Attention (MPCA) and Dual-Stream Feature Fusion (DSF) to effectively extract fine-grained features and improve generation quality. Moreover, the use of localized training reduces training time and computational resources while maintaining the model’s generalization ability. Experiments show that ALD-Net significantly improves PSNR, SSIM, and LPIPS scores on the DEID dataset for ethnic patterns.