Infrared small target detection is critical for applications such as military surveillance, target tracking, and disaster rescue. However, existing deep-learning-based approaches often struggle in complex backgrounds and low signal-to-noise ratio (SNR) conditions, leading to high false positive rates and missed detections. To address these challenges, we propose IRDiff, a novel framework that reformulates detection as a generative task using a conditional diffusion model. Specifically, IRDiff employs a dual-branch architecture, with a condition encoder extracting contextual features and a denoising branch refining segmentation masks. To better separate small targets from background interference, we introduce a Dynamic Frequency Attention (DFA) module, which selectively extracts target-relevant information from noisy intermediate segmentation maps generated in the reverse diffusion process. To avoid the side-effect from the substantial noise present in intermediate segmentation maps, we further design a Time-Aware Scale (TAS) module, which adaptively balances feature fusion using temporal embeddings, enabling more effective feature integration throughout different stages of the diffusion process. Extensive experiments on NUAA-SIRST and NUDT-SIRST demonstrate that IRDiff achieves state-of-the-art performance, significantly improving detection accuracy in cluttered backgrounds.