Defocus deblurring aims to reconstruct a high-quality image from its degraded counterpart, playing significant roles in many intelligent systems. However, current methods for processing defocused images have deficiencies in perceiving and enhancing the frequency information of images, resulting in the loss of image details. In view of this, we innovatively propose the Frequency-Aware Encoding Block (FAEB), aiming to strengthen the encoder’s ability to extract high-frequency information. In addition, areas with a high degree of blurring in the image often result in many artifacts during the processing. To solve this problem, we also propose the Lightweight Artifact Removal Module (LARM), which is applied in the decoding process to effectively handle the large number of artifacts that appear in the image. Experimental results show that our proposed Artifact Removal Frequency-Aware Network (ArFaNet) outperforms the state-of-the-art algorithms in terms of objective metrics on the popular Dual-Pixel Defocus Deblurring (DPDD) and Real Depth of Field (RealDOF) datasets.