Lithography defect detection plays a vital role in chip production. A sensitive and reliable defect detection method is very important to ensure the quality of chips. Deep learning has shown promising results in learning generalizable image priors, making it a viable approach for the lithography defect detection task. However, it is still challenging to obtain accurate and stable lithography defect detection results, because of the existence of real noise in Scanning Electron Microscope (SEM) images and the diversity of lithography defects. In this work, we propose a new SEM image lithography defect detection pipeline, which comprises a Transformer-based high-resolution SEM image denoising module to mitigate the interference of real noise on defect detection and a new detection model for detecting defects in denoised SEM images. Particularly, in our detection model, we present a new lightweight feature extraction module for efficiently extracting multi-scale features and a new cross-scale feature fusion module for fully integrating features at different scales. Moreover, we create an advanced lithography defect dataset annotated by experienced experts. Experimental results demonstrate that our method can accurately detect lithography defects in SEM images and our detection model surpasses other state-of-the-art detectors on the lithography defect dataset.