Shadows can lead to malfunctions in computer vision, making shadow removal an essential task for restoring underlying information. For a long time, researchers have proposed hand-crafted methods based on observing shadow formation models. Then, deep-learning-based solutions have further advanced performance in restoration quality. However, existing datasets have several limitations, such as lacking occluders, restricted camera angles, and inconsistency. In this paper, a novel benchmark called the Advanced Dataset for Shadow Processing (ADSP) is introduced. Through the synthesizing strategy, the ADSP becomes the first dataset containing outdoor images with occluders. Statistical analysis and experiments demonstrate that the ADSP has the advantages of lower domain shifting, matching real-world scenarios, and sufficient generalizing capability. Moreover, as a reference for the removal task, we also propose the Segmented Refinement Removal Network (SRRN), which includes three subnets for shadow removal, color adjustment, and boundary smoothing, respectively. It achieves state-of-the-art performance and can be set as a reference for shadow removal.