Shadow removal has long been a challenging task in the field of computer vision. Despite the significant progress since the advent of deep learning, existing approaches still fall short of robustness in generating high-quality shadow-free images due to the scarcity of training data and the complexity of shadow images. In this paper, we approach the task as an image-denoising process and present a deep neural network based on the ControlNet-driven stable diffusion model, whose rich prior knowledge compensates for the data shortage and better facilitates the task modeling of shadow removal. Moreover, we simulate multi-exposure shadow images as conditional inputs to regularize the denoising process during training. An intensity modulation block is also integrated to boost the intensity of the recovered scene. Experiments on two benchmark datasets, ISTD+ and SRD, witness the superior performance of the proposed approach.