Existing image restoration algorithms are typically designed for specific domains. It is extremely challenging to achieve enhancement for both low-light and underwater images through a single model. Additionally, due to the extremely limited number of annotated underwater images, the model’s generalization performance is poor. In this paper, we present an ambient illumination disentangled network based weakly-supervised image restoration (WSIR) approach, aiming to utilize incomplete labeled images to achieve the restoration of various low-quality images. On the one hand, we design an illumination disentanglement network (Idnet) to learn the mapping rules for Retinex theory, and establish a data-driven camera response function (DdCRF) for illumination adjustment. On the other hand, we design a Adaptive Pixel Retention Factor Network (APRFNet) for generating the parameter maps in DdCRF, that improves its robustness and flexibility in complex and changeable environments, promoting the authenticity and visual aesthetics of the reconstructed results. Extensive experiments on public datasets and self-collected images demonstrate that our proposed scheme outperforms state-of-the-art methods in both qualitative and quantitative metrics.