In real-world scenarios, the changes of illumination are noticeable and can significantly impact the performance of re-identification (ReID) algorithms. However, the existing person ReID methods predominantly concentrate on tackling challenges in scenarios with minimal illumination fluctuations. The key to tackling this issue is to extract illumination-invariant features. So we propose a joint learning framework which combines the recovery of reflectance map and extraction of illumination-invariant feature. By sharing parameters, the proposed module, which can be seamlessly detached during the inference phase, contributes to a reduction in inference computation time. We also introduce an adversarial learning mechanism, utilizing illumination category and person identity to facilitate the extraction of features that are invariant to illumination. Besides, due to the lack of person ReID dataset containing images under diverse lighting conditions, we construct a real-world dataset called ICReID in which images have drastic illumination changes. Extensive experiments demonstrate the effectiveness of the proposed method, which achieves significant performance on both synthetic and real datasets.