Accurate camera exposure control is essential in computer vision tasks such as robot navigation and object detection, where image quality significantly affects downstream performance. We propose an automatic exposure control framework based on deep reinforcement learning that dynamically adjusts exposure parameters in real time, maintaining high-quality imaging under rapidly changing lighting conditions. While reinforcement learning offers the advantage of learning without extensive manual labels, conventional handcrafted reward functions often fail to capture the nuanced relationship between exposure settings and perceived image quality. To address this, we introduce a pairwise human feedback reward model that leverages human comparative judgments to train a reward network more aligned with subjective visual perception. This learned reward model automatically scores candidate exposures during training, enabling precise and efficient policy optimization. Experiments conducted in real-world indoor and outdoor environments demonstrate that our method responds quickly to sudden lighting changes and achieves superior image quality across multiple evaluation metrics compared to existing learning-based baselines. These results confirm the practicality and effectiveness of our approach in enhancing visual perception under dynamic illumination.