Efficient query performance is critical for modern web information systems, where dynamic and unpredictable workloads demand adaptive indexing strategies. Traditional heuristic-based and static tuning approaches often fail to respond effectively to evolving query patterns, leading to suboptimal database performance. To address these challenges, we propose a Dueling NoisyNet framework that leverages deep Q-networks for adaptive online index selection. Our approach integrates a dueling architecture to separately estimate state values and action advantages, enhancing learning stability, while NoisyNet-based exploration enables adaptive, state-dependent indexing decisions. By dynamically adjusting indexing strategies in real time, our method significantly improves query execution efficiency without requiring explicit workload predictions. Extensive experiments on benchmark datasets demonstrate that our model achieves faster convergence, superior adaptability, and reduced query execution times compared to state-of-the-art indexing solutions.