<p>A redundant structure is often utilized to enhance the reliability of the long-run system, whereas the establishment of an appropriate maintenance decision framework will be conducive to achieving the long-term interests of the system. Thus, this paper develops a full-new maintenance decision framework for the multi-unit redundant system by integrating a customized deep reinforcement learning approach, and the deterioration process of each unit is delineated through a Wiener process with a drift model. Meanwhile, an agent is constructed to facilitate the maintenance decision process. Specifically, there are several important improvements: (i) An integrated dynamic maintenance decision framework is established for the multi-unit heterogeneous k-out-of-n: G systems by efficiently integrating multiple modules: the parameters estimation module, the environment model, and the DRL agent. (ii) A weighted double Q network estimator approach is employed to enhance the training performance, where the network estimator is designed as the weighted combination result of the online and target networks. (iii) An averaged approach of the target Q value is employed during the training process of the maintenance decision agent, which reutilizes the trained network parameters to achieve the average of the target Q value, thereby enhancing the training stability. In addition, a prioritized experience replay mechanism is incorporated into the integrated maintenance decision framework, which can further facilitate the training process by prioritizing learning some more valuable samples. Numerical experiments and comparison studies with practical deterioration datasets are conducted to verify the effectiveness and the results indicate a good performance for the presented approach.</p>