Deep reinforcement learning (DRL) has achieved great success in robot manipulation, but sample efficiency and generalization remain two major challenges. To this end, we propose a skill policy learning algorithm with multi-scale data reconstruction. In our work, each original trajectory segment is treated as a point-to-point motion with a fixed start and desired point. We first train an encoding-decoding model from offline data and implement data augmentation (DA) online. The decoder learns two different scale outputs, one is used to reconstruct the beginning and end segments of the trajectory with low error, and the other is used to reconstruct the middle segments of the trajectory with high error. High errors are used to increase the difference between the reconstructed data and the original data, enabling DA as well as increasing the diversity of trajectories. We then use the learned prior information to guide exploration and accelerate DRL training. Finally, we verify our method on complex robot operation tasks, the experiment results show that the proposed approach can improve the generalization of the trained model and accelerate downstream task training.