Off-policy reinforcement learning (RL) algorithms often suffer from unstable convergence and Q-value estimation errors. Techniques like prioritized experience replay address this by sampling valuable experiences more frequently. This work explores an alternative approach, controlling the experience pool distribution in Q-value-based off-policy RL by curating experiences in the Replay Buffer, selecting which experiences to replace (prune) with experiences generated by the current policy. Selection is performed according to a variety of pruning modes, replacing different types of experiences. For this purpose, a curation algorithm was developed, including fast-to-compute pruning metrics based on the distance to experience-prototypes and density-estimates. They are combined with K-means clustering for fine-grained curation. We provide empirical evidence that some pruning modes consistently improve learning stability and performance over a standard FIFO Replay Buffer. Our results indicate that the effect of pruning modes can vary for environments with different reward signals. Pruning neither many of the rarest nor many of the most abundant experiences generalizes best across tasks while mitigating Q-value overestimation in all tested environments and increasing performance in continuous control environments (MuJoCo). An approach is highlighted to achieve similar benefits in sparse-reward, image-based environments (Atari) through higher ‘pruning rates’ and using the agent’s feature encoder to structure the experience space. These findings open multiple pathways for future exploration of the experience pool curation technique and combining it with existing methods to improve the stability and performance of off-policy RL algorithms.