Counterfactual experience generation improves data utilization in off-policy reinforcement learning, yet existing methods often use simplified transition models and untargeted action sampling, limiting the realism and utility of virtual experiences. This paper introduces BEACON (Bayesian Uncertainty Estimation for Targeted Counterfactual Experience Generation), a novel framework integrating Bayesian uncertainty estimation to effectively guide counterfactual experience generation. BEACON employs a state-aware generative model to capture joint state-action dependencies in transitions, yielding more context-consistent virtual samples. Crucially, it leverages epistemic uncertainty, estimated via latent-space variability, to direct counterfactual action sampling towards under-explored, high-uncertainty regions, aiming to enhance augmented experience utility. Evaluations on diverse benchmarks suggest BEACON offers performance advantages over prior baselines, highlighting benefits of principled uncertainty modeling with counterfactual experience generation for robust reinforcement learning agents.