Reinforcement learning (RL) is a normative computational framework to account for reward-based learning. Whereas classical RL algorithms are based on experienced outcomes, humans and animals may generalize learning to unexperienced events based on internal world models, so-called fictive learning. We propose a simple, brain-inspired fictive learning rule to augment model-based RL and use the rodent two-step task to examine whether fictive learning can better explain the observed behavior. The learning rule uses the same reward prediction error (RPE) to update both experienced and unexperienced states and actions, with scaling by the event correlation inferred from the internal model for fictive update. Through simulations, we show that this model better reproduces key behavioral traits observed in the two-step task. Model fitting validates its superior fit over existing alternatives. Furthermore, the model replicates striatal dopaminergic dynamics observed in the same task, suggesting the brain might operate fictive learning for reward-based learning. This framework bridges normative RL theory and biological learning, offering new insights into adaptive behavior.