Partially observable environments pose a fundamental challenge for reinforcement learning (RL), requiring agents to infer hidden states from incomplete sensory input. We propose incorporating Temporal Predictive Coding (TPC) as a world model within RL agents to address this problem. By continuously predicting future observations, TPC builds robust latent representations that capture essential state information and temporal dependencies. We evaluate this approach in grid-world environments with varying levels of perceptual ambiguity. Across multiple tasks, TPC-augmented agents consistently outperform or match strong baselines, including LSTM, RWKV, Clone-Structured Cognitive Graphs (CSCG), and episodic control agents. Analysis of the learned representations shows that TPC effectively disentangles underlying state structure, resolving perceptual aliasing and supporting generalization across time. These results demonstrate that TPC enables the formation of stable, predictive internal states, improving both sample efficiency and decision-making under uncertainty. Our findings establish predictive coding as a promising framework for model-based RL in partially observable settings.