Non-exemplar online continual learning (NE-OCL) presents a significant challenge in scenarios involving learning from a non-stationary data stream without storing or revisiting past examples. While recent approaches have achieved remarkable efficiency, they suffer from three key limitations: (1) limited adaptability due to static representations, (2) poor capacity for modeling temporal dependencies in streaming data, and (3) lack of backward transfer, i.e., the ability to refine prior knowledge based on new experience. These limitations become especially pronounced in real-world settings where temporally coherent input streams are the norm rather than the exception. This paper proposes ReFACTNet, a novel architecture designed to overcome these limitations by unifying fast adaptation, temporal memory, and causal attention in an efficient, online continual learning framework. ReFACTNet introduces Sparse Fast Adapter Modules (SFAMs) for parameter-efficient, dynamic adaptation within a frozen backbone, allowing the model to learn new concepts without disrupting stability. We incorporate a compressed recurrent memory module that refines representations using causal cross-attention over recent latent embeddings, enabling temporal context modeling and backward transfer. A confidence-based routing mechanism over feature representations ensures the stability of learned knowledge while supporting plastic adaptation to new tasks. Empirical results on standard NE-OCL benchmarks demonstrate that ReFACTNet achieves competitive or superior performance compared to state-of-the-art methods.