Detecting traffic accidents in driving videos captured by on-board cameras is a critical task in vision-driven intelligent driving systems. The core challenge is to accurately capture the appearance information of each frame as well as the motion patterns between frames in the complex and dynamic background of driving videos. In this study, we propose a traffic accident detection framework based on two-stream memory-augmented networks, termed TMA-TAD, to enhance the ability of the driving system to detect traffic accidents. The two-stream structure of TMA-TAD consists of two separate detection streams: the appearance stream of RGB frames and the motion stream of optical flow frames, which capture the appearance and motion cues of traffic accidents respectively, and realize frame-level traffic accident detection through late fusion. In order to effectively leverage the appearance and motion information, we implement different Video Transformer-based memory-augmented networks for the different focuses of RGB and Flow streams. To validate the performance of TMA-TAD, we conducted exhaustive experiments on a real-world Detection of Traffic Anomaly (DoTA) dataset and compared its performance with that of ten state-of-the-art methods. The experimental results indicate that TMA-TAD demonstrates more advanced performance.