Advanced Traffic Safety Analysis: Leveraging Deep Learning and Large Language Models for Near-Crash Detection in Crowdsourced Videos

Traffic crashes are a significant global concern, requiring advanced detection and analysis systems to enhance road safety. Traditional data sources often fail to capture near-miss crashes, which are crucial for preventive measures. Detecting near crashes and taking a proactive approach to address potential black spots is more effective than waiting for crashes to occur and then fixing these spots. Near-crash detection allows for the identification of hazardous areas and behaviours before they result in actual crashes, enabling preventive measures that can significantly enhance road safety. This study addresses these limitations by integrating computer vision and large language models (LLMs) into a comprehensive real-time analysis framework. Initially, we employed a Convolutional Neural Network (CNN) to classify video frames into ‘crash’ and ‘non-crash’ classes using publicly available datasets. Subsequently, we used dashcam videos from Australia to feed into the trained CNN model. The model classifies incidents into three categories: crash, non-crash, and near-miss, based on classification probabilities. Crashes with probabilities between 0.4 and 0.5 are considered near-miss crashes, probabilities over 0.5 are considered crashes, and those below 0.4 are considered non-crashes. To enhance the depth of analysis, we utilized YOLOv8 for object detection and GPT-4 to generate detailed narratives for each classified incident. This methodology provides a novel approach to combining different computer vision technologies and LLMs, offering a robust framework for leveraging video data in traffic safety research and addressing gaps in traditional data sources. Our integrated model demonstrated robust performance, achieving high training accuracy for the CNN (84%) and effective real-time processing capabilities. This research contributes a solid framework for future traffic incident analysis studies and a curated dataset of crash and near-miss incidents, enhancing proactive traffic safety measures and real-time incident management.