In an attempt to understand how different roadways, driving situations, and other external factors contribute to stressful driving behavior, the current study develops a latent factor analysis model using questionnaire data collected from car drivers. Results of the exploratory factor analysis obtained from a 14-item questionnaire study revealed a 4-factored construct model, which was further evaluated by confirmatory factor analysis. Additionally, the study developed a second-order latent variable model to express drivers’ emotional states while driving under stressed conditions. A gradual decrease in factors’ scores was observed as the age of drivers increased. However, a positive relationship between self-reported accident frequency and stressful driving behavior was obtained, which justifies the impending risks and exposure to dangerous situations that would be associated with stressed drivers. Finally, the study revealed the most influencing driving situations that would contribute to stressful driving depending on the item-wise analysis of each factor.