We are witnessing an increasing interest in vision-language models (VLMs) as reflected in the impressive results across a large spectrum of tasks. In this context, we introduce in this paper a novel architecture that exploits the capabilities of VLMs for facial expression recognition in driving environment to enhance road safety. We present an approach called CLIVP-FER, which uses the Contrastive Language-Image Pretraining (CLIP) and combines both visual and textual data to overcome the environmental challenges and ambiguities in facial expression interpretation. In addition, we apply average pooling to improve the accuracy and the computational efficiency. The proposed approach is thoroughly evaluated on a benchmark driving dataset called KMU-FED. The experiments showed superior performance compared to state-of-the-art methods, achieving an average accuracy of 97.36%. Cross-database evaluation is also provided showing good generalization abilities. The ablation study gives more insights into the performance of our proposed architecture. The obtained results are interesting and confirm the capabilities of vision-language models in vision tasks, demonstrating their promising applications in efficient driver assistance and intervention systems. We are making the code of this work publicly available for research purposes at https://github.com/Ibtissam-SAADI/CLIVP-FER .