AI-Driven Multimodal Evaluation of Communication Skills in Virtual Interviews
摘要
The accurate and scalable evaluation of communication skills during virtual interviews poses a significant challenge in modern recruitment processes. This paper presents a multimodal Artificial Intelligence (AI) system that utilizes Deep Learning (DL) techniques to analyze both verbal and non-verbal behaviors. The system consists of three core modules: Emotional assessment through facial expression recognition, speech emotion recognition, and eye tracking. Each module operates independently and contributes a weighted score to a unified communication assessment metric. The proposed system is designed to evaluate key soft skills, such as emotional regulation, vocal delivery, and visual attentiveness—Factors that are often viewed as subjective in traditional interviews. A scoring engine aggregates the outputs from each module into a final communication score on a 100-point scale, providing interpretable and consistent feedback. The system’s effectiveness is demonstrated through evaluations using both benchmark datasets and real-time experiments. The emotional recognition module achieved an accuracy of 97.35% on the Labelled Faces in the Wild (LFW) dataset, while the tone recognition module secured an accuracy of 87.67% on the RAVDESS dataset. The eye tracking module, tested on a custom dataset called GazeEval-5, reached an overall accuracy of 92.73%. Additionally, ten live interview simulations were conducted to assess real-world performance, resulting in average communication scores ranging from 76.8 to 85.2. These results validate the robustness and practical applicability of the system in remote hiring environments.