Using Multimodal AI for Cognitive Behavior Analysis: Case Study of Deception Detection
摘要
Deception detection has gained widespread importance due to its applicability in critical domains like national security, judiciary, interrogation, and courtroom trials. Distinguishing between deceit and honest/truthful behavior is a vital decision. Deception is a crucial act of humans with many complex, diversified physiological and cognitive aspects. The existing methods of deceit detection with a single modality approach need to provide a comprehensive analysis of the task. Detecting deception by integrating and combining diverse heterogeneous modalities builds a comprehensive and complete picture of the underlying task with a holistic understanding of cognitive and emotional states. In the experimental study carried out on the “Bag of Lies” multimodal deception detection dataset, the individual models achieved an accuracy rate of 60.9% for audio, 56.73% for EEG, 59.7% for video, and 64.63% for the gaze modality. In contrast, the multimodal fusion or integration of all four modalities improved task performance and accuracy by up to 80.39% compared to their unimodal counterparts. Surpassing the existing reported results on the dataset. The insights gained from the study will aid in building improved, advanced deception detection algorithms and systems in the future.