Deepfake detection has to be prioritized for the purposes of fighting misinformation, identity fraud and manipulation. There are risks that AI deepfake technologies pose threats to privacy and trust as well as the health of information. This article points to the fact that there is a great necessity for detection tools by integrating facial features reacting to emotion and the voice features for potential deepfakes. Our methodology tracks the gap between tears of joy and sobs of sorrow – the visual and audio information that the content is changed. Embedding the FER library for emotion classification, and the sentiment classification using BERT model also helps to enhance the accuracy of detection as compared to the strategies focused on single model approaches with yielding 85.50%. No more than 40% is the agreement of sentiments expressed with images and sounds. However, below that level, our system has no doubt that the video is a suspicious deepfake. This multimodal approach therefore brings strength to the detection so that even the finest of changes have more chances of being detected. With rapid expansion of synthetic media and deepfake technologies, this study is of importance in the context of development of next generation detection systems, which are aimed at prevention of disinformation caused by AI and preservation of integrity of digital materials.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

A Multimodal Approach to Deepfake Detection with Audio Sentiment and Facial Emotion Recognition

  • Atharv Choughule,
  • Aarya Pawar,
  • Kiran Chinchawade,
  • Drushtant Patil,
  • S Poornima

摘要

Deepfake detection has to be prioritized for the purposes of fighting misinformation, identity fraud and manipulation. There are risks that AI deepfake technologies pose threats to privacy and trust as well as the health of information. This article points to the fact that there is a great necessity for detection tools by integrating facial features reacting to emotion and the voice features for potential deepfakes. Our methodology tracks the gap between tears of joy and sobs of sorrow – the visual and audio information that the content is changed. Embedding the FER library for emotion classification, and the sentiment classification using BERT model also helps to enhance the accuracy of detection as compared to the strategies focused on single model approaches with yielding 85.50%. No more than 40% is the agreement of sentiments expressed with images and sounds. However, below that level, our system has no doubt that the video is a suspicious deepfake. This multimodal approach therefore brings strength to the detection so that even the finest of changes have more chances of being detected. With rapid expansion of synthetic media and deepfake technologies, this study is of importance in the context of development of next generation detection systems, which are aimed at prevention of disinformation caused by AI and preservation of integrity of digital materials.