The current study deals with the automatic analysis of verbal irony using artificial neural networks. Detection of verbal irony is an important task nowadays, because the effectiveness of the communication depends on the correct interpretation of sentences with an ambiguous meaning. In the case, when the context is lacking, the correct sense can be understood not from the lexical content, but through phonetic features, as well as through co-speech mimics and gestures. Thus we accomplished a new research on the material of the multimedia corpus of Russian ironic speech, which contains the detailed phonetic annotation and irony evaluation by native listeners in perceptual auditory experiments. Two types of automated analysis were accomplished: based on acoustic feature and facial expression extraction. The use of the fully connected neural network and of the Wav2Vec 2.0 model for the automatic irony detection in audio signal demonstrated high level of irony recognition. We also tested on a part of the corpus the recognition of ironic facial expressions in video signal using convolutional neural network and the PyFeat library, which allowed us to conclude that this model can give good results when we increase the amount of the material.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Automatic Detection of Irony Based on Acoustic Features and Facial Expressions

  • Uliana Kochetkova,
  • Pavel Skrelin,
  • Vera Evdokimova,
  • Nikolai Borisov,
  • Pavel Scherbakov,
  • Petr Fedkin,
  • Rada German

摘要

The current study deals with the automatic analysis of verbal irony using artificial neural networks. Detection of verbal irony is an important task nowadays, because the effectiveness of the communication depends on the correct interpretation of sentences with an ambiguous meaning. In the case, when the context is lacking, the correct sense can be understood not from the lexical content, but through phonetic features, as well as through co-speech mimics and gestures. Thus we accomplished a new research on the material of the multimedia corpus of Russian ironic speech, which contains the detailed phonetic annotation and irony evaluation by native listeners in perceptual auditory experiments. Two types of automated analysis were accomplished: based on acoustic feature and facial expression extraction. The use of the fully connected neural network and of the Wav2Vec 2.0 model for the automatic irony detection in audio signal demonstrated high level of irony recognition. We also tested on a part of the corpus the recognition of ironic facial expressions in video signal using convolutional neural network and the PyFeat library, which allowed us to conclude that this model can give good results when we increase the amount of the material.