Multi-modal sarcasm detection using ensemble net model

Generally, sarcasm is expressed via various verbal and non-verbal words. Various existing works on the detection of sarcasm have been performed in either text or video. With the rapid growth of social media and internet technology, people express their emotions and feelings using text. Therefore, a multi-modal sarcasm detection task is crucial to understanding people’s real feelings and beliefs. However, it is still a challenge to detect sarcasm from multi-modal features. Therefore, this work presents a new hybrid ensemble deep learning approach for multi-modal sarcasm detection. The major goal of this research is to determine the different classes of sarcasm using a multi-modal dataset. Here, imaging modality-based sarcasm detection is performed using Deep Residual Net, and the visual features are extracted. For the generation of text modality, the text data are pre-processed with punctuation removal, and the textual features are extracted using Term Frequency-Inverse Average Document Frequency. The extracted features are used as input for the bidirectional long short-term memory model. The audio (acoustic) elements are extracted to form acoustic modality, which is subsequently sent to the visual geometry group. Furthermore, the weighted fusion modality process is used to combine all of the collected features. The softmax layer acts as the classification layer for performing multi-modal sarcasm detection. Here, the Tent chaotic snack optimization algorithm is employed to tune the hyperparameter and reduce the complexity of the proposed Hybrid EnsembleNet. PYTHON tool is used to evaluate the performance of the proposed classifier. The proposed hybrid EnsembleNet is trained using two datasets: Memotion 7k and MUStARD.