Multi-modal Deep Learning for Detecting Toxicity in Transcribed-Audio Conversations

Toxicology can take many forms, ranging from overt approaches such as abusive language and bullying to more subtle means. Almost all corners of the Internet are affected by this practice, but gaming, news, blogging, and social media are particularly prevalent. Using a Convolutional Spiking Neural Network (CSNN) leveraging multi-modality, we explore a method for detecting toxicity. To enhance the capability of toxicity detection, the method utilizes both text and audio features from the DeToxy dataset. The CSNN has been composed of two modalities, one for text and one for audio, and a late fusion was applied to combine the final output. An embedding layer has been applied to textual data in the first step. Text tokens can be mapped into vector representations in order to extract features. In the audio modality, the convolution and max-pooling layers are two-dimensional; a flattening layer is applied prior to the linear layer. We fuse audio and text outputs using a fusion layer. Concatenating the spikes from the two modalities will construct the fusion layer.