In contemporary times, monitoring criminal behavior and providing mental health therapy requires various activities related to an individual’s character. The development of an automated process for personality assessment through behavior analysis is still under research. Our contribution to this field involves combining deep learning with a regression-based machine learning model through a data pipeline for both audio and video data. We propose a comprehensive classification pipeline that utilizes supervised methods such as LSTM, GRU, and RNN, coupled with Visual Geometry Group (VGG) models for analyzing audio and visual data. The aim is to draw a comparative analysis between these methods to effectively analyze personalities. Our approach yields quality results, achieving an accuracy rate of 88.9% by combining VGG and NN for processing audio-visual data to analyze one’s personality composition.