Emotion recognition is contributing significantly to improving Human-computer interaction and systems’ capabilities of interpreting the emotional state of the users. This paper introduces a multimodal deep learning architecture to simultaneously consider the influence of facial expressions and the prosody of speech signals for basic emotion classification using six emotion categories: Angry, Disgust, Fear, Happy, Sad, and Surprise. We use Convolutional Neural Networks (CNN) for facial feature analysis for CK+ dataset, and employ Bidirectional Long Short-Term Memory (BiLSTM) networks for the temporal sequence processing for speech based RAVDESS dataset. In order to alleviate the data imbalance and enhance the generalization ability of the model, multiple augmentation methods are adopted in two datasets. The features extracted from both modalities are then concatenated and fed into a Multi-Layer Perceptron (MLP) for classification. Results report on the proposed multimodal model that outperforms unimodal models with that of 98.09% accuracy, in contrast to 94.36% for the facial emotion recognition and 87.84% for the speech emotion recognition model. These results verify the efficiency of multimodal fusion to boost emotion recognition performance under different conditions.