Speech emotion recognition (SER) is crucial for enabling machines to interpret human emotions through vocal cues, enhancing their interaction with users. This study introduces a deep learning-based SER system that incorporates various neural network architectures, such as artificial neural networks (ANN), convolutional neural networks (CNN), a hybrid CNN-LSTM model and a fine-tuned Wav2Vec transformer. The models have been trained on a combination of four speech datasets, RAVDESS, CREMA-D, TESS and SAVEE. A combination of datasets introduces bias, but also allows our models to generalize better to a diverse set of emotional expressions. A large set of physical features (Mel-Frequency Cepstral Coefficients (MFCCs), Delta MFCCs (∆-MFCC), Delta-Delta MFCCs (∆∆-MFCC), Mel-Spectrogram, Zero-Crossing Rate (ZCR), energy features, entropy of energy and spectral features) were extracted from the audio data to feed to the models. After thorough evaluation, it was found that the fine-tuned Wave2Vec transformer significantly outperformed the other models due to its ability to capture temporal patterns effectively. Various techniques such as data augmentation, dropout, L2 regularization and early stopping were applied to reduce overfitting. Speech signals were visualized using waveplots and spectrograms to analyze the difference between different emotions. The models were evaluated using metrics such as accuracy, precision, recall, F1 score and AUC-ROC, with the results indicating that the models show good reliability. Advanced transformer-based models and integration of speech with facial expressions could be applied to further enhance the work in the future such that it could be employed in real-world systems.