The task of video classification based on the content in the video is of great importance and is the primary task for many AI-based applications. However, the task remains challenging due to the presence of multiple objects, variations in human object appearance, dynamic backgrounds, occlusions, diverse object movements, and different lighting conditions. Deep learning models offer robustness and generalization capacity in solving the classification of related problems, which motivates the use of a deep learning-based approach for video classification. This work combines Convolution Neural Network (CNN) and Long Short-Term Memory (LSTM) to effectively address the challenging task of learning both short- and long-term spatio-temporal relationships across the video sequences. The entire input video is divided into meaningful segments before feeding it to the CNN-LSTM model. The extracted features are applied to the hidden Markov model (HMM) as a classifier and compared with the results obtained from the same features with the fully connected layer (CNN-LSTM-FCL). Then, the two fusion techniques are applied, average-max and majority voting, to predict the video category using a series of class probabilities for all the segments in the given video. The proposed approach for video categories: sports, cartoons, and news, gives the highest classification accuracy of 98.11%.