In this paper, we propose a novel approach for generating captions and LaTeX notations out of handwritten mathematical expression images (HMEIs). For the proposed approach, we have created an annotated dataset comprising of handwritten images of mathematical expressions along with corresponding textual descriptions and LaTeX notations. This dataset acts as the stepping stone for training and evaluation of our model. To accurately interpret HMEIs, we deployed a encoder–decoder-based model which is a fusion model utilizing convolutional neural network (CNN) and recurrent neural network (RNN). Pretrained CNN models viz. VGG16, VGG19, ResNet50, InceptionV3 are used for feature extraction from the images. Two different RNN models viz. long short-term memory (LSTM) and gated recurrent unit (GRU) have been used for handling the sequence of texts as captions. The RNN models capture the temporary dependencies and long-term dependencies in the text sequence. In the final step, we used a SoftMax layer to generate the output in either textual description or LaTeX notation of the given handwritten expression image. Our experimental results demonstrate the maximum accuracy of 68.9% for generating textual descriptions and 75.7% for generating LaTeX notations. This work emphasizes on the uses of Natural Language Processing and Computer Vision techniques that improves the interpretation of handwritten mathematical expressions (HME).