The paper focuses on the deep learning based model selection for the emotion recognition of a learner. The emotion is recognized through the facial images captured by the proctor module in a Virtual Learning Environment (VLE). A facial emotion recognition (FER) model is developed on the FER2013 dataset using transfer learning techniques. The purpose of the FER model is to propose an emotional construct in addition to cognitive constructs for the multi-modal evaluation of the learners performance in a virtual learning task. The work presented in this paper includes two main parts: first, propose a mathematical model for cognitive evaluation and second, selection of a deep learning model for emotion recognition. The proposed work assess the multi-modal performance in terms of memory, visual processing, processing speed, attention score and emotion. The proposed system provides relevant information about learners’ emotional health and the cognitive elements that need further support, which helps learners’ improve cognitively and emotionally.