The large language model has demonstrated its ability to reason and interpret in text-to-text applications. Current Chain of Thought (CoT) research focuses on either explaining reasoning steps or improving prediction results. This paper proposes a transformer-based language model to generate explanations as the Chain of Thoughts, then fuse visual (image) modality. The proposed model allows a multimodal reasoning chain based on text and image information. In particular, we adopt an attention layer with a gated fusion mechanism to form a joint representation for image and text input. The experiment results show that using CoT as part of the input can increase answer prediction by 19% in our study. Therefore, in our modelling process, CoT prompting is not only a part of the output to generate reasoning explanations but is also used as the input of the decoder to predict answers. The proposed method outperforms GPT-3.5 (two-shot) by 14% and GPT-4 (CoT) by 4.71% with a much smaller parameter size pre-trained language model.