Maritime vessel type recognition holds significant application value in fields such as maritime safety monitoring, environmental protection, and maritime traffic management. However, due to the complexity and variability of the marine environment, traditional single-modal image recognition methods face numerous challenges in practical applications. This paper proposes a multi-modal fusion network based on tensor alignment and domain adaptation. By combining visible light and infrared image data, the method fully leverages the complementary nature of different modalities, thereby enhancing the model’s performance in complex maritime environments. Additionally, adversarial training and tensor alignment techniques are introduced to effectively reduce the disparity between source and target domain features, improving cross-domain adaptation capability. Experiments conducted on a self-collected maritime vessel dataset demonstrate that the proposed method outperforms existing multi-modal fusion methods across multiple evaluation metrics, including accuracy, precision, recall, and F1 score, validating the effectiveness of our approach for maritime vessel type recognition.