CT-Net: an interpretable CNN-Transformer fusion network for fNIRS classification

Functional near-infrared spectroscopy (fNIRS), an optical neuroimaging technique, has been widely used in the field of brain activity recognition and brain-computer interface. Existing works have proposed deep learning-based algorithms for the fNIRS classification problem. In this paper, a novel approach based on convolutional neural network and Transformer, named CT-Net, is established to guide the deep modeling for the classification of mental arithmetic (MA) tasks. We explore the effect of data representations, and design a temporal-level combination of two raw chromophore signals to improve the data utilization and enrich the feature learning of the model. We evaluate our model on two open-access datasets and achieve the classification accuracy of 98.05% and 77.61%, respectively. Moreover, we explain our model by the gradient-weighted class activation mapping, which presents a high consistent between the contributing value of features learned by the model and the mapping of brain activity in the MA task. The results suggest the feasibility and interpretability of CT-Net for decoding MA tasks.