Vision Transformer (ViT) achieves advanced results of facial expression recognition (FER), and consistently surpasses the convolutional neural network based methods. However, most ViT based methods calculate self-attention among all patches within entire facial image, resulting in expensive computational cost and non-task-specificity. Moreover, existing ViTs are implemented at single-scale and single-dimension, which have some intrinsic defects in multi-scale and cross-dimension feature extraction. To address these issues, this paper proposes a sparse pruning ViT based  FER (SPViT-FER), which contains two core components, including Landmarks Guided Token Pruning (LGTP) block and Inter-Channel Cross-scale Self-Attention (ICCSA) block. Specifically, the LGTP block uses facial landmarks as guidance to activate the informative patches and discard the uninformative ones, which can save computational cost dramatically without performance reduction. The ICCSA block is carried out through inter-channel self-attention and Query Cycle Shift Operation (QCSO), which can capture the multi-scale long-distance dependencies among channels. The experimental results on several benchmarks demonstrate the superiority of SPViT-FER.