Using cross-attention to achieve heterogeneous modal interaction is a common technique in multimodal emotion recognition. However, there are still some challenges: 1) high computational complexity of bidirectional attention hinders model scalability; 2) uneven distribution of sentiment information within and between modalities requires more flexible multi-semantic feature learning. In this paper, we proposed an efficient interaction Transformer with bottleneck mixture of experts. The bottleneck mixture of experts module, replacing costly cross-attention fusion, consists of specific- and shared-modality experts in parallel to capture distinct modality-specific and modality-consistent features respectively. In expert fusion, the designed soft top-k gating fusion strategy employs variable selection rules based on token types (partitioned according to attention scores) to emphasize critical segment interactions. Extensive experiments on two benchmark datasets reveal that our model outperforms the state-of-the-arts in terms of both performance and computational cost.