Despite large-scale image-based vision-language models having revolutionized visual understanding, their extension to video action recognition still struggles to distinguish labels sharing highly similar semantics. We identify two major causes: (i) pretraining on static image–text pairs inherently biases models to emphasize global visual representation, and subsequent temporal modeling on these fixed frame-level embeddings suffers from loss of fine-grained temporal semantics. (ii) Relying on a unified classification head forcing it to discriminate both coarse and fine-grained categories simultaneously. This overload challenges the classifier to achieve a balanced trade-off between global and subtle separability. To address these limitations, we propose the Semantics-Aware Multi-Expert Network (SAME-Net) for improved adaptation of CLIP to action recognition. (i) First, we perform semantic clustering of label text and assign Cluster-Specific Experts (CSEs) for each cluster, thereby enhancing fine-grained precision among similar actions. (ii) We then design a SemanticGate Router (SGR) that learns to map samples to their corresponding expert based on cluster semantic centers. (iii) We further integrate Temporal Semantic Attention (TSA) into the backbone and jointly fine-tune it, extracting cross-frame context during feature extraction rather than relying on post-hoc fusion. Experimental results demonstrate that the proposed SAME-Net alleviates confusion among semantically similar labels, achieving state-of-the-art performance in complex action classification scenarios.