<p>Compressed video action recognition has gained increasing attention due to its advantages in reducing storage and computation compared to raw video-based approaches. Compressed videos inherently contain three modalities: I-frames provide detailed spatial appearance but are temporally sparse, whereas motion vectors and residuals offer motion cues with lower fidelity and higher noise. In this work, we propose CompViT, a computationally asymmetric two-stream Transformer framework that efficiently leverages the complementary properties of these modalities. CompViT introduces three key innovations. First, we design an asymmetric architecture in which a deep Transformer extracts spatial appearance features from I-frames, while a lightweight network models temporal dynamics from motion vectors and residuals. Second, we propose a position-aligned motion fusion strategy that separately encodes motion vectors and residuals and integrates them through position-wise addition, producing compact yet comprehensive motion representations. Third, we introduce a multi-stage feature fusion mechanism that partitions both streams into aligned stages and establishes cross-stream connections at each stage to enable progressive cross-modal interaction. Ultimately, motion and appearance features are fused into overall video representations. This design ensures modality-specific processing while systematically integrating complementary information across modalities. Extensive experiments on UCF-101, HMDB-51, and Kinetics-400 demonstrate that CompViT achieves state-of-the-art performance among compressed video methods, with significantly higher computational efficiency and competitive accuracy compared to raw video-based approaches.</p>