Traditional methods for creating robust machine learning models typically require extensive annotation, leading to a growing interest in leveraging semi-supervised learning for vision tasks. We hereby present a novel video action detection architecture capable of achieving high accuracy with minimal label requirements. Initially, the Video Swin Transformer employs local self-attention to extract meaningful video features. These features are then fed into a decoder pipeline inspired by UNet3+, effectively enhancing feature aggregation. Furthermore, skip connections between the encoder and decoder are reinforced by a transformer block. We evaluate our approach on two benchmarks, UCF101-24 and JHMDB-21, under annotation levels of 20% and 30%, respectively. Experimental results show that our method not only outperforms recent weakly supervised and semi-supervised approaches but also matches or surpasses certain fully supervised methods in terms of both f-mAP@0.5 and v-mAP@0.5.