Dual-Modality Latent Gloss Alignment for Gloss-Free Sign Language Translation

Sign Language Translation (SLT) remains a challenging task due to the inherent cross-modal complexity of processing visual-gestural language. In this paper, we propose GluNet, a novel gloss-free framework that learns latent gloss representations through fine-grained video-text alignment, eliminating the dependency on gloss annotations. Our method constructs a joint cross-modal embedding space by aggregating temporally consistent segments from sign language videos and semantically aligned subword token units from textual translations. These aligned representations form latent visual and textual glosses that inherently capture the structural semantics of both modalities. Extensive experiments on two public benchmarks demonstrate that GluNet achieves state-of-the-art performance, surpassing all gloss-free methods by a significant margin, including a +2.57% BLEU-4 improvement on the Phoenix-2014T dataset compared to the second-best gloss-free competitor. Notably, GluNet even outperforms the leading some gloss-based methods on key metrics such as BLEU-4 and ROUGE, establishing a new benchmark for SLT. The codes are available at here .