Multimodal Sign Language Model
摘要
Sign language translation (SLT) plays a key role in improving accessibility for deaf communities, but faces major challenges such as limited parallel video-text data. In this work, we present Imitator, an architecture that learns to produce text embeddings that align with those of a pretrained large language model (LLM), such as LLaMA. Our approach combines lightweight 1D convolutions and a Transformer encoder with a cross-attention mechanism using learnable token queries to flexibly map variable-length video sequences into fixed-length text representations. The preliminary results show promising alignment metrics, suggesting the potential of this imitation-based strategy for SLT. Imitator model achieves a mean squared error with cosine similarity (MSE + CosSim) of \(8 \times 10^{-4}\) on the validation set. These results highlight the potential of imitation-based embedding alignment as a lightweight alternative for sign language translation in low-resource contexts.