Sign language translation requires accurate modelling of video sequences’ spatial and temporal dependencies. Existing methods often ignore the fine-grained interactions between intra-frame spatial structure and inter-frame temporal dynamics, thus limiting their ability to capture semantically rich representations. In this work, we propose HPE-SLT, a novel Transformer-based framework that jointly learns CSLR and CSLT through a shared spatiotemporal encoder. Our method introduces a dual-branch positional embedding module that combines relative spatial encoding to capture local dependencies within frames and absolute temporal encoding to model global temporal structures. To address redundancy and enhance temporal modelling, we apply a Top-K temporal sampling strategy in the CSLT branch to select semantically important frames. We also design a multi-task joint optimization scheme to simultaneously learn CSLR and CSLT, enabling the model to align fine-grained gloss-level and sentence-level representations. Further experimental analysis demonstrates the proposed method’s superior robustness across diverse datasets.