Object tracking has made remarkable progress in recent years, particularly with the advancement of sequence-based learning paradigms. However, lightweight trackers still predominantly rely on conventional prediction head networks, frequently characterized by intricate architectures and sophisticated objective functions. In this work, we propose ESTrack, a novel family of efficient tracking models designed around a streamlined encoder-decoder architecture. The encoder extracts multi-scale hierarchical features, while the decoder generates bounding box sequences in an autoregressive manner. A key innovation of our work is the Expandable Multi-Scale Feature Fusion Module (EMSFF), which facilitates the integration of hierarchical stage features while bridging the encoder and decoder. Furthermore, we extend the decoder’s input to accommodate multiple features. This enhancement addresses the limitation of single feature inputs, enriching the diversity of information available to the model. The proposed ESTrack achieves competitive performance while maintaining exceptional efficiency. For instance, it is 2.44x faster than Ocean with comparable accuracy. Additionally, we demonstrate the capability of ESTrack to compete with larger models in specific scenarios, highlighting its practical value for resource-constrained environments.