In recent years, Siamese-based object tracking methods have gained significant attention in the UAV field. However, the computational limitations of UAV platforms and the specific challenges of aerial tracking make it difficult for existing methods to achieve both real-time performance and high accuracy. This paper proposes PEBTrack, a tracker designed to balance performance and efficiency for UAV scenarios. PEBTrack uses MobileNetV3, a lightweight network optimized for mobile devices, to reduce model parameters and increase inference speed. Additionally, two attention modules are introduced: the Channel Shift Attention Module and the Context Local Perception Attention Module. These modules enhance the robustness of the algorithm by extracting detailed visual information and improving the network’s ability to capture local features. Additionally, a network for feature fusion is utilized to combine various types of semantic information, enhancing the flow of information. The regression and classification operations are decoupled to refine the tracked object and improve tracking accuracy. Experimental findings illustrate the effectiveness of PEBTrack, which achieves a processing speed of 45 FPS using CPU inference. It outperforms existing algorithms on the UAV123 and DTB70 benchmarks, showing accuracy and robustness comparable to TCTrack, a state-of-the-art tracker.