Due to the unique shooting angles, UAV imagery typically encompasses a multitude of targets. Throughout the capture process, these targets are not static; they continuously alter their orientation, moving in and out of the camera’s view. This dynamic behavior can lead to frequent changes in target identity, resulting in tracking errors and poor generalization in existing UAV tracking models. To address these challenges, this paper introduces the EL-MOTR model, specifically designed for UAV tracking. This model incorporates the E-ELAN structure within the target detector to enhance network learning capabilities without disrupting the original gradient pathways. Additionally, a target trajectory association mechanism is implemented during the trajectory association phase to refine trajectory predictions by integrating detection and tracking data. The ADMM algorithm further optimizes the predicted trajectories. This algorithm partitions the complex problem of optimizing trajectories for numerous targets into simpler, concurrent single-target tracking sub-problems, swiftly deriving optimal solutions for comprehensive multi-target tracking across the image. Testing on the VisDrone-MOT UAV dataset has shown that the EL-MOTR model increases accuracy by approximately 3.3% and precision by 0.5% over the benchmark model MOTRv2, validating its efficacy for multi-target tracking tasks in urban UAV aerial imagery.