<p>Accurate multi-object tracking is essential for intelligent surveillance systems. However, deploying trackers that rely on both appearance cues and motion information for object association remains challenging on resource-constrained platforms. This difficulty stems primarily from the heavy computational cost of the additional CNN-based appearance encoding model, such as MARS-128 network used by DeepSORT, which is responsible for generating visual embeddings used to compute the appearance similarity between objects. These models, while effective, significantly increase processing time and memory usage, making real-time tracking impractical on resource-constrained devices. To address this gap, this paper introduces a lightweight appearance association method that integrates seamlessly into the DeepSORT framework. The proposed approach replaces the traditional MARS-128 CNN model by directly exploiting the native multi-scale feature maps produced by YOLO. These internal representations naturally encode rich visual information originally designed for detection task. Therefore, our contribution lies in reusing these features as appearance descriptors within the tracking pipeline, avoiding the need for any encoding network. This is achieved by extracting object descriptors via a simple centre-cell lookup, which extracts effective appearance embeddings with a negligible computational cost. Furthermore, an adaptive fusion strategy uses size-aware weights to combine multi-scale similarities, enabling more robust association. Experiments conducted on the MOT20 dataset show that the proposed tracker achieves accuracy close to original DeepSORT, recovering around 95% of its accuracy, while reducing association time from 0.28 to 0.07 s per frame, resulting in a 4 × speed-up on CPU-only execution. These results confirm that leveraging native detector features could provide an efficient real-time tracking.</p>