Monocular 3D object detection presents significant challenges due to the inherent absence of depth and geometric information, rendering it more complex than 2D detection. This paper introduces MonoTCM, a Semantic-Depth Fusion Transformer that leverages a Token Clustering and Merging (TCM) module to enhance the efficiency and accuracy of monocular 3D object detection. The TCM module aggregates multi-scale grid-based tokens into clustering-based tokens, dynamically adjusting their shapes and sizes based on local density and distance metrics. This allows for finer granularity in critical areas while consolidating less informative regions. The aggregated tokens are subsequently decomposed into semantic and depth features, processed through dedicated transformer-based encoders, and integrated using a semantic-depth fusion decoder modeled after DETR. This approach enhances the model’s ability to capture implicit global geometric information and provides a cost-effective solution for real-time intelligent driving applications. Experimental results demonstrate the superiority of MonoTCM in enhancing detection performance compared to other advanced methods, highlighting its potential to advance the field of monocular 3D object detection.