Trajectory clustering, a core component of Location-Based Services (LBS), partitions trajectory data into clusters based on similarity criteria. Despite recent advances using deep neural networks to enhance clustering performance, these models struggle with learning adequate representations from trajectory data, especially under conditions of uneven or low sampling rates. This limitation compromises the security and robustness of the models, resulting in incorrect predictions. In this work, we investigate the adversarial robustness of trajectory clustering models and propose a hard label-based black-box attack named Trajectory Granularity Adaptive Iteration (TGAI). TGAI generates real world adversarial trajectory examples by efficiently utilizing global and local features of the trajectory data, minimizing added perturbation. Our experimental results demonstrate the effectiveness of TGAI, achieving a 96.37% attack success rate on the E2DTC model with minimal perturbations, highlighting significant vulnerabilities in current trajectory clustering approaches.