Hidden Follower Detection (HFD) aims to identify hidden followers in surveillance videos. In recent years, research in this field has mainly relied on gaze and distance to distinguish hidden followers from normal pedestrians. However, these two features are not fully robust, making it difficult to reliably separate hidden followers from normal pedestrians in complex scenarios. To better recognize hidden following behavior, we propose BF-HFD a novel framework that integrates trajectory features, pre-trained large models, and a behavioral feature augmentation (BFA) module to enrich behavioral representation. Specifically, we employ GPT-2 to encode temporal dynamics from distance and gaze sequences, and leverage the Vision Transformer (ViT) to extract spatial information from trajectory images. Furthermore, we improve the model’s discriminative power by refining feature representations using learned behavioral feature anchors. Extensive experiments validate the effectiveness of the proposed approach and demonstrate superior performance over state-of-the-art methods.