Accurate identification of lesions, including anatomical lesion localization, is critical for automated radiology report generation. However, this task is particularly challenging in whole-body PET/CT imaging due to large amount of diverse anatomical regions throughout the whole body. Existing studies mainly rely on anatomical detection or segmentation. These methods are generally limited to only a small subset of anatomical regions due to difficulty of manual segmentation and annotation for large set of anatomical regions in the training stage. To address this issue, we propose a hierarchical CLIP-based 3D model to precisely and efficiently identify 387 anatomical lesion locations within whole-body PET/CT scans. Our model is built on three strategies: (1) Hierarchical localization, based on which anatomical locations are identified from coarse to fine to improve localization accuracy, robustness, and scalability; (2) Semantic location augmentation, which incorporates anatomical knowledge of relative location to adjacent regions to encourage neighborhood preservation of text feature representations; and (3) Location ambiguity mitigation, which excludes penalties on the top \(K\) ambiguous localizations in a modified CLIP loss to alleviate the cases with lesions residing at the boundaries of multiple regions. Notably, this work is the first to achieve accurate, robust, and efficient whole-body anatomical lesion localization, with significant performance improvement compared to the SOTA methods on a large whole-body PET/CT dataset comprising 1748 subjects acquired from multiple scanner makers.