In self-supervised pre-training, learning consistent and hierarchical representations that capture relationships among anatomical semantics holds promise for enhancing the performance and interpretability of downstream tasks. However, the representations learned by existing methods are vulnerable to scale variations, which manifests as inconsistency on some scales and misjudgments of hierarchy. Therefore, we propose a scale-robust anatomical representation learning framework with self-supervision, which incorporates contrastive learning with our newly proposed pretext tasks: location-scale prediction(LSP) and decomposition prediction(DP). Our method addresses the vulnerability from three aspects: 1) It uses multi-scale patches as inputs to embrace diverse anatomical semantics in pre-training. 2) LSP promotes consistency at multi-scales by enhancing the model’s sensitivity to scale and resolving representation conflicts caused by multi-scale inputs. 3) DP eliminates hierarchy misjudgments by producing hierarchical representations for anatomies and their constituent parts that better balance the similarity and discriminability. Evaluations across six chest X-ray datasets demonstrate that the representations learned by our method are consistent and hierarchical at multi-scales and have great transferring ability to various downstream tasks. The code is publicly available at https://github.com/SurongChu/SRHRS .