Medical vision-language models (MVLMs) have shown significant promise in medical diagnosis. However, most existing approaches are constrained by limited modality coverage and often rely on datasets lacking real-world radiological diversity and semantic precision. Moreover, current models exhibit insufficient evaluation on fine-grained tasks. In this paper, we introduce RadiSim, a large-scale Radiology dataset comprising 10.6 million image-text pairs from CT, MRI, and DR modalities, designed to Simulate the learning resources during radiologist training. Building on RadiSim, we propose RadiSimCLIP, a radiology-specific MVLM pretrained using a CLIP-style contrastive framework. The model progressively acquires multilevel capabilities—ranging from anatomical recognition and pixel-level interpretability to vision-language alignment—mirroring how radiologists gain expertise. To systematically evaluate generalizability, we further propose a three-part evaluation framework: (1) zero-shot classification, which includes a novel extension to 3D volumetric data, evaluating anatomical, modality, and disease recognition; (2) zero-shot organ detection, which evaluates pixel-level semantic localization without parameter tuning; and (3) cross-modal retrieval, measuring alignment between visual features and clinical text. RadiSimCLIP achieves state-of-the-art results on 15 of 16 downstream benchmarks. This work underscores the feasibility and potential of large-scale radiology-specific MVLMs for robust, zero-shot 3D medical understanding, representing a practical step toward clinically applicable foundation models. The GitHub repository is https://github.com/so-ux/RadiSimCLIP .