In the context of software supply chain security and IoT device firmware analysis, binary vulnerability detection faces dual challenges of detection efficiency and coverage due to scarce annotated binary data. Although the open-source ecosystem has accumulated vast amounts of source-level vulnerability data, direct migration to binary vulnerability detection inevitably encounters a semantic gap caused by cross-modal representation differences such as compiler optimizations and symbol stripping. To address data scarcity in binary vulnerability detection and bridge the semantic gap in cross-modal matching with source code, this paper proposes a hierarchical semantic fusion framework for binary-source alignment. Through heterogeneous modal semantic bridging and hierarchical attention mechanisms, our approach significantly enhances cross-modal matching precision and scalability between binary and source code, achieving 94.3% accuracy. Furthermore, we introduce a vulnerability detection task-driven transfer framework that maps source-level vulnerability patterns to binary code feature space via cross-modal alignment. Leveraging dimensional expansion within the model’s knowledge space enables exponential scaling of usable data for binary vulnerability detection, thereby transcending data scarcity constraints. We collected 400 CVEs from 8 real-world vulnerable projects, achieving 80.3% detection accuracy. This research establishes an effective technical pathway for expanding usable data resources in automated binary vulnerability detection.