Early diagnosis of Branch Atheromatous Disease is critical for reducing associated disability rates. Recent advancements underscore the potential of deep learning methods in developing automated diagnostic tools. However, the limited availability of clinical data poses a significant challenge to their practical application. Traditional deep learning models, which rely on large-scale datasets, perform poorly in few-shot scenarios, limiting their effectiveness in this context. To address these challenges, we propose a few-shot learning framework for Branch Atheromatous Disease diagnosis that leverages Lesion Localization prior knowledge. This approach incorporates domain expertise to guide data augmentation, effectively mitigating the issue of limited training data. Specifically, our framework is developed based on 251 BAD slices, 51 Non-BAD slices, and 400 lesion-free slices obtained from preprocessed clinical DWI images. Furthermore, we introduce a two-stage training strategy and an adapter module for parameter-efficient fine-tuning, enabling effective model optimization even with constrained data. Our method was evaluated on clinical cases from multiple medical centers, demonstrating superior diagnostic accuracy and robustness compared to various baseline models. These results highlight the potential of our approach to enhance the efficiency and accuracy of early BAD diagnosis and alleviate clinical diagnostic workload.