Neurological diseases encompass a diverse range of conditions such as neurodegenerative diseases and neurodevelopmental disorders. Developing a general model to assist in the diagnosis of multiple neurological diseases is essential in clinical practice, as it can help reduce misdiagnosis rates and alleviate the burden on physicians. However, most existing diagnostic models are designed for specific neurological disease scenarios and show poor performance when applied to multiple diseases. To this end, we present a semantic-assisted framework, called \(\mathbf {Neuro{\textbf {-}}AMS}\) , a \(\textbf{Neuro}\) -informed \(\textbf{A}\) ge-aware and \(\textbf{M}\) edical knowledge-integrated \(\textbf{S}\) trategy for diagnosis of multiple brain disorders. Specifically, we employ a vision encoder based on age-aware strategy to further enhance performance by leveraging the potential relationship between age and neurological diseases. Additionally, we extract semantic features from labels and integrate corresponding medical knowledge embeddings, constructing knowledge-level label features with enhanced semantics. These knowledge-level label features guide the vision encoder for capturing higher-level semantic representations through the alignment of image-text pairs. Our method is evaluated on four public brain disease datasets, and experimental results demonstrate that our method achieves consistent and statistically significant improvement compared with three public benchmarks and three specialized models.