Respiratory sound classification is an important way to detect respiratory-related diseases. However, the scarcity of respiratory sou-nd data is a challenge, and these sounds are also influenced by patient demographic characteristics and recording location. We effectively address the above challenges through a Cross-Model Knowledge Distillation (CMKD) and metadata fusion approach. We use Cross-Model Knowledge Distillation techniques to merge the Transformer model’s strengths in global modeling with the CNN model’s capabilities in local feature extraction and inductive bias. This approach can help the model quickly achieve excellent classification results with limited training data. Considering that we also need to incorporate the patient’s clinical information to assist diagnosis during consultation, We first generate embedding for patient metadata using one-hot encoding, and then fuse them with audio embeddings obtained through cross-model knowledge distillation. This comprehensive approach enables accurate and thorough analysis of respiratory sounds. Our method was validated on the public SPRSound dataset and demonstrated a significant improvement in the experiments, with a 3.84%, 4.84%, and 4.34% increase in specificity, sensitivity, and average score, respectively, compared to the current optimal approach.