Deep learning has achieved remarkable success in medical image analysis, yet its translation to clinical settings is often impeded by the opaque nature of these models. While interpretable models like Concept Bottleneck Models (CBMs) maintain good interpretability, they often require manual designed and annotated concepts, or rely heavily on pre-trained vision-language models. In the medical domain, however, the concepts are often too complicated to be described precisely by pure plain text, and the unified foundation models of high performance are still missing. To address these challenges, we propose a novel framework that extracts human-understandable concepts from pre-trained opaque models and then builds surrogate CBMs for interpretable diagnosis. We first employ sparse autoencoders to disentangle learned representations into a limited set of clinically relevant concepts, which are then transformed into plain text with the assistance of domain experts or large language models (LLMs). Utilizing concept activation vectors (CAVs), we can project these concepts into a shared representation space and apply submodular optimization to select the most informative concepts for model inference. The interpretable surrogate CBMs are finally constructed through sparsely decomposing the visual representation into concepts representations. We validate our framework on three medical diagnostic benchmarks: HAM10000, Harvard-FairVLMed, and MIMIC-CXR. The results indicate that our method achieves performance comparable to opaque models while significantly enhancing interpretability, outperforming previous CBMs. The code of this work is publicly available at https://github.com/med-air/CFN .