Predicting emergency room revisits is a significant challenge due to data imbalance and the complexity of patient conditions. Traditional machine learning models have shown promise, but their lack of interpretability limits their practical use in clinical settings. In this paper, we present a novel framework for predicting emergency room revisits within 24 h. To address the challenge of model explainability, we incorporate a concept bottleneck layer. This layer allows the model to first predict a set of predefined, human-understandable concepts, enabling the integration of domain-specific knowledge from physicians into the prediction process. We consulted experienced emergency department physicians to identify the top 10 diseases most associated with revisits, such as unspecified abdominal pain, COVID-19 exposure, and chest pain. A key challenge in this approach is generating auxiliary labels for the concept layer, which we address by leveraging large language models (LLMs) to interpret patient electronic health records (EHRs) and generate patient-specific embeddings. These embeddings are then used to predict the likelihood of revisits based on their similarity to the top 10 diseases. Experimental results demonstrate the effectiveness of this approach, providing both high predictive performance and enhanced model interpretability.