Electrocardiogram (ECG) interpretation is essential for cardiac diagnosis, yet machine learning models often fail to adapt to real-world data, particularly when certain conditions, such as rare diseases, are underrepresented in training sets. In-Context Learning (ICL) has shown promise for adapting Large Language Models (LLMs) to new tasks using only a few labeled examples at inference time, but its potential in the domain of medical imaging remains largely unexplored. In this work, we investigate whether ICL can enable large Vision-Language Models (VLMs) to interpret ECG images. Our approach leverages VLMs to perform classification in data-constrained scenarios without parameter updates, offering a practical alternative when fine-tuning is not feasible. To enhance both interpretability and predictive performance, we incorporate a concept-based prompting framework inspired by Concept Bottleneck Models (CBMs). Specifically, the VLM is prompted to first predict clinically meaningful intermediate concepts, which then guide the final diagnosis, providing structured, interpretable reasoning. We evaluate our method on detecting Left Bundle Branch Block and Brugada syndrome. Results indicate that combining ICL with CBM-style prompting yields interpretable ECG analyses and promising diagnostic performance in data-limited settings. This suggests that VLMs could be a practical tool for extending machine learning to underrepresented clinical conditions without the need for extensive labeled data.