Vision language models (VLMs) have become effective tools for image recognition, primarily due to their self-supervised training on large datasets. Their performance can be enhanced further through test-time prompt tuning (TPT). However, TPT ’s singular focus on accuracy improvement often leads to a decline in confidence calibration, restricting its use in safety-critical applications. In this work, we make two contributions: (1) We posit that random or naive initialization of prompts leads to overfitting on a particular test sample, and is one of the reasons for miscalibration of VLMs after TPT. To mitigate the problem, we propose careful initialization of test time prompt using prior knowledge about the target label attributes from a large language model (LLM). (2) We propose a novel regularization technique to preserve prompt calibration during test-time prompt tuning (TPT). This method simultaneously minimizes intraclass distances while maximizing interclass distances between learned prompts. Our approach achieves significant calibration improvements across multiple CLIP architectures and 15 diverse datasets, demonstrating its effectiveness for TPT. We report an average expected calibration error (ECE) of 4.11 with our method, TCA, compared to 11.7 for vanilla TPT [29], 6.12 for C-TPT [55] (ICLR’24), 6.78 for DiffTPT [8] (CVPR’23), and 8.43 for PromptAlign [44] (NeurIPS’23). The code is publicly accessible at https://github.com/rhebbalaguppe/TCA_PromptWithoutPanic .