Test-time adaptation (TTA) intends to adapt the model after deployment with only unlabeled test samples. Pseudo-labeling, one of the popular methods, is easily hampered by incorrect pseudo-labels in its effectiveness. To mitigate this problem, we propose a novel TTA method with angular distance-based model prediction called TAD. It consists of two key components, including spatial local clustering for pseudo-label refinement and weighted angular distance loss for stable adaptation. Specifically, we replace the original softmax output with the prediction based on the angular distance. Then we build a memory bank that keeps the features of the most recent reliable samples and generates pseudo-labels by aggregating knowledge from neighboring samples. Finally, we introduce a weighted angular distance loss to adapt the pre-trained model, which gives higher weight to reliable pseudo-labels and difficult samples. Extensive experiments show that our method outperforms the previous methods on both domain generalization and image corruption scenarios. Our method can be applied to different network architectures, making it a good choice for deployment.