Machine unlearning is essential for large language models (LLMs) to guarantee data privacy, model flexibility, and adherence to ethical standards. It allows the elimination of certain knowledge, addressing privacy issues and alleviating biases or misinformation without necessitating complete retraining. Retrieval-based techniques enhance LLMs by integrating external knowledge during inference. It improves model accuracy, reduces hallucinations, and enables real-time access to updated information without retraining. Recently, retrieval-based techniques have also demonstrated their capability to achieve machine unlearning without the adjustment of model parameters. However, the application of these parameter-agnostic unlearning algorithms remains inadequately investigated. In this paper, we examined the performance of retrieval-based unlearning method on different LLMs. Specifically, we established different evaluation metrics to explore the effectiveness of unlearning, the cost of unlearning, etc. We also emphasized the influential aspects that impact unlearning efficacy across various unlearning tasks. Our study provides insight into the application of LLM unlearning approaches in real-world scenarios.