As machine learning models are increasingly deployed in high-stake domains, explainable AI (XAI) has become essential for ensuring transparency, accountability, and trust. Counterfactual explanations are a widely used XAI approaches that provide intuitive insights by identifying minimal changes to input features that alter model predictions. However, generating counterfactuals that are simultaneously plausible, causally consistent, and actionable at the same time, remains a fundamental challenge. Existing methods for counterfactual generation either rely on the unrealistic assumption that input features are either independent or require explicitly specified causal relationships between features, both of which limit the practical applicability of these methods. In this work, we propose a novel approach that leverages large language models (LLMs) to generate counterfactuals that are plausible, causally consistent, and actionable, without requiring explicit causal knowledge. The central premise is that LLMs, by virtue of being trained on large and diverse data sources, implicitly acquire knowledge of causal relationships between real-world variables, enabling them to suggest counterfactual changes that respect causal dependencies. A key concern with using LLMs is hallucination, i.e. generation of outputs that appear coherent, but are factually incorrect or logically inconsistent. To mitigate hallucinations, we embed the LLM within an agentic framework that proposes candidate counterfactuals iteratively, evaluates them via prediction feedback from the model, and refines them through step-by-step self-critique. Empirical results on multiple datasets demonstrate the efficacy of the proposed method in generating counterfactuals that are more causally consistent, plausible, and actionable than existing baselines.