Nature-inspired optimization algorithms have gained significant attention due to their effectiveness in solving complex optimization problems. This paper introduces a novel algorithm inspired by the communal hunting behavior of wild dogs, which exhibit highly coordinated and efficient strategies to capture prey. The proposed algorithm, termed Wild Hounds Optimization Algorithm (WHOA), mimics the dynamic social structure, cooperative tactics, and adaptive strategies of wild dogs during a hunt. WHOA employs a heterarchy mechanism where a group of virtual agents, representing wild dogs, work collaboratively to explore and exploit the search space. The algorithm balances exploration and exploitation by dynamically adjusting the roles of individual agents based on their performance and the proximity to potential solutions, analogous to the wild dogs’ real-world strategies of encircling and isolating prey. Experimental results on benchmark optimization problems demonstrate that the proposed algorithm outperforms several state-of-the-art algorithms in terms of convergence speed, solution quality, and robustness. The adaptability and cooperative nature of WHOA make it a promising tool for tackling a wide range of optimization challenges especially characterized by complex landscapes.