Generating detailed explanations that are easy to comprehend and interact with is a challenging problem for complex Reinforcement Learning (RL) agents. While various methods explain different aspects of the agents, it is difficult to aggregate and generate tailored insights for different users. Thus, we propose a comprehensive explainability approach that utilizes interactive natural language queries and generates different types of explanations. First, we introduce a new approach to generate meaningful counterfactual explanations using natural language queries. Further, we complement the natural language explanations with customized feature attributions for detailed insights. This helps in facilitating the interaction with explanations as well as tailoring the explanations for different purposes and levels of expertise. We demonstrate our proposal using an industrial telecommunication use case which shows its applicability and utility in a complex real-world scenario.