The application deployment scenario has changed due to cloud computing, which makes effective resource management. For performance optimization and cost reduction, auto-scaling, or the dynamic adjustment of resources in response to workload variations, is essential. Reinforcement learning, in particular, is a promising machine learning technique that offers answers to this problem. This paper thoroughly analyzes reinforcement learning-based auto-scaling in cloud computing. It introduces reinforcement learning, cloud computing, and auto-scaling, laying the groundwork for how these three concepts will come together. The paper examines reinforcement learning algorithm types such as deep reinforcement learning and extensively used policy gradient approaches for cloud auto-scaling and machine learning alternatives. The main issues of reinforcement learning-based auto-scaling systems with state representations, incentive schemes, and scalability are highlighted. Training reinforcement learning agents for efficient auto-scaling, the importance of simulation setup, and historical data are examined. The paper discusses technical and financial ramifications while showcasing successful reinforcement learning-based and ML-based auto-scaling case studies in web services, data analytics, and container orchestration. Future research areas include explainable AI integration and multi-agent reinforcement learning. This study addresses important technical issues such as scalability, incentive structures, and state representation in relation to RL-based auto-scaling frameworks in cloud computing. It looks at how RL agents are trained, focusing on the use of historical data and simulation setups. In this work, the methodologies of deep reinforcement learning and policy gradient are reviewed and their applications in data analytics, container orchestration, and web services auto-scaling are demonstrated. Important contributions include discussing the financial and technical ramifications, providing case examples, and making recommendations for future multi-agent RL and explainable AI research.