This study examines the role of AI in the financial sector, focusing on its applications, challenges, and emerging trends. A systematic literature review uses the Web of Science (WoS) database. Bibliometric analysis is employed with tools such as VOSviewer and CitNetExplorer to identify research trends, keyword co-occurrences, and author collaborations. Thematic analysis categorizes AI applications in finance and assesses dominant research clusters. The findings reveal that AI transforms financial services by automating processes, improving risk assessment, and enhancing decision-making. AI may be used for fraud detection, portfolio management, credit scoring, and regulatory compliance. However, AI adoption presents challenges, including security risks, algorithmic biases, and ethical concerns. This study provides valuable insights for academics, practitioners, and policymakers on effectively integrating AI into financial services. It highlights the need to address ethical and regulatory issues associated with AI-driven finance. By systematically analyzing AI applications in finance and identifying emerging trends, this research contributes to existing literature and offers insights into future research directions.