The rapid advancement of large language models (LLMs) is revolutionizing industries, with finance emerging as one of the most promising beneficiaries. Financial LLMs (FinLLMs), developed on their foundations, can leverage advanced natural language processing techniques to process and generate insights from vast volumes of unstructured financial data. Particularly in specialized areas like quantitative investing, FinLLMs are poised to redefine the landscape by emulating the decision-making process of top traders. They can more efficiently capture market expectations, evaluate the impacts of market events, and aid investors across a wide range of investment practices. However, systematic research in the field of FinLLMs remains in its early stages. This paper provides a comprehensive overview of FinLLMs, aiming to encourage broader exploration of their mature applications in finance. The main content is summarized as follows: Firstly, we present a chronological overview tracing the evolution from general-domain pre-trained language models (PLMs) to specialized FinLLMs, highlighting critical advancements such as FinBERT, BloombergGPT, and FinMA. Secondly, we compare major FinLLMs by examining their training methods, datasets, and corresponding fine-tuning strategies. Thirdly, we summarize the characteristics and performance evaluations of seven benchmark financial NLP tasks. In addition, we explore the practical applications of FinLLMs in traditional finance and behavioral finance. Finally, we discuss the challenges and opportunities in the adoption of FinLLMs, including issues like data privacy and ethical considerations, while proposing directions for future research.