CSPRD: A Financial Policy Retrieval Dataset for Chinese Stock Market

Recently, Large language models (LLMs) have demonstrated formidable capabilities, yet challenges persist in real-world applications, particularly in aspects of hallucination, misinformation and outdated knowledge. Retrieval-Augmented Generation (RAG) addresses these challenges by pre-retrieving pertinent information from external knowledge bases prior to utilizing LLMs for answering queries. Although RAG has been empirically validated to enhance response accuracy and reduce error rates, the paucity of domain-specific datasets hampers the development of proficient retrievers, therefore becoming a bottleneck in deploying RAG pipelines within professional fields. In this work, we propose a novel task termed stock policy retrieval and introduce the Chinese Stock Policy Retrieval Dataset (CSPRD), comprising 700+ prospectus excerpts annotated by seasoned experts, correlated with relevant articles from a collection of more than 10,000 entries in our amassed Chinese policy corpus. Our experiments with lexical, embedding, and fine-tuned bi-encoder models not only attests to the efficacy of our proposed CSPRD but also indicates considerable potential for enhancement. To capitalize on high quality encodings, we proposed CSPR-MQA, a retrieval-oriented pre-training paradigm that amalgamates various supervised natural language processing (NLP) tasks into an unsupervised framework for masked question-answering (MQA). Our CSPR-MQA model, after pre-training on 61GB Chinese corpus and fine-tuning on CSPRD, achieves the best performance on the CSPRD development set, with metrics including 57.9% MRR@10, 29.1% NDCG@10 and 39.3% Recall@10.