While massive web corpora form the foundation for training state-of-the-art Large Language Models (LLMs), the scarcity of high-quality, open-source Chinese datasets necessitates effective text quality assessment. Traditional heuristic filtering methods often struggle to capture nuanced semantic quality features, and employing large LLMs as evaluators entails prohibitively high computational costs. A viable alternative involves using LLMs to annotate small batches of data to train a parameter-efficient scorer model. However, existing research frequently overlooks critical aspects of prompt design during LLM annotation and the detailed architecture and training strategy for the scorer model, leading to increased experimental costs and suboptimal results. To address these gaps, this paper introduces structured prompt templates for the data annotation phase and proposes specific strategies for the architecture design and training of the scorer model. Experimental results validate the effectiveness of our approach, offering a practical and efficient solution for enhancing data quality in Chinese LLM training corpora.