Helping people to write well is a fundamental area of Natural Language Processing (NLP) research. The purpose of writing assistance is to improve the accuracy and quality of the input text, with character checking being one of the essential abilities to detect and correct erroneous characters. The erroneous characters appear in humans’ handwriting include faked characters (nonexistent characters due to handwriting errors) and misspelled characters (incorrect use of real characters due to spelling errors). In this paper, we introduce YoloGPT, a novel model that combines YOLO and GPT for detecting and correcting erroneous characters within visual form human written texts in real-world scenarios, including faked and misspelled Chinese characters. YoloGPT begins by using a YOLO object detection model customized to recognize character sets to form sentences with Chinese characters and label the faked ones. Then, it employs the powerful large language model GPT equipped with an automatic prompt optimization method specially designed to find an optimized prompt. The LLMs use the explicit guidance of the prompt to correct the erroneous characters and generate the final answer. The YoloGPT model has achieved remarkable results, securing first place in Task 1 of NLPCC2024, demonstrating the effectiveness of our method.