Text image editing is a critical task in computer vision and graphics research, with diverse applications such as advertising design and social media content creation. Generating high-quality text images with readability and consistency using Diffusion Models (DFMs) poses a challenge due to their inadequate handling of text styles and glyphs. In this paper, we present DiffCTE (Consistent Visual Text Editing with High Style Fidelity via Diffusion Model), a novel approach that enhances the generation of readable and stylistically consistent text images by providing DFMs with improved style condition guidance. Leveraging a vision language model (VLM) consisting of a Vision Encoder, Style Abstractor, and Text Decoder, the Style Abstractor effectively extracts style condition and represents style information across languages, encompassing aspects such as glyph, font, and color. By integrating the style condition into DFMs, DiffCTE enables the generation of visually coherent and stylistically consistent text. Specifically, the generated text achieves style transfer by maintaining consistency with the reference style. Our comprehensive experiments demonstrate the superior performance of DiffCTE compared to existing methods, setting new directions for future research in text image editing with DFMs.