Manipulating 3D faces using text is an important technology in the entertainment industry. However, text-based manipulation of 3D faces remains a challenging area due to the scarcity of data pairs consisting of 3D faces and corresponding text. Additionally, inference for manipulating 3D faces using text prompts often requires several minutes due to the large model sizes or the optimization process to fit the text prompt. In this paper, we propose the ITFaceEdit model, a text and image-based 3D face manipulation model. ITFaceEdit constructs a framework trainable only with image and text data pairs, allowing it to learn a direct relationship between the text latent space and the 3D face latent space. By utilizing vectors from the learned text embeddings, we can manipulate 3D faces, employing face parsing for disentangled manipulation. Through this approach, we not only extend the reconstructed 3D face space using images with text-based manipulation but also configure an inference process without relatively heavy model structures and optimization steps, enabling 3D face manipulation in a few seconds. We demonstrate the superiority of our proposed method through comparisons with existing methods in various ways.