We introduce InfoDesignLM, a model that enables interactive infographic layout design using only textual descriptions, mimicking the creative process of a human designer. In contrast to traditional methods that depend on multimodal vision models, our approach leverages large language models (LLMs) to complete both layout planning and generation tasks through text alone. To facilitate this, we developed a semantically-driven dataset, InfoLap, which helps the LLM establish an accurate mapping between text semantics and layout design. Despite eliminating the need for vision encoders, our method outperforms state-of-the-art multimodal models like GPT-4o in a series of experiments. This advancement not only highlights the potential of LLMs in cross-modal understanding and the generation of 2D layouts but also paves the way for new automated design applications in the Visual Rich Document (VRD) domain.