System architecture design is a crucial skill for software engineering students, yet it presents challenges due to its complexity and the need for technical experience. Therefore, helpful feedback for students is essential but difficult to provide in large university-level courses. Recent developments related to large language models (LLMs) show promising possibilities for supporting students in different educational scenarios. In this study, we developed an LLM-based mentor to aid students in designing system architectures based on textual project descriptions and technical requirements. The tool follows a Socratic method to guide students toward a possible solution by posing relevant questions and thoughts. Therefore, we developed the System Architecture Dialogue (SAD) Framework with input from 18 experienced software engineers who offered insights into their architectural development experiences. We conducted a study in a first-year project management course at a European university involving 39 students, comparing a group using the LLM-based approach with a traditional method group. We investigated the educational impact of LLMs through surveys, chat protocol analysis, and comparing final system architectures. This paper presents our findings and experiences with developing an LLM-based mentor for creating system architectures and its educational possibilities. While the study found no significant differences in the final architectures between the groups, it showed that LLMs can help reduce complexity and provide additional context by following a structured and logical approach. LLMs have the potential to broaden students’ horizons and can be a beneficial tool for early career students and software engineers.