Detecting mistakes in a domain model: a comparison of three approaches

Domain models are a fundamental part of software engineering, thus it is important for every software engineer to know the principles of domain modeling. To teach students these essential modeling principles, instructors play a vital role. Instructors check models created by students for mistakes by comparing them with a correct solution, keeping in mind the possible variations. While this did not use to be an overwhelming task, this is not the case anymore nowadays due to a rapid increase in the number of students wanting to become software engineers, leading to larger class sizes. Hence, students may need to wait for a longer time to get feedback on their solutions and the feedback may be more superficial due to time constraints. In this paper, we evaluate three approaches for a mistake detection system (MDS) that aim to automate the manual approach of checking student solutions and help save both students’ and instructors’ time: (i) the basic approach, (ii) the basic plus synonyms approach, and (iii) the basic plus synonyms plus variations approach. In all cases, MDS automatically indicates the exact location and the type of the mistake to the student. At present, MDS accurately detects 83 out of 97 identified different types of mistakes that may exist in a student solution. A prototype tool verifies the feasibility of the proposed MDS. When synonyms and variations (i.e., multiple correct instructor solutions) are considered by MDS, recall of 0.97 and precision of 0.84 are achieved based on the results for real student solutions. This is an improvement of 0.20 both in terms of recall and precision over the basic approach. The proposed MDS takes us one step closer to automating the existing manual approach, freeing up instructor time and helping students learn domain modeling more effectively.