Empirical Evidence of DMN Errors in the Wild - An SAP Signavio Case Study

While the Decision Model and Notation standard (DMN) is considered to be an increasingly popular standard, there is a broad consensus that human modelling errors can easily occur in the creation of DMN models. Yet, while this consensus is clear, there is only limited evidence of which error types exactly may occur in practice. In this work, we therefore present some empirical evidence on DMN errors in the wild. Specifically, we analyze the SAP-SAM dataset by SAP Signavio, containing over 500 000 real-world conceptual models. Our results show that modelling errors, such as missing rules, occur frequently in real-life settings (36.1% of all models contained some form of issue). Furthermore, we analyze the distribution of which error types have occurred (relative to an existing classification of DMN error types from a previous work). To the best of our knowledge, this is the largest DMN study conducted to date (N = 5 668 DMN models).