Conceptual modelling aims to abstractly represent aspects of the perceivable world for a wide range of application areas so that the relevant part of an application can be represented and appropriate data included in a useful information system. Such abstraction efforts can be challenging and the resulting research papers have a wide variety of reported results. This makes it difficult to correctly identify and categorize the contribution of a specific research paper. Prior research has proposed a framework for characterizing the contributions of conceptual modelling research. In this paper, we assess the effectiveness of the framework’s use to understand whether and how it could be applied to the large field of conceptual modeling. An experiment was conducted with 24 junior researchers as subjects. The evaluation of the experiment consisted of analysing the level of agreement among the subjects in the classification of a research paper’s characteristics and analysing the correctness when classifying a paper. The data analysis showed a medium level of agreement and a good level of correctness. The outcome implies that framework helps identify the type of contribution made by a research paper, even when subjects are non-experts in conceptual modelling.