Comparing the Performance of GPT-3 with BERT for Decision Requirements Modeling

Operational decisions such as loan or subsidy allocation are taken with high frequency and require a consistent decision quality which decision models can ensure. Decision models can be derived from textual descriptions describing both the decision logic and decision dependencies. Whilst decision models already help with modeling, implementing and automating decisions, the modelling step would still benefit from a (semi)-automated approach. The introduction of ChatGPT and GPT-3 offers opportunities to automatically discover decision dependencies from a given text. This paper evaluates the performance of two approaches that automatically extract decision dependencies from text, namely the best performing version of GPT-3 with a BERT-based approach. An evaluation with 36 experiments with a dataset of real-life cases and various levels of creativity allowed for GPT-3 concludes that theBERT BERT-based approach outperforms GPT-3 on the real-life dataset but that GPT-3 has promising results and requires further investigation.