Evaluating Baselines for Type Inference: Static Code Analysis Versus Large Language Model

Automatic type inference in dynamic languages (such as Python) is important for developing novel software engineering tools as well as for the quality assurance and testing. A successful solution of the task improves software development and maintenance processes. Modern approaches typically combine static code analyzers and machine learning methods, including large language models, pre-trained on large corpora of source code. However, the evaluation on these approaches is still under a question, because a machine learning model is usually applied after static analysis. Another issue is taking into account the whole source code in a repository or a project. In the paper, we address this gap: (i) we propose a dataset that contains Python repositories from three different categories and (ii) we compare an industry-level static type inference tool to a state-of-the-art machine learning model that is based on a pre-trained large language model for source code.