Evaluating the Performance of LLMs on Technical Language Processing Tasks

In this paper we present the results of an evaluation study of the performance of LLMs on Technical Language Processing tasks. Humans are often confronted with tasks in which they have to gather information from disparate sources and require making sense of large bodies of text. These tasks can be significantly complex for humans and often require deep study including rereading portions of a text. Towards simplifying the task of gathering information we evaluated LLMs with chat interfaces for their ability to provide answers to standard questions that a human can be expected to answer based on their reading of a body of text. The body of text under study is Title 47 of the United States Code of Federal Regulations (CFR) which describes regulations for commercial telecommunications as governed by the Federal Communications Commission (FCC). This has been a body of text of interest because our larger research concerns the issue of making sense of information related to Wireless Spectrum Governance and usage in an automated manner to support Dynamic Spectrum Access. The information concerning this wireless spectrum domain is found in many disparate sources, with Title 47 of the CFR being just one of many. Using a range of LLMs and providing the required CFR text as context we were able to quantify the performance of those LLMs on the specific task of answering the questions below.