Are We Nearing Singularity? A Study of Language Capabilities of ChatGPT

There has been a huge uproar about ChatGPT in recent times. There are widespread fears that generative models such as ChatGPT will affect employability and bring humanity closer towards singularity. In this paper, we assessed the capabilities and limitations of ChatGPT empirically. We subjected ChatGPT to a series of questions with different levels of lexical and syntactic language complexity and manually scored the responses. We measured the lexical and syntactic complexity of the questions using text analytics techniques. The analysis of the responses showed ChatGPT as a highly capable language model, with certain limitations. ChatGPT showed an increased likelihood of generating an inaccurate response with increase in syntactic language complexity. Though there have been sporadic efforts and reports about ChatGPT’s limitations, this paper presents a pioneering structured approach towards assessing ChatGPT’s language capabilities. This study takes a small, yet important step in research towards assessing language models. It opens avenues for future research towards strengthening the generator model and developing the discriminator dataset.