Navigating NLU Challenges in Pair Programming Agents: A Study on Data Size, Gender, Language, and Domain Effects

Recent strides in Natural Language Understanding (NLU) for pair programming conversational agents underscore the importance of expanding datasets and constructing models applicable across genders, languages, and domains. The difficulty arises from the resource-intensive nature of gathering data through lab studies. Our study explores the potential use of vast amounts of pre-existing data for the training of conversational agents. We introduced software engineering-specific slot labels through an open-coding process by identifying key words and phrases. Our slot labels were integrated with a dataset of developer-developer (DD) and developer-agent (DA) utterances, annotated with software engineering-specific intent labels, from pair programming conversations. We employed the transformer-based language model, jointBERT, to explore the required training size and gender-related impacts on intent and slot accuracy. To gauge the model’s generalizability, we analyzed 5 pair programming video conversations sourced from YouTube. These conversations were fully labeled for software engineering-specific intent and slot, allowing us to investigate language and domain effects on the model’s performance. Our findings reveal that 5 conversations, without a strict gender balance, can be used to train a pair programming NLU. Our study paves the way for expanding datasets used in the training of conversational agents.