Recent advances in AI have made pair programming conversational agents feasible. However, ML algorithms underpinning these agents require developers’ conversational data. No benchmark dataset currently exists for pair programming conversations, thus requiring the collection of data from diverse sources of pair programming session recordings. However, there remains ambiguity regarding the optimal approaches for utilizing data from these sessions to create conversational pair programming agents. Hence, we collected data from three major sources: (1) five YouTube videos (353 mins; 4,822 utterances) of 10 developers pair programming in different programming languages and domains, and (2) 23 lab studies (1,280 mins; 8,320 utterances) of 18 developer-developer and (3) 14 developer-agent pairs tasked to implement a two player Tic-Tac-Toe game in Java. We found that YouTube developers’ pair dynamics depended upon their prior relationship, as most pairs planned the conversations in advance and focused on increasing audience engagement. Further, we investigated creativity strategies and dialogue styles employed by pairs in YouTube videos (vlogs) vs. lab studies (labs). For creative strategies, YouTube developers clarified more and collected data from diverse sources, while developer-developer lab studies generated more ideas. For dialogue styles, we found that YouTube developers asked leading questions and were more positive, developer-developer participants were neutral and directed their partners, and developer-agent participants made explicit role changes and commands to run code. Our results demonstrate that vlog data can be cautiously used for training conversational agents, with implications for their application in programming tasks.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Pair Programming in the Lab Vs. Wild: A Qualitative Analysis of Creativity Strategies and Dialogue Styles for Agent Training Data

  • Sandeep Kaur Kuttal,
  • Jacob Hart,
  • Marcus Ensley,
  • Shandler A. Mason

摘要

Recent advances in AI have made pair programming conversational agents feasible. However, ML algorithms underpinning these agents require developers’ conversational data. No benchmark dataset currently exists for pair programming conversations, thus requiring the collection of data from diverse sources of pair programming session recordings. However, there remains ambiguity regarding the optimal approaches for utilizing data from these sessions to create conversational pair programming agents. Hence, we collected data from three major sources: (1) five YouTube videos (353 mins; 4,822 utterances) of 10 developers pair programming in different programming languages and domains, and (2) 23 lab studies (1,280 mins; 8,320 utterances) of 18 developer-developer and (3) 14 developer-agent pairs tasked to implement a two player Tic-Tac-Toe game in Java. We found that YouTube developers’ pair dynamics depended upon their prior relationship, as most pairs planned the conversations in advance and focused on increasing audience engagement. Further, we investigated creativity strategies and dialogue styles employed by pairs in YouTube videos (vlogs) vs. lab studies (labs). For creative strategies, YouTube developers clarified more and collected data from diverse sources, while developer-developer lab studies generated more ideas. For dialogue styles, we found that YouTube developers asked leading questions and were more positive, developer-developer participants were neutral and directed their partners, and developer-agent participants made explicit role changes and commands to run code. Our results demonstrate that vlog data can be cautiously used for training conversational agents, with implications for their application in programming tasks.