This paper focuses on exploring how the inner context of developers affects their communication with AI-powered tools. Inner context refers to the personality of developers, their communication style, perception of information, and their habits. The goal of this research is understand how inner context affects the perception of AI-generated responses on various topics, from question answering, to code generation, debugging and refactoring. By analyzing publicly available code repositories, particularly on GitHub, we aim to understand how developers interact with each other, with AI responses and the code they are working on in their development environment. Through methods inspired by cognitive psychology, we will investigate differences in how developers perceive ChatGPT-generated code and responses, as well as their perception and attitudes towards other developers’ code and comments. The research uses sentiment and emotion analysis, as well as classification and clustering techniques. This analysis aims to understand the differences in perception between developers’ and ChatGPT-generated output, which can then be used to synthesize human-to-AI feedback from human-to-human feedback. This feedback can later be applied to fine-tune the code generation model, increasing the level of acceptance for AI-generated code in the software development process. The expected outcomes of the project include creation of models for difference identification as well as training advanced large language models.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Evaluation and Prediction of Human Software Developers’ Perception of Large Language Models Suggestions Using GitHub Data

  • Arsenii Pimenov,
  • Sergey Kovalchuk

摘要

This paper focuses on exploring how the inner context of developers affects their communication with AI-powered tools. Inner context refers to the personality of developers, their communication style, perception of information, and their habits. The goal of this research is understand how inner context affects the perception of AI-generated responses on various topics, from question answering, to code generation, debugging and refactoring. By analyzing publicly available code repositories, particularly on GitHub, we aim to understand how developers interact with each other, with AI responses and the code they are working on in their development environment. Through methods inspired by cognitive psychology, we will investigate differences in how developers perceive ChatGPT-generated code and responses, as well as their perception and attitudes towards other developers’ code and comments. The research uses sentiment and emotion analysis, as well as classification and clustering techniques. This analysis aims to understand the differences in perception between developers’ and ChatGPT-generated output, which can then be used to synthesize human-to-AI feedback from human-to-human feedback. This feedback can later be applied to fine-tune the code generation model, increasing the level of acceptance for AI-generated code in the software development process. The expected outcomes of the project include creation of models for difference identification as well as training advanced large language models.