Better Results Through Ambiguity Resolution: Large Language Models that Ask Clarifying Questions

Here we present a pilot study on the Clarifying Questions Document Generator (CQDG), an AI document-generation application designed to ask follow-up questions to the user after receiving their initial prompt. Study participants wrote a prompt requesting the AI to generate a short document such as an email, letter, or other short document. The AI then generated follow-up questions and engaged in a short question-and-answer dialog before creating the requested document. This study examines users’ willingness to engage in a question-and-answer exchange with an AI, as well as their satisfaction with the output of this exchange compared to a baseline output generated using only the users’ original prompts. It was predicted that users would prefer the output that included the solicited information over the baseline result. However, the initial results suggest that there was little to no overall improvement in the final output, with about half of users preferring the baseline output to the result of the question-and-answer exchange. This paper will discuss possible reasons for this result as well as suggestions for how future systems could be improved, which will be incorporated into a larger study later this year.