Typical usability tests may use task performance and subjective ratings as the main forms of feedback on the prototype. We take a step further and analyze the transcripts of usability tests for the quality and quantity of feedback, assessed through interview duration, speech rate, unique adjective use, and negativity percentage. We had six interviewers (three female, three male) recruit strangers and friends to complete a usability test of an intentionally frustrating prototype. We had them rate their experienced ease and frustration with the tasks and complete questionnaires about their personality traits and the perceived personality traits of the interviewer. We then transcribed their interviews and compared these objective metrics to their subjective ratings, considering their relationship with the interviewer, their gender and the personality traits of the interviewee and interviewer. We found significant correlations between our objective metrics (e.g., adjectives and negativity, speech rate and adjectives), differences between groups (friends spoke faster and more), and influences of personality on objective metrics (e.g., extraversion on adjectives and negativity) and perceptions of the interviewer (e.g., conscientiousness sees conscientiousness). The correlations we found between speech rate and extraversion were loosely consistent with previous literature. We discuss the implications of our findings for recruiting participants for and analyzing the data of usability tests.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Speaking Fast, Speaking More: How Personality and Social Context Affect User Interview Dynamics - A Transcript Text Analysis

  • Lucy Cui,
  • Matthew Cross,
  • Brandon Day,
  • Chelsea Wan

摘要

Typical usability tests may use task performance and subjective ratings as the main forms of feedback on the prototype. We take a step further and analyze the transcripts of usability tests for the quality and quantity of feedback, assessed through interview duration, speech rate, unique adjective use, and negativity percentage. We had six interviewers (three female, three male) recruit strangers and friends to complete a usability test of an intentionally frustrating prototype. We had them rate their experienced ease and frustration with the tasks and complete questionnaires about their personality traits and the perceived personality traits of the interviewer. We then transcribed their interviews and compared these objective metrics to their subjective ratings, considering their relationship with the interviewer, their gender and the personality traits of the interviewee and interviewer. We found significant correlations between our objective metrics (e.g., adjectives and negativity, speech rate and adjectives), differences between groups (friends spoke faster and more), and influences of personality on objective metrics (e.g., extraversion on adjectives and negativity) and perceptions of the interviewer (e.g., conscientiousness sees conscientiousness). The correlations we found between speech rate and extraversion were loosely consistent with previous literature. We discuss the implications of our findings for recruiting participants for and analyzing the data of usability tests.