Personality inducing has emerged as a critical research area in modern intelligent systems, which focuses on adapting to traits of specific individuals for delivering tailored experiences. Although large language models (LLMs) have become increasingly proficient at simulating personality traits, two major challenges remain. First, existing research focuses on psychological questionnaires, which exhibit a significant gap from real-world scenarios, making it unclear to measure personality induction performance in scenario situations. Second, subtle differences between personalities can also lead to significantly different behaviors. In this paper, we present a benchmark, VTPI (Vignette Tests for Nuanced Personality Inducing), comprising vignette questions that assess whether induction methods successfully induce the personality traits. We find that current inducing approaches fail catastrophically on inducing nuanced personalities under our constructed questions from real-world scenarios. We thus develop a simple yet effective induction method (DPI) that is capable of capturing subtle differences between nuanced personality traits for precise behavior induction. While VTPI remains challenging, we show that DPI scales well with LLMs (e.g., ChatGPT-4o and DeepSeek-R1) and outperforms previous methods by a large margin (average 19.65% improvement of F1 on Qwen2.5-14B and 32B models).

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Assessing Nuanced Personality Inducing in Language Models via Vignette Tests

  • Xingsheng Zhang,
  • Luxi Xing,
  • Chen Zhang,
  • Yanbing Liu,
  • Yifan Deng,
  • Yue Hu,
  • Zhengxu Hou

摘要

Personality inducing has emerged as a critical research area in modern intelligent systems, which focuses on adapting to traits of specific individuals for delivering tailored experiences. Although large language models (LLMs) have become increasingly proficient at simulating personality traits, two major challenges remain. First, existing research focuses on psychological questionnaires, which exhibit a significant gap from real-world scenarios, making it unclear to measure personality induction performance in scenario situations. Second, subtle differences between personalities can also lead to significantly different behaviors. In this paper, we present a benchmark, VTPI (Vignette Tests for Nuanced Personality Inducing), comprising vignette questions that assess whether induction methods successfully induce the personality traits. We find that current inducing approaches fail catastrophically on inducing nuanced personalities under our constructed questions from real-world scenarios. We thus develop a simple yet effective induction method (DPI) that is capable of capturing subtle differences between nuanced personality traits for precise behavior induction. While VTPI remains challenging, we show that DPI scales well with LLMs (e.g., ChatGPT-4o and DeepSeek-R1) and outperforms previous methods by a large margin (average 19.65% improvement of F1 on Qwen2.5-14B and 32B models).