The detection of micro-expression Action Units (AUs) is a formidable challenge in affective computing, pivotal for decoding subtle, involuntary human emotions. While Large Language Models (LLMs) demonstrate profound reasoning abilities, their application to the fine-grained, low-intensity domain of micro-expression AU detection remains unexplored. This paper introduces AU-LLM, the first framework to use an LLM for detecting AUs in micro-expression datasets, addressing subtle intensities and data scarcity. We address the vision-language semantic gap with the Enhanced Fusion Projector (EFP). The EFP uses a Multi-Layer Perceptron (MLP) to fuse mid-level (local texture) and high-level (global semantics) visual features from a 3D-CNN backbone into a single, information-dense token. This compact representation effectively empowers the LLM to perform nuanced reasoning over subtle facial muscle movements. Extensive evaluations on CASME II and SAMM datasets, including LOSO and cross-domain protocols, show AU-LLM establishes a new state-of-the-art, validating the potential of LLM-based reasoning for micro-expression analysis. The codes are available at Link .

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

AU-LLM: Micro-Expression Action Unit Detection via Enhanced LLM-Based Feature Fusion

  • Zhishu Liu,
  • Kaishen Yuan,
  • Bo Zhao,
  • Yong Xu,
  • Zitong Yu

摘要

The detection of micro-expression Action Units (AUs) is a formidable challenge in affective computing, pivotal for decoding subtle, involuntary human emotions. While Large Language Models (LLMs) demonstrate profound reasoning abilities, their application to the fine-grained, low-intensity domain of micro-expression AU detection remains unexplored. This paper introduces AU-LLM, the first framework to use an LLM for detecting AUs in micro-expression datasets, addressing subtle intensities and data scarcity. We address the vision-language semantic gap with the Enhanced Fusion Projector (EFP). The EFP uses a Multi-Layer Perceptron (MLP) to fuse mid-level (local texture) and high-level (global semantics) visual features from a 3D-CNN backbone into a single, information-dense token. This compact representation effectively empowers the LLM to perform nuanced reasoning over subtle facial muscle movements. Extensive evaluations on CASME II and SAMM datasets, including LOSO and cross-domain protocols, show AU-LLM establishes a new state-of-the-art, validating the potential of LLM-based reasoning for micro-expression analysis. The codes are available at Link .