This study investigates the classification of humor into six distinct genres—irony, sarcasm, exaggeration, incongruity-absurdity, self-deprecating humor, and wit-surprise—using data from the JOKER @ CLEF 2024 shared task. We evaluate the effectiveness of fine-tuning Large Language Models (LLMs), with a primary focus on Llama 3-8B, and compare its performance against a RoBERTa baseline and advanced Large Reasoning Models (LRMs) like DeepSeek-R1. By employing prompt engineering, including the Stanford Alpaca format and Chain-of-Thought (CoT) prompting, our fine-tuned Llama 3 model achieved a high accuracy of 89.68% on development set. However, in the official evaluation, its accuracy was 69.78%, highlighting a significant discrepancy likely due to data distribution differences. A key finding is that LRMs, despite their sophisticated reasoning capabilities enhanced by techniques like Group Relative Policy Optimization (GRPO), performed poorly on this task. This suggests that the nuanced, context-dependent nature of humor comprehension presents a unique challenge that is not directly addressed by improvements in general logical reasoning. Our results affirm that while specialized fine-tuning makes LLMs highly effective for humor classification, genuine humor understanding remains a distinct frontier for AI, requiring more than advanced reasoning alone.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Humour Classification According to Genre and Technique by Fine-Tuning LLMs

  • Shih-Hung Wu,
  • Tsz-Yeung Lau,
  • Yu-Feng Huang

摘要

This study investigates the classification of humor into six distinct genres—irony, sarcasm, exaggeration, incongruity-absurdity, self-deprecating humor, and wit-surprise—using data from the JOKER @ CLEF 2024 shared task. We evaluate the effectiveness of fine-tuning Large Language Models (LLMs), with a primary focus on Llama 3-8B, and compare its performance against a RoBERTa baseline and advanced Large Reasoning Models (LRMs) like DeepSeek-R1. By employing prompt engineering, including the Stanford Alpaca format and Chain-of-Thought (CoT) prompting, our fine-tuned Llama 3 model achieved a high accuracy of 89.68% on development set. However, in the official evaluation, its accuracy was 69.78%, highlighting a significant discrepancy likely due to data distribution differences. A key finding is that LRMs, despite their sophisticated reasoning capabilities enhanced by techniques like Group Relative Policy Optimization (GRPO), performed poorly on this task. This suggests that the nuanced, context-dependent nature of humor comprehension presents a unique challenge that is not directly addressed by improvements in general logical reasoning. Our results affirm that while specialized fine-tuning makes LLMs highly effective for humor classification, genuine humor understanding remains a distinct frontier for AI, requiring more than advanced reasoning alone.