Many real-world NLU deployments require coverage of thousands of dynamically evolving topics beyond what manually curated data can sustain. This paper presents an automated pipeline deployed at Technische Universität Berlin (TU Berlin) to extract and train an intent classification model over 2,600 topics with zero manual annotations. We evaluate the effectiveness and generalizability of this pipeline using both curated and LLM-generated data, showing that our automatically trained models surpasses manually engineered systems in some dimensions. While lacking standardized benchmarks, we use a combination of internal evaluation strategies to provide a well-rounded assessment of model robustness. These findings support the use of scalable, self-maintaining NLU systems in complex and dynamic information environments.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Analyzing Web-Scraped and Generated Inputs for Automatic and Scalable Intent Classification

  • Philine Kowol,
  • Stefan Hillmann

摘要

Many real-world NLU deployments require coverage of thousands of dynamically evolving topics beyond what manually curated data can sustain. This paper presents an automated pipeline deployed at Technische Universität Berlin (TU Berlin) to extract and train an intent classification model over 2,600 topics with zero manual annotations. We evaluate the effectiveness and generalizability of this pipeline using both curated and LLM-generated data, showing that our automatically trained models surpasses manually engineered systems in some dimensions. While lacking standardized benchmarks, we use a combination of internal evaluation strategies to provide a well-rounded assessment of model robustness. These findings support the use of scalable, self-maintaining NLU systems in complex and dynamic information environments.