This panel discussion will address the top-of-mind subject of Agentic AI Systems. With the rise of Generative AI and associated Large Language Models (LLMs) and Foundation Models (FMs), the focus of research and practice is shifting to building AI Agents for targeted tasks. Furthermore, a collection of AI Agents are being stitched together to design and deploy Agentic AI systems for enterprise-level workflows and use cases. While the excitement around building such systems that employ LLMs and FMs as building blocks grows, the need to assess the performance of multi-agent systems is growing in importance as well. Metrics such as latency, throughput, accuracy and power consumption are, of course, necessary, but also not sufficient. Additional consideration must be given to the type of workflow, the criticality and resiliency of the use-case, and the target infrastructure for deployment. The panelists will bring their perspectives about building distributed autonomous Agentic AI systems, leveraging their experience spanning industry as well as academia. Factors crucial to developing a comprehensive framework for useful benchmarks will be discussed, with the goal of highlighting best practices for as well as limitations in current state of the art in AI system benchmarking.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Benchmarking Considerations for Agentic AI Systems

  • Ajay Dholakia,
  • Sachin Gopal Wani,
  • David Ellison,
  • Miro Hodak,
  • Debojyoti Dutta,
  • Shishir Nagaraja,
  • Raj Ranjan

摘要

This panel discussion will address the top-of-mind subject of Agentic AI Systems. With the rise of Generative AI and associated Large Language Models (LLMs) and Foundation Models (FMs), the focus of research and practice is shifting to building AI Agents for targeted tasks. Furthermore, a collection of AI Agents are being stitched together to design and deploy Agentic AI systems for enterprise-level workflows and use cases. While the excitement around building such systems that employ LLMs and FMs as building blocks grows, the need to assess the performance of multi-agent systems is growing in importance as well. Metrics such as latency, throughput, accuracy and power consumption are, of course, necessary, but also not sufficient. Additional consideration must be given to the type of workflow, the criticality and resiliency of the use-case, and the target infrastructure for deployment. The panelists will bring their perspectives about building distributed autonomous Agentic AI systems, leveraging their experience spanning industry as well as academia. Factors crucial to developing a comprehensive framework for useful benchmarks will be discussed, with the goal of highlighting best practices for as well as limitations in current state of the art in AI system benchmarking.