Benchmarking Considerations for Agentic AI Systems
摘要
This panel discussion will address the top-of-mind subject of Agentic AI Systems. With the rise of Generative AI and associated Large Language Models (LLMs) and Foundation Models (FMs), the focus of research and practice is shifting to building AI Agents for targeted tasks. Furthermore, a collection of AI Agents are being stitched together to design and deploy Agentic AI systems for enterprise-level workflows and use cases. While the excitement around building such systems that employ LLMs and FMs as building blocks grows, the need to assess the performance of multi-agent systems is growing in importance as well. Metrics such as latency, throughput, accuracy and power consumption are, of course, necessary, but also not sufficient. Additional consideration must be given to the type of workflow, the criticality and resiliency of the use-case, and the target infrastructure for deployment. The panelists will bring their perspectives about building distributed autonomous Agentic AI systems, leveraging their experience spanning industry as well as academia. Factors crucial to developing a comprehensive framework for useful benchmarks will be discussed, with the goal of highlighting best practices for as well as limitations in current state of the art in AI system benchmarking.