Co-BioBench and SDTL: Advancing Human-AI Collaboration in Open-Ended Scientific Discovery
摘要
Achieving autonomous scientific discovery remains a formidable challenge for even the most advanced Large Language Models (LLMs). Seamlessly integrating LLMs into scientific workflows particularly by delineating the optimal boundary between human and machine capabilities to foster synergistic collaboration presents a complex, underexplored issue. To address this, we introduce Co-BioBench, a novel tool learning benchmark designed to assess LLM agents’ ability to collaborate with human scientists in exploring open-ended scientific research domains. Derived from peer-reviewed bioinformatics literature, Co-BioBench encompasses 3,785 meta-tasks across 15 diverse research scenarios. Each scenario guides agents through the iterative process of refining an initial hypothesis to produce a scientifically robust study. Evaluations using Co-BioBench reveal a key limitation in current LLM agents: their inadequate proficiency in leveraging and orchestrating external tools. To overcome this, we propose SDTL, a fine-tuning framework tailored to complex tool-calling tasks. Utilizing a robust corpus generated during development, SDTL enhances a tool-augmented model’s ability to retrieve, comprehend, and integrate diverse tools to achieve sophisticated objectives. Extensive experiments on Co-BioBench and other benchmarks confirm SDTL’s efficacy, with fine-tuned models significantly outperforming recent state-of-the-art function-calling approaches.