Collaborative Benchmarking Rule-Reasoners with B-Runner
摘要
Conducting experimental analysis on rule reasoners is a mainstream task for validating novel algorithms and systems. Nevertheless, providing robust, verifiable, and reproducible experiments can still raise a sensible challenge. This paper introduces B-Runner, an open library for collaborative benchmarking focusing on the deployment of extended tests for knowledge and rule-based systems with low cost and high robustness. B-Runner reduces the benchmarking setup time while guaranteeing experiment repeatability. Also, it improves the scrutability of experimental protocols thereby enhancing the fairness of system comparisons.