Scholarlink

<p>Scenario-based testing with driving simulators is extensively used to identify failing conditions of automated driving assistance systems (ADAS) and reduce the amount of in-field road testing. However, existing studies have shown that repeated test execution in the same as well as in distinct simulators can yield different outcomes, which can be attributed to sources of flakiness or different implementations of the physics, among other factors. In this paper, we present <Emphasis FontCategory="NonProportional">MultiSim</Emphasis>, a novel approach to multi-simulation ADAS testing based on a search-based testing approach that leverages an ensemble of simulators to identify failure-inducing, simulator-agnostic test scenarios. During the search, each scenario is evaluated jointly on multiple simulators. Scenarios that produce consistent results across simulators are prioritized for further exploration, while those that fail on only a subset of simulators are given less priority, as they may reflect simulator-specific issues rather than generalizable failures. Our empirical study, which involves testing three lane-keeping ADAS with increasing complexity on different pairs of three widely used simulators, demonstrates that <Emphasis FontCategory="NonProportional">MultiSim</Emphasis> outperforms single-simulator testing by achieving, on average, a higher rate of simulator-agnostic failures by 66%. Compared to a state-of-the-art multi-simulator approach that combines the outcome of independent test generation campaigns obtained in different simulators, <Emphasis FontCategory="NonProportional">MultiSim</Emphasis> identifies on average up to <InlineEquation ID="IEq1"> <EquationSource Format="TEX">\(3.4\times\)</EquationSource> </InlineEquation> more simulator-agnostic failing tests and higher failure rates. To avoid the costly execution of test inputs on which simulators disagree, we propose an enhancement of <Emphasis FontCategory="NonProportional">MultiSim</Emphasis> that leverages surrogate models to predict simulator disagreements and bypass test executions. Our results show that utilizing a surrogate model during the search does not only retain the average number of valid failures but also improves its efficiency in finding the first valid failure. These findings indicate that combining an ensemble of simulators during the search is a promising approach for the automated cross-replication in ADAS testing.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Simulator ensembles for trustworthy autonomous driving systems testing

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2026年2月20日

查看全文
  • Lev Sorokin,
  • Matteo Biagiola,
  • Andrea Stocco

摘要

Scenario-based testing with driving simulators is extensively used to identify failing conditions of automated driving assistance systems (ADAS) and reduce the amount of in-field road testing. However, existing studies have shown that repeated test execution in the same as well as in distinct simulators can yield different outcomes, which can be attributed to sources of flakiness or different implementations of the physics, among other factors. In this paper, we present MultiSim, a novel approach to multi-simulation ADAS testing based on a search-based testing approach that leverages an ensemble of simulators to identify failure-inducing, simulator-agnostic test scenarios. During the search, each scenario is evaluated jointly on multiple simulators. Scenarios that produce consistent results across simulators are prioritized for further exploration, while those that fail on only a subset of simulators are given less priority, as they may reflect simulator-specific issues rather than generalizable failures. Our empirical study, which involves testing three lane-keeping ADAS with increasing complexity on different pairs of three widely used simulators, demonstrates that MultiSim outperforms single-simulator testing by achieving, on average, a higher rate of simulator-agnostic failures by 66%. Compared to a state-of-the-art multi-simulator approach that combines the outcome of independent test generation campaigns obtained in different simulators, MultiSim identifies on average up to \(3.4\times\) more simulator-agnostic failing tests and higher failure rates. To avoid the costly execution of test inputs on which simulators disagree, we propose an enhancement of MultiSim that leverages surrogate models to predict simulator disagreements and bypass test executions. Our results show that utilizing a surrogate model during the search does not only retain the average number of valid failures but also improves its efficiency in finding the first valid failure. These findings indicate that combining an ensemble of simulators during the search is a promising approach for the automated cross-replication in ADAS testing.