This chapter provides a definition of the term “benchmark” followed by definitions of the major system quality attributes that are typically subject of benchmarking. After that, a classification of the different types of benchmarks is provided, followed by an overview of strategies for performance benchmarking. Next, the quality criteria for good benchmarks are discussed in detail. Reproducibility has gained increasing attention over the past decade as a basis for realizing the principles of findability, accessibility, interoperability, and reusability (FAIR) of research data. The chapter includes a definition of reproducibility as well as differentiation from related terms such as repeatability and replicability. It further elaborates on approaches for enhancing reproducibility, including how benchmarks help to improve the reproducibility of research results. Finally, the chapter is wrapped up by a discussion of application scenarios for benchmarks.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Benchmarking Basics

  • Samuel Kounev,
  • Klaus-Dieter Lange,
  • Jóakim von Kistowski

摘要

This chapter provides a definition of the term “benchmark” followed by definitions of the major system quality attributes that are typically subject of benchmarking. After that, a classification of the different types of benchmarks is provided, followed by an overview of strategies for performance benchmarking. Next, the quality criteria for good benchmarks are discussed in detail. Reproducibility has gained increasing attention over the past decade as a basis for realizing the principles of findability, accessibility, interoperability, and reusability (FAIR) of research data. The chapter includes a definition of reproducibility as well as differentiation from related terms such as repeatability and replicability. It further elaborates on approaches for enhancing reproducibility, including how benchmarks help to improve the reproducibility of research results. Finally, the chapter is wrapped up by a discussion of application scenarios for benchmarks.