Scholarlink

<p>Click-through rate (CTR) prediction aims to infer the probability that users will click on advertisements, which is crucial for platforms to plan and optimize the expected revenues in online advertising. Substantial research efforts have been invested to develop a variety of CTR prediction models. However, there remains a lack of standardized benchmarks for validating and comparing the performance of CTR prediction models, especially in the situation with numerous models developed by researchers and practitioners. This research designs a unified architecture of CTR prediction benchmark (Bench-CTR) platform that offers flexible interfaces with datasets and components of a wide range of CTR prediction models. Moreover, we construct a comprehensive system of evaluation protocols encompassing real-world and synthetic datasets, a taxonomy of metrics, standardized procedures and experimental guidelines for calibrating the performance of CTR prediction models. Furthermore, we implement the proposed benchmark platform and conduct a comparative study to evaluate a wide range of state-of-the-art models from traditional multivariate statistical to modern large language model (LLM)-based approaches on three public datasets and two synthetic datasets. Experimental results reveal that, (1) high-order models largely outperform low-order models, though such advantage varies in terms of metrics and on different datasets; (2) LLM-based models demonstrate a remarkable data efficiency, i.e., achieving the comparable performance to other models while using only 2% of the training data; (3) the performance of CTR prediction models has achieved significant improvements from 2015 to 2016, then reached a stage with slow progress, which is consistent across various datasets. This benchmark is expected to facilitate model development and evaluation and enhance practitioners’ understanding of the underlying mechanisms of models in the area of CTR prediction. Code is available at <a href="https://github.com/NuriaNinja/Bench-CTR">https://github.com/NuriaNinja/Bench-CTR</a>.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Toward a benchmark for CTR prediction in online advertising: datasets, evaluation protocols and perspectives

  • 期刊论文

  • 发表日期：

    2025年11月7日

查看全文
  • Shan Gao,
  • Yanwu Yang

摘要

Click-through rate (CTR) prediction aims to infer the probability that users will click on advertisements, which is crucial for platforms to plan and optimize the expected revenues in online advertising. Substantial research efforts have been invested to develop a variety of CTR prediction models. However, there remains a lack of standardized benchmarks for validating and comparing the performance of CTR prediction models, especially in the situation with numerous models developed by researchers and practitioners. This research designs a unified architecture of CTR prediction benchmark (Bench-CTR) platform that offers flexible interfaces with datasets and components of a wide range of CTR prediction models. Moreover, we construct a comprehensive system of evaluation protocols encompassing real-world and synthetic datasets, a taxonomy of metrics, standardized procedures and experimental guidelines for calibrating the performance of CTR prediction models. Furthermore, we implement the proposed benchmark platform and conduct a comparative study to evaluate a wide range of state-of-the-art models from traditional multivariate statistical to modern large language model (LLM)-based approaches on three public datasets and two synthetic datasets. Experimental results reveal that, (1) high-order models largely outperform low-order models, though such advantage varies in terms of metrics and on different datasets; (2) LLM-based models demonstrate a remarkable data efficiency, i.e., achieving the comparable performance to other models while using only 2% of the training data; (3) the performance of CTR prediction models has achieved significant improvements from 2015 to 2016, then reached a stage with slow progress, which is consistent across various datasets. This benchmark is expected to facilitate model development and evaluation and enhance practitioners’ understanding of the underlying mechanisms of models in the area of CTR prediction. Code is available at https://github.com/NuriaNinja/Bench-CTR.