Scholarlink

Objectives <p>To investigate the intra- and inter-rater reliability of the total methodological radiomics score (METRICS) and its items through a multi-reader analysis.</p> Materials and methods <p>A total of 12 raters with different backgrounds and experience levels were recruited for the study. Based on their level of expertise, raters were randomly assigned to the following groups: two inter-rater reliability groups, and two intra-rater reliability groups, where each group included one group with and one group without a preliminary training session on the use of METRICS. Inter-rater reliability groups assessed all 34 papers, while intra-rater reliability groups completed the assessment of 17 papers twice within 21 days each time, and a “wash out” period of 60 days in between.</p> Results <p>Inter-rater reliability was poor to moderate between raters of group 1 (without training; ICC = 0.393; 95% CI = 0.115–0.630; <i>p</i> = 0.002), and between raters of group 2 (with training; ICC = 0.433; 95% CI = 0.127–0.671; <i>p</i> = 0.002). The intra-rater analysis was excellent for raters 9 and 12, good to excellent for raters 8 and 10, moderate to excellent for rater 7, and poor to good for rater 11.</p> Conclusion <p>The intra-rater reliability of the METRICS score was relatively good, while the inter-rater reliability was relatively low. This highlights the need for further efforts to achieve a common understanding of METRICS items, as well as resources consisting of explanations, elaborations, and examples to improve reproducibility and enhance their usability and robustness.</p> Key Points <p><Emphasis Type="BoldItalic">Questions</Emphasis> <i>Guidelines and scoring tools are necessary to improve the quality of radiomics research; however, the application of these tools is challenging for less experienced raters</i>.</p> <p><Emphasis Type="BoldItalic">Findings</Emphasis> <i>Intra-rater reliability was high across all raters regardless of experience level or&#xa0;previous training, and inter-rater reliability was generally poor to moderate across raters</i>.</p> <p><Emphasis Type="BoldItalic">Clinical relevance</Emphasis> <i>Guidelines and scoring tools are necessary for proper reporting in radiomics research and for closing the gap between research and clinical implementation. There is a need for further resources offering explanations, elaborations, and examples to enhance the usability and robustness&#xa0;of these guidelines</i>.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Reproducibility of methodological radiomics score (METRICS): an intra- and inter-rater reliability study endorsed by EuSoMII

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2025年2月19日

查看全文
  • Tugba Akinci D’Antonoli,
  • Armando Ugo Cavallo,
  • Burak Kocak,
  • Alessandra Borgheresi,
  • Andrea Ponsiglione,
  • Arnaldo Stanzione,
  • Emmanouil Koltsakis,
  • Fabio Martino Doniselli,
  • Federica Vernuccio,
  • Lorenzo Ugga,
  • Matthaios Triantafyllou,
  • Merel Huisman,
  • Michail E. Klontzas,
  • Romina Trotta,
  • Roberto Cannella,
  • Salvatore Claudio Fanni,
  • Renato Cuocolo

摘要

Objectives

To investigate the intra- and inter-rater reliability of the total methodological radiomics score (METRICS) and its items through a multi-reader analysis.

Materials and methods

A total of 12 raters with different backgrounds and experience levels were recruited for the study. Based on their level of expertise, raters were randomly assigned to the following groups: two inter-rater reliability groups, and two intra-rater reliability groups, where each group included one group with and one group without a preliminary training session on the use of METRICS. Inter-rater reliability groups assessed all 34 papers, while intra-rater reliability groups completed the assessment of 17 papers twice within 21 days each time, and a “wash out” period of 60 days in between.

Results

Inter-rater reliability was poor to moderate between raters of group 1 (without training; ICC = 0.393; 95% CI = 0.115–0.630; p = 0.002), and between raters of group 2 (with training; ICC = 0.433; 95% CI = 0.127–0.671; p = 0.002). The intra-rater analysis was excellent for raters 9 and 12, good to excellent for raters 8 and 10, moderate to excellent for rater 7, and poor to good for rater 11.

Conclusion

The intra-rater reliability of the METRICS score was relatively good, while the inter-rater reliability was relatively low. This highlights the need for further efforts to achieve a common understanding of METRICS items, as well as resources consisting of explanations, elaborations, and examples to improve reproducibility and enhance their usability and robustness.

Key Points

Questions Guidelines and scoring tools are necessary to improve the quality of radiomics research; however, the application of these tools is challenging for less experienced raters.

Findings Intra-rater reliability was high across all raters regardless of experience level or previous training, and inter-rater reliability was generally poor to moderate across raters.

Clinical relevance Guidelines and scoring tools are necessary for proper reporting in radiomics research and for closing the gap between research and clinical implementation. There is a need for further resources offering explanations, elaborations, and examples to enhance the usability and robustness of these guidelines.