Multi-Institutional Evaluation of Interrater Agreement of Biomarker-Drug Pair Rankings Based on the ESMO Scale for Clinical Actionability of molecular Targets (ESCAT) and Sources of Discordance

The widespread use of next-generation sequencing in clinical practice has contributed to the accumulation of a large number of genomic findings associated with targeted therapy; therefore, the problem of ranking the detected findings has become acute. The European Society for Medical Oncology Scale of Clinical Actionability of molecular Targets (ESCAT) system was designed by the European Society for Medical Oncology to rank biomarkers into levels of evidence that reflect their potency and clinical significance based on published clinical data. However, the ESCAT system remains imperfect, as it is based on a subjective assessment of the levels of evidence.

The objective of this study was to determine whether the ranking of LOE for biomarker-drug pairs based on the ESCAT system is dependent on the human factor, and to uncover potential issues associated with the use of the framework.

To evaluate the inter-rater agreement, we created a dataset of a total of 154 biomarker-drug pairs for 18 unique tumor types. We aimed to include biomarker-drug pairs that could be considered standard of care as well as less common and under investigated pairs. Fourteen precision oncology experts were invited to assign an ESCAT level of evidence for biomarker-drug pairs. Statistical analysis was carried out using Cohen’s kappa and the Kolmogorov–Smirnov test.

The inter-rater agreement was low with some exceptions, and significant deviations from the consensus level of evidence were observed. For biomarker-drug associations, the deviations from the consensus were observed for more than 50% of the contributors’ rankings. The most agreement between the contributors was observed for lung adenocarcinoma (p < 0.005), while the most disagreement was observed for esophageal cancer (p < 0.01) biomarker-drug pairs in our dataset.

This study demonstrates noteworthy discordances between the precision oncology experts and may provide the directions for future developments in modifying the ESCAT framework and the overall applicability of the results of genomic profiling into clinical practice.