Interrater agreement and variability in visual reading of [18F] flutemetamol PET images

The purpose of this study was to validate the concordance of visual ratings of [18F] flutemetamol amyloid positron emission tomography (PET) images and to investigate the correlation between the agreement of each rater and the Centiloid (CL) scale.

A total of 192 participants, clinically classified as cognitively normal (CN) (n = 59), mild cognitive impairment (MCI) (n = 65), Alzheimer’s disease (AD) (n = 55), or non-AD dementia (n = 13), participated in this study. Three experts conducted visual ratings of the amyloid PET images for all 192 patients, assigning a confidence level to each rating on a three-point scale (certain, probable, or neither). The positive or negative determination of amyloid PET results was made by majority vote. The CL value was calculated using the CapAIBL pipeline.

Overall, 101 images were determined to be positive, and 91 images were negative. Of the 101 positive images, the three raters were in complete agreement for 92 images and in disagreement for 9 images. Of the 91 negative images, the three raters were in complete agreement for 75 images and in disagreement for 16 images. Interrater reliability among the three experts was particularly high, with both Fleiss’ kappa and Conger’s kappa measuring 0.83 (0.76–0.89). The CL values of the unanimous positive group were significantly greater than those of the other groups, whereas the CL values of the unanimous negative group were significantly lower than those of the other groups. Images with rater disagreement had intermediate CLs. In cases with a high confidence level, the positive or negative visual ratings were in almost complete agreement. However, as confidence levels decreased, experts’ visual ratings became more variable. The lower the confidence level was, the greater the number of cases with disagreement in the visual ratings.

Three experts independently rated 192 amyloid PET images, achieving a high level of interrater agreement. However, in patients with intermediate amyloid accumulation, visual ratings varied. Therefore, determining positive and negative decisions in these patients should be performed with caution.