Perfect Sampling

This paper introduces a method for validating Cohen’s kappa ( \(\kappa \) ) as a measure of agreement between two raters by determining the minimum sample size needed to reliably show that \(\kappa \) exceeds a specified threshold \({\tau }_{\kappa }\) . . If two raters achieve perfect agreement on a random sample of that size (i.e., \(\kappa =1\) for the sample), then it can be concluded that \(\kappa >{\tau }_{\kappa }\) . The paper shows that although attaining perfect agreement may be challenging, it is more efficient than sampling approaches that allow disagreements between raters. The paper concludes that perfect sampling is the most efficient way to validate interrater reliability measured using \(\kappa \) .