The increasing use of large language models (LLMs) in qualitative research means that more researchers are using automated coding. Developing better tools and methods to assess the reliability of automated codes is thus a critical concern. In Quantitative Ethnography (QE), Cohen’s kappa (κ) is the standard measure of agreement. In what follows, we propose two new methods for estimating confidence intervals for κ: a Finite Exact test and Finite Bayesian estimation. These new methods take advantage of two pieces of information that are available to researchers in a QE context: the size of the dataset and the base rate of the automated classifier. We compare these new approaches to two existing methods: asymptotic standard error and bootstrapping. Results from 720 simulations show that the two existing methods produce inflated Type I error rates under QE-conditions: a combination of low classifier base rate (<10%), high κ threshold (>0.7), and where only a small number of human-coded samples (<1000) are available. The proposed methods have acceptable Type I error rates. Under these conditions. The proposed methods outperform make it possible to more reliably validate automated coding approaches in QE, particularly for rare but meaningful behaviors.