Assessing the Quality of Behavioral Data Obtained by Human Observers Using Cohen’s Kappa and Accessory Metrics: Development of the Algorithms and an Open-Source Library

Behavioral recordings made by human observers (HOs) are central to animal pre-clinical behavioral models (ABM) of neurobiological diseases, where behaviors (e.g., swimming or immobility) are transcripted from video recordings of experi-ments by HOs. These models face criticism due to their vulnerability to reproductibility issues; evaluation of HO’s reliability during training can help to control this source of error. Here, we propose and test algorithms for estimation of Co-hen’s Kappa (K) index and accessory measures (maximum K, prevalence, bias) associated with bootstrapping (BS) of behavioral ratings produced during a real experiment using the rat’s Forced Swimming Test (FST), to evaluate intra-Hos reliability for the recorded categories. Present results indicate that the use of repli-cas after BS faithfully mirrors most of the concordance attributes of the original transcripts while allowing a statistical evaluation of intra-HO’s reliability, and their differences concerning the maximum agreement (Kmax), and the probabili-ties of under-or overestimation of K (bias and prevalence). The use of these tools can inform and optimize the performance of HOs in the use of ABM, without re-quiring time-intensive re-testing, favoring the reproducibility of the data obtained by these procedures.