Understanding the Impact of Observer Effects on Student Affect

The measurement of affect presents a challenge for researchers in quantitative ethnography and related communities, as each of the several possible methods for obtaining ground truth have downsides and often disagree with each other. One common method is field observations; some accounts have raised concerns about observer effects, but further research is needed to understand how much observer effect is present and exactly how observer effects manifest. In this study, we attempt to quantify and detail observer effects in classroom data collected using the Baker Rodrigo Ocumpaugh Monitoring Protocol (BROMP). We leveraged prediction models developed in prior work to assess students’ affect and disengaged behavior, during periods where observers were present or not present. Statistical analyses revealed differences in both affective states and behaviors on days when field observations were conducted. When observers were present, students showed increases in concentration and decreases in frustration, off-task behavior, and gaming the system. Findings suggest that the collection of observation data changes the proportion of behavior and affect observed, which should be taken into account when designing, implementing, and analyzing observation-based research. We discuss implications of this finding for quantitative ethnography, when observation data is systematized, visualized, and compared. Our findings do not suggest that the individual observations of specific affective states or behaviors are invalid but offer insights for QE scholars to consider when collecting and analyzing observation data.