Current AI systems, including socially interactive robots, rely heavily on annotated datasets to train machine learning models. However, annotator biases can significantly affect the ground truth of these datasets, leading to biased AI behaviors. This paper investigates how the characteristics of the individuals comprising facial image datasets and online annotators may influence the reported perception of facial expressions. A between-subjects experiment with 119 online annotators revealed that Girls of Color were more prone to misclassification, even with similar facial expressions to White Girls. Annotators who identified as People of Color generally performed better at classifying expressions in Girls of Color, while White annotators were more accurate in classifying expressions in White Girls. Assistive AI-generated recommendations show potential in reducing racial bias, demonstrating a tangible approach toward fairness and accurate representation within ground truth data used to develop affective recognition systems. This work further describes critical implications for the development and deployment of socially interactive robots in diverse social contexts.