We present a lightweight keypoint-based Recurrent Neural Network (RNN) approach to the 2023 RoboCup Standard Platform League Visual Referee Challenge. The goal of the challenge is to classify 13 different static and dynamic referee gestures during a robotic soccer match. The developed solution must be lightweight and perform well from anywhere on the field. Multiple referees may be present on the field, with the distinguishing trait of the visual challenge referee being their red gloves. We use the lightweight Convolutional Neural Network pose detector BlazePose to extract the pose key-points. To isolate the referee in the frame, HSV colour segmentation is used to find red glove regions and mask unneeded sections of the image. A lightweight RNN is used to classify the sequence of keypoints. We compare the results of our work to previous solutions developed for the 2022 Visual Referee Challenge. Analysis on real-time testing shows our method achieves strong performance for all gestures on many different locations on the soccer field.