Humans share a universal and fundamental set of emotions which are exhibited through consistent facial expressions. An algorithm that performsdetection, extraction, and evaluation of these facial expressions will allow for automatic recognition of human emotion in images and videos. Machine learning based approaches to facial analysis and video analysis have recently demonstrated high performance on a variety of key tasks such as face recognition, emotion recognition and activity recognition. In the case of video,information often must be aggregated across a variable length sequence of frames to produce a classification result. Prior work using convolutional neural networks (CNNs) for emotion recognition in video has relied on temporal averaging and pooling operations reminiscent of widely used approaches for the spatial aggregation of information.