Video emotion classification often utilizes fully RGB data, which can raise privacy and bias concerns and has an impact on autonomous systems that rely on data-efficient processing. As a remedy, we propose multiple approaches to use temporal difference videos as a sparse alternative representation. We aim to evaluate whether RGB data is needed to achieve computer vision goals. To this end, we first create an RGB baseline on Video Emotion Classification benchmarks with a Vision Transformer for videos. We evaluate the impact of underlying motion versus appearance features and the influence of the temporal position on classification accuracy in RGB material. Second, we introduce and evaluate our different approaches of temporal difference videos as input. Third, we propose an extension with a SSL reconstruction task, which we evaluate in transfer learning. In summary, we demonstrate new possibilities for utilizing temporal difference videos as a sparse alternative to fully colored RGB video data, thereby promoting privacy and data efficiency.