We propose a novel supervised contrastive learning framework for video representation learning that can leverage temporally global context. First, we introduce a video-to-image aggregation strategy, where several frames from each video are spatially arranged into a single input image. This design allows us to use pre-trained off-the-shelf CNN backbones (e.g. ResNet-50) and avoids the computational overhead of more complex video transformer models [1]. Second, we design a contrastive learning objective that directly compares the pairwise projections generated by the model. Positive pairs are defined as projections from videos with the same label, whereas all other projections are contrasted as negatives. We create multiple natural “views” of the same video using different temporal frame samplings of the same underlying video. Rather than relying on augmentations, these frame-level changes yield diverse positive samples with global context, mitigating over-fitting and creating good discriminative representations. Empirical results in Penn Action and HMDB51 demonstrate the effectiveness of our approach, surpassing state-of-the-art methods in classification accuracy while demanding fewer computational resources. The proposed Supervised Contrastive Frame Aggregation (SCFA) strategy can learn quality video representations in both supervised and self-supervised settings, thus providing an efficient solution for video-based tasks like classification, captioning, etc. In particular, our Supervised Contrastive Frame Aggregation (SCFA) method achieves 76% classification accuracy in Penn Action over 43% with the ViVIT model. We also get 48% accuracy in the video classification on HMDB51, which is significantly superior to the ViVIT model that achieves 37% accuracy.