Surgical triplet video understanding is essential for accurately recognizing and classifying surgical actions in real-time video data, enabling improved surgical planning and support. However, the recognition of surgical video triplets is particularly influenced by the varying frequency and duration of specific actions. Furthermore, the similarity in motion trajectories across various surgical actions presents another issue in video-based fine-grained surgical action recognition, complicating the differentiation of similar actions. To address these challenges, we propose a new comprehensive paired image-text surgical activity event dataset (SAE), consisting of 90,500 pairs of images and text depicting surgical actions. Additionally, we introduce TriClip, a novel dual-branch contrastive multimodal framework, which effectively bridges the gap between visual and textual modalities in surgical action recognition. By leveraging transferable visual models from natural language supervision, TriClip was evaluated using the CholecT45 dataset, where it achieved an SOTA average precision of 42.1%, setting a new state-of-the-art in the field of surgical action recognition.