Abnormal or non-standard operations by ship drivers are one of the major factors leading to water traffic accidents, making the development of a real-time and reliable method for detecting ship driver behavior crucial. This paper proposes a novel approach for multi-person behavior recognition based on the Temporal Shift Module (TSM) algorithm for single-target behavior detection. This algorithm integrates multi-target tracking with single-target behavior recognition methods, achieving simultaneous recognition and detection of multiple drivers within the ship’s bridge. Additionally, a dataset named “SC Action” was created for ship bridge behavior, containing data samples from various ship bridge surveillance videos, including over 1,000 video samples of routine and violation behaviors. Experimental results show that this method, while accurately tracking multiple drivers, achieved a behavior recognition accuracy of 81.72% and is capable of providing early warnings of dangerous behaviors by drivers.