To address the highly individualized and variable nature of muscle fatigue among workers on assembly lines, this paper proposes a novel, purely vision-based muscle fatigue detection network (VMFDNet). The algorithm is based on the Mediapipe-hands hand joint key point detection algorithm, which tracks the movement trajectories of hand key points. It calculates the root mean square jitter, trajectory standard deviation, and fatigue level labels obtained from subjective questionnaire surveys from a specific time sliding window, and inputs them into an improved temporal model LSTM for training. Using a laboratory assembly line simulating the assembly of a bottom-mounted worm gear reducer as an example, a video dataset involving eight operators was constructed, with the inference model achieving an accuracy of 87.4%. To validate its generalization performance, the proposed method was tested on a vacuum cleaner assembly line in an industrial setting, achieving an accuracy of 82.3%. This study aims to decouple worker fatigue characteristics from specific task content, enabling generalized fatigue detection and contributing to the realization of human-centered Industry 5.0.