Detecting the imbalances in the assembly line is important for work productivity. Moreover, it is important to facilitate an assembly worker who has a higher workload or no idle time between the work repetitions. It is a difficult task for an assembly manager to measure the idle time of each assembly line worker and balance the workload between the workers. This research uses Artificial Intelligence (AI) to measure the idle time of an assembly worker. Computer vision, object detection and motion tracking methods are used to measure the idle time of an assembly worker. Idle time ratio (ITR) is the measure used to calculate the real-time workload intensity of the assembly workers. The measured ITR values are helpful to the assembly manager while identifying higher or lower workload workers and making a workload balance between them. Details of AI-based workers’ work monitoring, idle time measurement, and ITR calculation are elaborated. Furthermore, a use-case illustration is given to demonstrate the scope of the proposed AI-based worker’s idle time measurement for a three-pin plug assembly line, where the developed AI algorithm calculated the ITR value of the assembly worker from their work recording.