Real Time In-Situ Activity Support for Human Behavior Using Eye Tracking and Motion Capturing

Supporting human activities using automated systems is an inherent aspect of making work more performant. We know support systems from process industries, driver assistance, or human–robot-collaboration. Recently, also exoskeletons were established to support human handling in manufacturing or to assist disabled persons in mobility. An important enabler for good human support is to understand those human intentions that need to be supported. Intentions need to be recognized by the support system early enough to generate the support function timely in the first place. Besides support functions need to be user-intention specific, e.g., need to provide assistance when and what the user really needs in the given contextual conditions. For this, support systems need to be able to identify users’ behavior and to then interpret and assist the user with the right content. This work shows an approach to identify the user intentions then to generate appropriate assistance. The approach makes practical use of task recognition based on pupillary dynamics and artificial neural networks. To realize context-related user support, a concept for context-related user support system using binocular See-Through Displays is shown. To identify tasks during their performance, characteristic features have been gathered from literature. Based on these basic parameters, further features have been developed for context identification. An artificial neural network is trained with the compiled and developed features to determine their suitability to classify activities. To generate task specific eye tracking data, activities are defined that correspond to industry-typical activities. During the performance of the defined tasks, pupillary dynamics are recorded on 9 subjects using an eye tracking system. From this, data sets, the features are extracted and an artificial neural network is trained to generate a classifier, which is able to classify the corresponding activity in-situ. In order to optimize the classification performance, a genetic algorithm (GA) is used to identify the features and feature sets with the highest relevance for the task identification using the artificial neural network.