Towards Using Biometric Transcripts for Analyzing Multimodal Sensory Information Processing Data
摘要
In immersive environments like virtual reality (VR), task performance can be affected by different visual and auditory stimuli. Analyzing multiple aspects of user experience, such as cognitive load and cognitive alertness, in these scenarios can yield multimodal datasets that may make it difficult to detect patterns or relationships between different data modalities. Moreover, these large data sets would cause the researcher to split their attention. This study proposes a synchronized method for analyzing temporal multimodal data more efficiently, called Biometric Transcript for Information Processing (BTIP). In this paper, we demonstrate the proposed approach and the usefulness of viewing temporal multimodal data concurrently. We used the following data modalities collected during an interactive VR task in the presence of visual and auditory distractors: electroencephalogram (EEG) (alpha and theta waves), pupil dilation, fixation location, and gaze duration for each fixation location for the first engagement of the first instance with each distractor. We assessed sensory overloading and changes to attention through the fluctuations of alpha and theta wave activity and explored the connection to pupil dilation by analyzing the patterns found in the biometric transcript. Our data indicated a significantly positive correlation between alpha and theta waves. Peaks in alpha activity correspond to enhanced focus, whereas peaks in theta wave activity correspond to overloading. We further discuss the implications of using this method to research nuances in sensory processing research, specifically with biometric feedback. These insights suggest that this method is promising to be used effectively during the analysis process of temporal multimodal data.